Question and answer text data processing method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of question and answer text data processing, and provides a question and answer text data processing method and device, electronic equipment and a medium. The method comprises the following steps: obtaining to-be-processed document data, and splitting the document data into a plurality of first text sequences; retrieving a second text sequence matched with each first text sequence from a document library based on each first text sequence through a vector retrieval model of the initial version, wherein the document library comprises a plurality of pieces of to-be-processed document data; splicing each first text sequence and the corresponding second text sequence to obtain a plurality of training text sequences; the vector retrieval model of the current version is trained based on all the training text sequences in an unsupervised mode in a loop iteration mode, a vector retrieval model of the target version is obtained, and the vector retrieval model of the current version comprises the vector retrieval model of the initial version. According to the method and the device, the problems of insufficient retrieval model response input content and low correlation in the prior art are solved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of question and answer text data processing, and in particular to a question and answer text data processing method, device, electronic device and medium. Background Art

[0002] With the rapid development of Internet technology, the acquisition, storage and processing of data have become increasingly important. In many business scenarios, it has become common practice to configure and use document data. These document data not only support daily business operations, but also provide necessary information support for decision-making. Traditionally, business-related personnel need to manually review these documents, which is time-consuming and inefficient. In order to improve efficiency, in recent years, technologies that use vector retrieval models to optimize the document retrieval process have emerged. This technology converts document content into vector form and uses machine learning algorithms to train the model, so that business-related personnel can quickly retrieve relevant documents from a large document library by entering queries through applications or web interfaces. In theory, this method can greatly improve the speed and accuracy of information retrieval.

[0003] However, when studying and applying existing vector retrieval technologies, the inventors found that although these technologies perform well in certain scenarios, they still have some shortcomings. Specifically, the existing vector retrieval models cannot effectively respond to the input content when processing certain business-related queries, that is, the match between the input query and the retrieval results is not high. In addition, even if the model is able to return results, the relevance of these results is often low with the actual required answers. This inefficient retrieval not only affects the efficiency of business operations, but may also lead to wrong business decisions. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a question-answer text data processing method, device, electronic device and computer-readable storage medium to solve the technical problems in the prior art that the retrieval model responds to insufficient input content and has low relevance.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing question and answer text data is provided, including: obtaining document data to be processed, and splitting the document data into multiple first text sequences; retrieving second text sequences matching each first text sequence from a document library based on each first text sequence through an initial version of a vector retrieval model, wherein the document library contains multiple document data to be processed; concatenating each first text sequence with its corresponding second text sequence to obtain multiple training text sequences; and training a current version of a vector retrieval model based on each training text sequence in an unsupervised manner in a loop iteration manner to obtain a target version of a vector retrieval model, wherein the current version of the vector retrieval model includes the initial version of the vector retrieval model.

[0006] According to a second aspect of an embodiment of the present disclosure, a question-and-answer text data processing device is provided, including: an acquisition module, used to acquire document data to be processed, and split the document data into multiple first text sequences; a matching module, used to retrieve second text sequences matching each first text sequence from a document library based on each first text sequence through an initial version of a vector retrieval model, wherein the document library contains multiple document data to be processed; a splicing module, used to splice each first text sequence with its corresponding second text sequence to obtain multiple training text sequences; a training module, used to train the current version of the vector retrieval model based on each training text sequence in an unsupervised manner in a cyclic iteration manner to obtain a target version of the vector retrieval model, wherein the current version of the vector retrieval model includes the initial version of the vector retrieval model.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the disclosed embodiment has the following beneficial effects: the disclosed embodiment can create a richer and more detailed training data set by splitting the document data into multiple first text sequences and splicing these sequences with the matching second text sequences retrieved from the document library. This data splicing method can effectively enhance the model's ability to understand and process the document content, thereby achieving higher retrieval accuracy in practical applications. The use of a cyclic iterative training method enables the vector retrieval model to continuously learn from new training text sequences and gradually optimize and adjust its internal parameters. This iterative training method not only improves the model's adaptability to current data, but also enhances the model's ability to process similar data in the future. The target version of the vector retrieval model obtained through iterative training has better performance and can more accurately identify and respond to the query content input by the user. In this way, the retrieved documents will have a higher correlation with the user's query intent, thereby effectively improving user satisfaction and the accuracy of business decisions. Through automated and optimized data processing and model training processes, the need for manual intervention is reduced and the overall retrieval efficiency is improved. At the same time, more accurate retrieval results reduce the time users spend sifting through error information and improve overall work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0012] Figure 2 It is a flowchart of a method for processing question and answer text data provided by an embodiment of the present disclosure;

[0013] Figure 3 It is a flowchart of another question-answer text data processing method provided by an embodiment of the present disclosure;

[0014] Figure 4 is a block diagram of a question-answer text data processing device provided by an embodiment of the present disclosure;

[0015] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0017] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0018] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied is shown.

[0019] like Figure 1As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0020] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0021] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0022] The server 105 may be a server that provides various services. For example, the server 105 may obtain the document data to be processed from the first terminal device 103 (or the second terminal device 102 or the third terminal device 103), split the document data into multiple first text sequences; retrieve the second text sequences matching each first text sequence from the document library based on each first text sequence through the initial version of the vector retrieval model, and the document library contains multiple document data to be processed; concatenate each first text sequence with its corresponding second text sequence to obtain multiple training text sequences; adopt a loop iteration method, train the current version of the vector retrieval model based on each training text sequence in an unsupervised manner, and obtain the target version of the vector retrieval model, and the current version of the vector retrieval model includes the initial version of the vector retrieval model.

[0023] In some embodiments, the question-answer text data processing method provided in the embodiments of the present invention is generally executed by the server 105, and accordingly, the question-answer text data processing device is generally arranged in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the question-answer text data processing method provided in the embodiments of the present invention is not limited to being executed on the server side.

[0024] The question and answer text data processing method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 2 is a flowchart of a method for processing question and answer text data provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0026] like Figure 2 As shown, the question-answer text data processing method includes steps S210 to S240.

[0027] In step S210, document data to be processed is obtained, and the document data is split into a plurality of first text sequences.

[0028] Step S220 , using the initial version of the vector retrieval model to retrieve second text sequences matching each first text sequence from a document library based on each first text sequence, where the document library contains a plurality of document data to be processed.

[0029] In step S230, each first text sequence is concatenated with its corresponding second text sequence to obtain a plurality of training text sequences.

[0030] In step S240, the vector retrieval model of the current version is trained based on each training text sequence in an unsupervised manner in a loop iteration manner to obtain the vector retrieval model of the target version, where the vector retrieval model of the current version includes the vector retrieval model of the initial version.

[0031] The method can create a richer and more detailed training data set by splitting the document data into multiple first text sequences and splicing these sequences with the matching second text sequences retrieved from the document library. This data splicing method can effectively enhance the model's ability to understand and process document content, thereby achieving higher retrieval accuracy in practical applications. The use of a cyclic iterative training method enables the vector retrieval model to continuously learn from new training text sequences and gradually optimize and adjust its internal parameters. This iterative training method not only improves the model's adaptability to current data, but also enhances the model's ability to process similar data in the future. The target version of the vector retrieval model obtained through iterative training has better performance and can more accurately identify and respond to the query content entered by the user. In this way, the retrieved documents will have a higher correlation with the user's query intent, thereby effectively improving user satisfaction and the accuracy of business decisions. Through automated and optimized data processing and model training processes, the need for manual intervention is reduced and the overall retrieval efficiency is improved. At the same time, more accurate retrieval results reduce the time users spend sifting through error information and improve overall work efficiency.

[0032] In some embodiments of the present disclosure, the document data to be processed is configured according to specific business needs. This means that the selected data set is closely related to the business and can reflect the actual operations, strategies, customer interactions, etc. of the business. For example, in the financial industry, document data may include transaction records, customer credit reports, market analysis reports, etc.; in the medical industry, it may include medical records, clinical trial reports, etc. These document data are usually stored in the company's internal document libraries, which are electronic document management systems, databases, or other forms of information storage systems. The data in the document library can be structured data (such as database tables), semi-structured data (such as XML, JSON files), or unstructured data (such as text files, PDF documents, etc.). The selected document data not only needs to cover a wide range of aspects of the business, but also be highly practical and relevant.

[0033] Based on the above embodiments, business document data may include various types of files, such as contracts, reports, emails, meeting minutes, project plans, financial statements, etc. These documents cover all aspects of business operations, such as marketing, human resources, financial management, customer service, etc. Document data includes not only traditional text files, but also formats such as PDF, Excel tables, PowerPoint presentations, etc. In addition, with the development of information technology, it may also include more new formats of documents, such as video, audio recordings, and image files. All business document data is stored in a designated document library. Such a centralized storage method is easy to manage and maintain, ensuring the security and accessibility of data. The document library may be a storage system on a physical server or a data storage service provided by a cloud infrastructure. The document library has comprehensive security measures and permission management functions. This includes security mechanisms such as data encryption, access control, and user authentication to ensure that sensitive information is not accessed without authorization. At the same time, the permission management system allows enterprises to configure access rights according to the roles and responsibilities of employees to ensure the reasonable use of data. In order to ensure the integrity and persistence of data, the document library usually has the functions of regular backup and data recovery. These functions can prevent data loss and damage, and ensure that enterprises can quickly resume business operations when facing hardware failures, software failures or other unexpected situations. The data in the document library is not only passively stored, but also for effective retrieval and use. In the embodiment, the data in the document library is used to train and optimize the vector retrieval model to improve the retrieval efficiency and accuracy of the model. This application demonstrates the active role of document data in actual business operations. With the development of business and the updating of information, the data in the document library also needs to be updated and managed regularly. This includes adding new documents, updating the content of existing documents, deleting outdated or no longer needed documents, etc. Good data management not only ensures the timeliness of data, but also improves the quality and use value of data. Through the above detailed description, we can see the diversity and complexity of business document data, as well as the key role of the document library in data storage, security protection, data management, etc. These factors together support the effective training of the vector retrieval model and the precise execution of business decisions.

[0034] In some embodiments of the present disclosure, obtaining the document data to be processed first involves selecting a suitable document data source. These data sources are usually document libraries within the enterprise, which may contain business-related documents of various formats and types. The selection of document data sources depends on the business problems to be solved and the specific needs of model training. Extracting data from the document library usually requires the use of specific tools or APIs. During the extraction process, specific query statements or filtering conditions can be applied to ensure that the acquired data is highly relevant to business needs. For example, if the goal is to improve customer service, customer communication records or service feedback documents can be extracted first. The extracted document data often contains some unnecessary information, such as formatting errors, duplicate content, or irrelevant data. The above situation can be avoided by data cleaning, including removing useless information, correcting errors, and unifying data formats. Preprocessing includes language detection and conversion to ensure data consistency and availability. The acquired and cleaned document data needs to be split into multiple first text sequences. This step is to refine the content of larger documents to facilitate model learning and understanding of specific information in the documents. The text sequences obtained by splitting can be further standardized, including steps such as removing stop words, stem extraction, and part-of-speech tagging. These processes help reduce data noise and improve the efficiency and effectiveness of model training.

[0035] In some embodiments of the present disclosure, the vector retrieval model may be, for example, "bge-large-zh-v1.5" or "Sentence-Transformer". These models are usually based on deep learning technology and can convert text into vectors in a high-dimensional space. Select a model that matches the language and domain of the document data to be processed. For example, "bge-large-zh-v1.5" is suitable for processing Chinese business documents. Convert the first text sequence into a vector using the selected vector retrieval model. Preprocess the first text sequence, such as removing noise, standardizing, etc. Use the vector retrieval model to convert the processed text into a numerical vector. These vectors can represent the semantic information of the text and are used for subsequent similarity calculations. Perform similar vectorization processing on the document data to be processed in the document library. Store the converted vectors in a data structure suitable for fast retrieval, such as a vector database or a specific index structure. Calculate the similarity between the vector of the first text sequence and the vector of each document in the document library. This usually uses cosine similarity or other metrics. For example, based on the similarity, select a matching second text sequence. Through this process, the vector retrieval model can effectively find documents that are semantically similar to the query text from a large number of documents, greatly improving the efficiency and quality of information retrieval.

[0036] In some embodiments of the present disclosure, each first text sequence is spliced ​​with its corresponding second text sequence to obtain multiple training text sequences. For example, two sequences are distinguished by certain separators (such as [SEP]). The first text sequence may be a vector corresponding to "soak the clean cotton head [MASK] in wax water", the second text sequence may be a vector corresponding to "wax evenly, from the inside to the outside, in a single direction", and the spliced ​​training text sequence may be a vector corresponding to "soak the clean cotton head [MASK] in wax water [SEP] and wax evenly, from the inside to the outside, in a single direction".

[0037] Based on the above embodiments, in order to make the spliced ​​text more natural, appropriate conjunctions or phrases, such as "therefore", "for example", etc., can be added between the two sequences. Before splicing, the text sequence can be further preprocessed. This includes: ensuring that the two text sequences are consistent in language style and format, such as uniformly using simplified Chinese or traditional Chinese, unified tenses, etc. Clear typos, unnecessary punctuation marks, etc. in the text to improve the quality of model training. Use a program to automatically splice the first text sequence and the second text sequence according to a predetermined strategy. For example, this can be achieved using string manipulation functions in a programming language (such as Python). The spliced ​​text sequence will be used as an example in the training data set.

[0038] In some embodiments of the present disclosure, a vector retrieval model of the current version is trained based on each training text sequence in a loop iteration manner to obtain a vector retrieval model of the target version, and the vector retrieval model of the current version includes the vector retrieval model of the initial version. For example, before starting training, it is necessary to determine the vector retrieval model of the current version, which can be an initial version or a version that has been trained to a certain extent. This version of the model will serve as the starting point for training. Using the method described previously, a training text sequence is generated by concatenating the first text sequence and the second text sequence. These sequences will be used for unsupervised learning, with the goal of enabling the model to capture and understand the relationship and semantics between texts. In unsupervised learning, the model needs to learn without explicit labels. For the vector retrieval model, one of the following unsupervised learning methods can be used:

[0039] Autoencoders: Use autoencoders to learn efficient representations of text data. Autoencoders learn a compressive encoding of text by trying to reconstruct its input.

[0040] Contrastive Learning: The model is trained by maximizing the similarity between similar texts and minimizing the similarity between dissimilar texts.

[0041] Clustering algorithms: Use clustering algorithms to discover inherent structures and patterns in text data, and then use this information to optimize vector representations.

[0042] In the iterative training process, the current version of the vector retrieval model will be repeatedly trained using the training dataset, and each iteration may fine-tune the model parameters. This process includes:

[0043] Batch processing: The training data is input into the current version of the vector retrieval model in batches, and the model parameters are updated in each iteration.

[0044] Model evaluation: During the iteration process, the performance of the current version of the vector retrieval model is regularly evaluated using the validation set to monitor learning progress and avoid overfitting.

[0045] Parameter adjustment: Adjust the learning rate, regularization parameters, etc. according to the performance of the current version of the vector retrieval model to optimize the training effect.

[0046] Through repeated iterative training and optimization, we can finally get the target version of the vector retrieval model with satisfactory performance. This model should be able to more accurately retrieve documents that match the semantics of the query text from a large number of documents. Through this cyclic iterative unsupervised training method, the performance of the vector retrieval model can be effectively improved to better adapt it to specific application requirements and data characteristics. This method is particularly suitable for situations where data annotation is expensive or it is difficult to obtain labeled data.

[0047] In some embodiments of the present disclosure, the method further includes: obtaining a real-time version of the vector retrieval model during the loop iteration process; retrieving a second text sequence matching each first text sequence from the document library based on each first text sequence through the initial version of the vector retrieval model includes: retrieving a second text sequence matching each first text sequence from the document library based on each first text sequence through the real-time version of the vector retrieval model. For example, first, the retrieval process is started using the initial version of the vector retrieval model. This initial model is usually a basic model based on pre-training, previous projects, or training from scratch. During the iteration process, the model will be continuously updated based on new data or feedback. These updates can be regular or can be performed immediately after each model is run on new data. The updated model is called a "real-time version" and reflects the latest learning and adjustments. At the beginning of the iteration, the initial version of the model is used to retrieve a matching second text sequence from the document library based on the first text sequence. This process includes converting the first text sequence into a vector and finding the most similar text vector in the document library. As the model is iterated and updated, the real-time version of the model will be used for further retrieval tasks. This means that after each model update, the new version of the model will be used to perform the same retrieval task in the hope of obtaining more accurate or more relevant retrieval results. In each iteration, the model is adjusted based on the performance of the retrieval and possible user feedback. This may include adjusting the parameters of the model, optimizing the algorithm, or retraining some parts of the model.

[0048] In some embodiments of the present disclosure, when the number of second text sequences that match the first text sequence is multiple, the method further includes: calculating the similarity between the first text sequence and each second text sequence; and determining at least one second target text sequence that matches the first text sequence from each second text sequence according to each similarity. For example, a similarity measurement method is used to calculate the similarity between the first text sequence and each second text sequence. Common similarity measurements include cosine similarity, Euclidean distance, or Manhattan distance. According to the calculated similarity, all second text sequences are sorted. According to specific needs, one or several text sequences with the highest similarity can be selected as the second target text sequence. In some cases, a similarity threshold is set, and when the similarity of the second text sequence is higher than this threshold, it is selected as the target text sequence. This helps to ensure that the selected text sequence meets certain standards in quality. In an information retrieval system, this method can help users find the most relevant documents to the query from a large number of documents. In a question-answering system, it can be used to select the answer that best matches the question from the candidate answers. In a document clustering task, similarity calculation can help group documents into related topics or categories. In this way, it can be ensured that the system can not only find relevant text sequences, but also select the target text sequence that is most relevant to the first text sequence, thereby improving the overall effectiveness of the system and user satisfaction.

[0049] In some embodiments of the present disclosure, splicing each first text sequence with its corresponding second text sequence to obtain a plurality of training text sequences includes: splicing each first text sequence with at least one second target text sequence corresponding to it to obtain a plurality of training text sequences. For example, the first text sequence is a query, a question, or any text for which relevant information needs to be found. The second target text sequence is a text with a high degree of match with the first text sequence, as determined by similarity calculation and selection in the previous steps. In order for the model to better distinguish between two different text sequences, a specific separator is usually inserted between the two sequences. For example, special characters such as [SEP] can be used. Repeat the above splicing process to generate a training instance for each pair of the first text sequence and the second target text sequence. In this way, each instance contains a pair of related text sequences, which is a valid input for the training model. All these spliced ​​text sequences constitute a data set for training.

[0050] Figure 3 FIG. 1 is a flow chart of another method for processing question and answer text data provided by an embodiment of the present disclosure. Figure 3 The above method also includes steps S310 to S330.

[0051] In step S310, a corresponding question text sequence is generated based on each training text sequence through a large language model.

[0052] In step S320, each training text sequence is concatenated with its corresponding question text sequence to obtain a plurality of question-answering training text sequences.

[0053] In step S330, the vector retrieval target of the target version is updated based on each question-answer training text sequence in a supervised manner.

[0054] This method can generate corresponding question text sequences based on each training text sequence through a large language model. Each training text sequence is spliced ​​with its corresponding question text sequence to obtain multiple question-answer training text sequences, and the vector retrieval target of the target version is updated based on each question-answer training text sequence in a supervised manner. In this way, the method of using a language model to generate questions, splice question-answer text sequences, and supervised learning can significantly improve the performance of the vector retrieval model, making it more effective and accurate in various applications.

[0055] In some embodiments of the present disclosure, a corresponding question text sequence is generated based on each training text sequence through a large language model. For example, the training text sequence is "Soak a clean cotton head [MASK] in wax water [SEP] and wax evenly, from the inside to the outside, in a single direction." [MASK] is used to indicate that the model predicts missing vocabulary at this position, and [SEP] is used to separate different parts of the text, indicating the preparation and operation steps before the operation. This text provides specific steps on how to wax, which is applicable to specific floor material processing. Through a large language model, a question text sequence is generated based on the training text sequence. The generated question text sequence is "Floor material, how to wax." The question text sequence is associated with the operation steps in the training text, asking how to wax a specific floor material. Large language models are not only used to understand and generate text, but also can convert text into a machine-processable form through vector representation, greatly enhancing the model's ability to handle specific tasks (such as question answering).

[0056] In some embodiments of the present disclosure, when the number of corresponding question text sequences generated based on each training text sequence by a large language model is multiple, the method further includes: calculating the matching degree between the training text sequence and the corresponding question text sequences; determining at least one target question text sequence corresponding to the training text sequence from the multiple question text sequences according to the matching degrees. For example, a large pre-trained language model such as QWEN, GPT, etc. is used to generate multiple possible question text sequences based on each training text sequence. The generated questions should be diverse, that is, cover different information points in the training text as much as possible, which helps to fully understand and use the text content. First, the training text sequence and the generated question text sequence are converted into a vector form. This is usually done by inputting the text into a pre-trained language model, and the model outputs a vector representation of the text. The matching degree between the training text sequence vector and each question text sequence vector is calculated using a metric such as cosine similarity. According to the calculated matching degree, the relevance of each question text sequence to the training text sequence is evaluated. One or more questions with the highest matching degree are selected from the generated multiple questions as the target question text sequence. This selection is based on the relevance of the question and the ability to lead to useful answers. In some cases, multiple questions with high matching scores are selected, especially when these questions cover the training text from different perspectives. The selected target question text sequences can be used to further train the question answering system, enabling the question answering system to understand and answer more complex and diverse questions. In this way, the accuracy and efficiency of the question answering system can be significantly improved. This approach not only improves the relevance and accuracy of question generation, but also enhances the performance of the final application system by ensuring a high match between the questions and the text content.

[0057] In some embodiments of the present disclosure, each training text sequence is spliced ​​with its corresponding question text sequence to obtain multiple question-answer training text sequences. For example, the question text sequence is appended before or after the corresponding training text sequence. For example, if the question text sequence is "What is the capital?", the training text sequence may be a description of a country, then the spliced ​​question-answer training text sequence may be "What is the capital? [SEP][training text]". In order to help the model better distinguish between questions and training texts, a specific separator, such as "[SEP]", is usually inserted between the two, so that the position where the question ends and the training text begins can be clearly marked.

[0058] In some embodiments of the present disclosure, a vector retrieval target of a target version is updated based on each question-answer training text sequence in a supervised manner. For example, a pre-trained large language model is used to convert the question-answer training text sequence into a vector representation in a high-dimensional space. The vector representation is adjusted according to a supervised learning signal (for example, the correlation between a question and its correct answer) so that the vectors of related texts are closer and the vectors of irrelevant texts are farther apart. A model is trained to learn the similarity measure between text vectors, and commonly used methods include cosine similarity, Euclidean distance, etc. The loss function is configured to optimize the model parameters so that the vector representations of correct question-answer pairs are as close as possible and the vector representations of incorrect question-answer pairs are as separated as possible. Through this supervised approach, the effect of the vector retrieval system in specific question-answering tasks can be significantly improved, enabling it to answer users' queries more accurately, thereby improving user experience and satisfaction.

[0059] In some embodiments of the present disclosure, the method further includes: obtaining a question text sequence currently input by the user; processing the question text sequence currently input by the user through a target version of the vector retrieval model to obtain at least one answer text sequence corresponding to the question text sequence. For example, when a user asks a query (i.e., a question) to the system, it is first necessary to capture and process the question text. This usually involves the following steps: the user inputs a question text sequence through an interface (such as a web interface, a mobile application, or an API). The question text sequence input by the user is standardized, including removing redundant spaces, standardizing punctuation marks, lowercase conversion of text, etc., to ensure the consistency and accuracy of the input format. The processed question text is converted into a vector. This vector should be able to capture the semantic features of the question so that it can be used for subsequent similarity comparisons. The vector representation of the question is input into a vector retrieval model. This model has been trained and can find the most similar answer vector in a pre-built index. The model calculates the similarity between the input question vector and the answer vector stored in the index. This is usually performed using a metric such as cosine similarity. Based on the similarity score, one or more most relevant answer text sequences are retrieved from the index. Typically, the highest-scoring answers are returned. If there are multiple answers, these answers are sorted according to the similarity score, and the answer with the highest score will be shown to the user first. The retrieved answers are returned to the user through a user interface. This can be a simple text list or a more complex interactive interface that allows the user to further interact with the answer or request more information. A user feedback mechanism is provided, where users can rate the relevance and quality of the answer. This feedback can be used to further train and optimize the vector retrieval model. Through this series of steps, the vector retrieval model can effectively process user questions and provide accurate and relevant answers, thereby enhancing the user experience and the practicality of the system.

[0060] Figure 4It is a block diagram of a question and answer text data processing device provided by an embodiment of the present disclosure.

[0061] like Figure 4 As shown, the question and answer text data processing device 400 includes an acquisition module 410, a matching module 420, a splicing module 430 and a training module 440.

[0062] Specifically, the acquisition module 410 is used to acquire document data to be processed and split the document data into multiple first text sequences.

[0063] The matching module 420 is used to retrieve second text sequences matching each first text sequence from a document library based on each first text sequence by using an initial version of the vector retrieval model, wherein the document library contains a plurality of document data to be processed.

[0064] The concatenation module 430 is used to concatenate each first text sequence with its corresponding second text sequence to obtain a plurality of training text sequences.

[0065] The training module 440 is used to train the vector retrieval model of the current version based on each training text sequence in an unsupervised manner in a loop iteration manner to obtain the vector retrieval model of the target version. The vector retrieval model of the current version includes the vector retrieval model of the initial version.

[0066] The question-answer text data processing device 400 can create a richer and more detailed training data set by splitting the document data into multiple first text sequences and splicing these sequences with the matching second text sequences retrieved from the document library. This data splicing method can effectively enhance the model's ability to understand and process document content, thereby achieving higher retrieval accuracy in practical applications. The use of a cyclic iterative training method enables the vector retrieval model to continuously learn from new training text sequences and gradually optimize and adjust its internal parameters. This iterative training method not only improves the model's adaptability to current data, but also enhances the model's ability to process similar data in the future. The target version of the vector retrieval model obtained through iterative training has better performance and can more accurately identify and respond to the query content input by the user. In this way, the retrieved documents will have a higher correlation with the user's query intent, thereby effectively improving user satisfaction and the accuracy of business decisions. Through automated and optimized data processing and model training processes, the need for manual intervention is reduced and the overall retrieval efficiency is improved. At the same time, more accurate retrieval results reduce the time users spend sifting through error information and improve overall work efficiency.

[0067] In some embodiments of the present disclosure, the question and answer text data processing device 400 is also used to: obtain a real-time version of the vector retrieval model during the loop iteration process; retrieve second text sequences matching each first text sequence from the document library based on each first text sequence through the initial version of the vector retrieval model, including: retrieve second text sequences matching each first text sequence from the document library based on each first text sequence through the real-time version of the vector retrieval model.

[0068] In some embodiments of the present disclosure, when the number of second text sequences that match the first text sequence is multiple, the question and answer text data processing device 400 is also used to: calculate the similarity between the first text sequence and each second text sequence; and determine at least one second target text sequence that matches the first text sequence from each second text sequence based on the respective similarities.

[0069] In some embodiments of the present disclosure, the concatenation module 430 is configured to concatenate each first text sequence with its corresponding at least one second target text sequence to obtain a plurality of training text sequences.

[0070] In some embodiments of the present disclosure, the question and answer text data processing device 400 is also used to: generate corresponding question text sequences based on each training text sequence through a large language model; concatenate each training text sequence with its corresponding question text sequence to obtain multiple question and answer training text sequences; and update the vector retrieval target of the target version based on each question and answer training text sequence in a supervised manner.

[0071] In some embodiments of the present disclosure, when a plurality of corresponding question text sequences are generated based on respective training text sequences through a large language model, the question-answer text data processing device 400 is further used to: calculate the matching degree between the training text sequence and the corresponding respective question text sequences; and determine at least one target question text sequence corresponding to the training text sequence from the plurality of question text sequences according to the respective matching degrees.

[0072] In some embodiments of the present disclosure, the question and answer text data processing device 400 is also used to: obtain the question text sequence currently input by the user; process the question text sequence currently input by the user through the vector retrieval model of the target version to obtain at least one answer text sequence corresponding to the question text sequence.

[0073] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules in the above-mentioned device embodiments are implemented.

[0074] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0075] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0076] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0077] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0079] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for processing question and answer text data, characterized in that: include: Acquire document data to be processed, and split the document data into multiple first text sequences; Retrieving, from the document library based on each of the first text sequences, a second text sequence matching each of the first text sequences by using an initial version of the vector retrieval model, wherein the document library contains a plurality of the document data to be processed; splicing each of the first text sequences and its corresponding second text sequence to obtain a plurality of training text sequences; The vector retrieval model of the current version is trained based on each of the training text sequences in an unsupervised manner in a loop iteration manner to obtain the vector retrieval model of the target version, wherein the vector retrieval model of the current version includes the vector retrieval model of the initial version.

2. The method according to claim 1, characterized in that The method further comprises: During the loop iteration process, a real-time version of the vector retrieval model is obtained; Retrieving second text sequences matching each of the first text sequences from the document library based on each of the first text sequences by using the initial version of the vector retrieval model includes: The real-time version of the vector retrieval model is used to retrieve second text sequences matching each of the first text sequences from the document library based on each of the first text sequences.

3. The method according to claim 1, characterized in that When the number of the second text sequences matched by the first text sequence is multiple, the method further includes: Calculating the similarity between the first text sequence and each of the second text sequences; At least one second target text sequence matching the first text sequence is determined from each of the second text sequences according to the respective similarities.

4. The method according to claim 3, characterized in that Each of the first text sequences is concatenated with its corresponding second text sequence to obtain a plurality of training text sequences including: Each of the first text sequences is concatenated with at least one of the second target text sequences corresponding thereto to obtain a plurality of the training text sequences.

5. The method according to claim 1, characterized in that The method further comprises: Generate corresponding question text sequences based on each of the training text sequences using a large language model; Concatenating each of the training text sequences and its corresponding question text sequence to obtain a plurality of question-answering training text sequences; The vector retrieval target of the target version is updated based on each of the question-answer training text sequences in a supervised manner.

6. The method according to claim 5, characterized in that When the number of corresponding question text sequences generated by the large language model based on each of the training text sequences is multiple, the method further includes: Calculating the matching degree between the training text sequence and each corresponding question text sequence; At least one target question text sequence corresponding to the training text sequence is determined from the plurality of question text sequences according to the respective matching degrees.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Get the question text sequence currently input by the user; The question text sequence currently input by the user is processed by the vector retrieval model of the target version to obtain at least one answer text sequence corresponding to the question text sequence.

8. A question and answer text data processing device, characterized in that: include: An acquisition module, used for acquiring document data to be processed, and splitting the document data into a plurality of first text sequences; A matching module, configured to retrieve, from the document library, a second text sequence matching each of the first text sequences based on each of the first text sequences by using an initial version of a vector retrieval model, wherein the document library contains a plurality of the document data to be processed; A concatenation module, used for concatenating each of the first text sequences with its corresponding second text sequence to obtain a plurality of training text sequences; The training module is used to train the vector retrieval model of the current version based on each of the training text sequences in an unsupervised manner in a cyclic iteration manner to obtain the vector retrieval model of the target version, wherein the vector retrieval model of the current version includes the vector retrieval model of the initial version.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images