Text retrieval enhancement generation method, device and equipment and readable storage medium

Abstract

The invention provides a text retrieval enhancement generation method and device, equipment and a readable storage medium, and the method comprises the steps: obtaining document original data, and storing the document original data to a first database; performing text vectorization processing on the document original data to obtain document vector data, and storing the document vector data to a second database; retrieving a plurality of document related data from the first database and the second database based on question information input by a user; and generating a target keyword based on the related data of each document, and analyzing the target keyword through a large model to obtain a reply text for the question information. By adopting the method, the normal generation of the reply text is ensured, and the reliability, effectiveness and accuracy of the generation of the reply text are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing technology, and in particular to a text retrieval enhancement generation method, device, equipment and readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence, large language models (LLMs) have gradually become an indispensable technical pillar in all walks of life. They have shown great application potential in natural language processing, text generation, question-answering systems, etc. In the existing technology, large language models are usually used to parse questions raised by users, and answers that match the questions are generated by the built-in database of the large model.

[0003] However, the study found that since the data stored in the built-in database of the big model is limited, it is impossible to achieve comprehensive coverage in terms of breadth or depth. As a result, when users use the big model to ask questions, they may not be able to retrieve and generate reply texts that match the user's questions, and the normal generation of reply texts cannot be ensured, which reduces the reliability, effectiveness and accuracy of reply text generation. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a text retrieval enhanced generation method, device, equipment and readable storage medium to ensure the normal generation of reply text and ensure the reliability, effectiveness and accuracy of the reply text generation.

[0005] In a first aspect, an embodiment of the present application provides a text retrieval enhancement generation method, the method comprising:

[0006] Acquire original document data, and store the original document data in a first database;

[0007] Performing text vectorization processing on the original document data to obtain document vector data, and storing the document vector data in a second database;

[0008] Retrieving a number of document-related data from the first database and the second database respectively based on the question information input by the user;

[0009] Target keywords are generated based on relevant data of each document, and the target keywords are parsed through a large model to obtain a reply text for the question information.

[0010] Optionally, performing text vectorization processing on the original document data to obtain document vector data includes:

[0011] Converting the original document data into a document string;

[0012] Segment the document string based on a preset delimiter to obtain a plurality of discrete strings;

[0013] Each discrete character string is mapped to a vector space through an embedding model to obtain the document vector data.

[0014] Optionally, the embedded model includes an M3E model, a GTE model and a BGE model.

[0015] Optionally, the retrieving a number of document-related data from the first database and the second database based on the question information input by the user respectively includes:

[0016] Perform keyword analysis on the question information to obtain a number of question keywords, and perform keyword search on the original document data in the first database according to the question keywords to obtain first document related data;

[0017] The question information is mapped to a vector space to obtain question vector data, and a semantic search is performed on the document vector data in the second database according to the question vector data to obtain second document related data.

[0018] Optionally, performing semantic retrieval on the document vector data in the second database according to the question vector data to obtain the second document related data includes:

[0019] Calculating the similarity between the question vector data and each document vector data in the second database respectively;

[0020] The second document related data is screened out from the second database according to the similarity between the question vector data and each document vector data in the second database.

[0021] Optionally, generating target keywords based on the relevant data of each document includes:

[0022] Calculate the correlation between the relevant data of each document and the question information;

[0023] Filtering out a preset number of target related data with the highest relevance to the question information from each document related data;

[0024] The target keywords are obtained by fusing the target-related data.

[0025] Optionally, the first database is an ElasticSearch database, and the second database is a Milvus vector database.

[0026] In a second aspect, an embodiment of the present application provides a text retrieval enhancement generation device, the device comprising:

[0027] A first database storage module, used to obtain original document data and store the original document data in a first database;

[0028] A second database storage module is used to perform text vectorization processing on the original document data to obtain document vector data, and store the document vector data in a second database;

[0029] A text retrieval module, used to retrieve a number of document-related data from the first database and the second database respectively based on the question information input by the user;

[0030] The reply text generation module is used to generate target keywords based on the relevant data of each document, and parse the target keywords through a large model to obtain a reply text for the question information.

[0031] Optionally, performing text vectorization processing on the original document data to obtain document vector data includes:

[0032] Converting the original document data into a document string;

[0033] Segment the document string based on a preset delimiter to obtain a plurality of discrete strings;

[0034] Each discrete character string is mapped to a vector space through an embedding model to obtain the document vector data.

[0035] Optionally, the embedded model includes an M3E model, a GTE model and a BGE model.

[0036] Optionally, the retrieving a number of document-related data from the first database and the second database based on the question information input by the user respectively includes:

[0037] Perform keyword analysis on the question information to obtain a number of question keywords, and perform keyword search on the original document data in the first database according to the question keywords to obtain first document related data;

[0038] The question information is mapped to a vector space to obtain question vector data, and a semantic search is performed on the document vector data in the second database according to the question vector data to obtain second document related data.

[0039] Optionally, performing semantic retrieval on the document vector data in the second database according to the question vector data to obtain the second document related data includes:

[0040] Calculating the similarity between the question vector data and each document vector data in the second database respectively;

[0041] The second document related data is screened out from the second database according to the similarity between the question vector data and each document vector data in the second database.

[0042] Optionally, generating target keywords based on the relevant data of each document includes:

[0043] Calculate the correlation between the relevant data of each document and the question information;

[0044] Filtering out a preset number of target related data with the highest relevance to the question information from each document related data;

[0045] The target keywords are obtained by fusing the target-related data.

[0046] Optionally, the first database is an ElasticSearch database, and the second database is a Milvus vector database.

[0047] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the text retrieval enhancement generation method described in any optional implementation manner of the first aspect are performed.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the text retrieval enhancement generation method described in any optional implementation manner of the first aspect above.

[0049] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0050] This application obtains the original document data, stores the original document data in the first database, performs text vectorization processing on the original document data to obtain document vector data, and stores the document vector data in the second database. It can provide strong support for subsequent text retrieval based on the dual databases, thereby improving the hit rate and accuracy of the retrieval. Then, based on the question information input by the user, several document-related data are retrieved from the first database and the second database respectively, and effective document-related data can be formed with the strong support of the dual databases; finally, target keywords are generated based on each document-related data, and the target keywords are parsed through the large model to obtain the reply text for the question information. It can combine the large model's own database with the document data uploaded by the user to generate an effective reply text, thereby ensuring the normal generation of the reply text, and ensuring the reliability, effectiveness and accuracy of the reply text generation.

[0051] In addition, this application introduces three cutting-edge embedding models, M3E, GTE, and BGE, so that users can flexibly choose based on their own needs and preferences to achieve the best embedding effect. This design not only improves the efficiency of retrieval enhancement generation, but also ensures the accuracy and relevance of the output results. This application also innovatively combines the Milvus vector database with the ElasticSearch search engine, combines keyword retrieval and semantic retrieval, and significantly improves the hit rate of retrieval. In addition, the retrieved document collection is sorted by relevance and the best is taken. This groundbreaking design enables the performance of the present invention on open source retrieval data to surpass the existing most advanced models, ensuring the high accuracy and efficiency of the retrieval results. At the same time, users can easily import private document data and perform questions and answers based on its content without having to deeply understand the underlying implementation of the large model, achieving seamless docking with the enterprise data warehouse, which not only simplifies the deployment process, but also strengthens the security of enterprise data, and achieves the dual advantages of data protection and intelligent question and answer. This application supports multiple common document formats including PDF, JPG, DOC, TXT, XLS, and PPT. Through advanced parsing technology, this application can deeply understand and answer text and table content, allowing the large model to easily expand its knowledge boundaries like in an "open-book exam", thereby further improving the reliability, effectiveness, accuracy, and comprehensiveness of the generated response text.

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of a text retrieval enhancement generation method provided by Embodiment 1 of the present invention is shown;

[0055] Figure 2 A flowchart of a method for generating document vector data provided by the first embodiment of the present invention is shown;

[0056] Figure 3 A flowchart of a document-related data retrieval method provided by Embodiment 1 of the present invention is shown;

[0057] Figure 4A flow chart of a method for retrieving data related to a second document provided in the first embodiment of the present invention is shown;

[0058] Figure 5 A flow chart of a method for generating target keywords provided by the first embodiment of the present invention is shown;

[0059] Figure 6 A schematic diagram of a text retrieval enhancement generation platform provided in the first embodiment of the present invention is shown;

[0060] Figure 7 A schematic diagram of a knowledge storage process provided by the first embodiment of the present invention is shown;

[0061] Figure 8 A schematic diagram showing an enhanced search generation process provided by the first embodiment of the present invention is shown;

[0062] Fig. 9 A schematic diagram showing the structure of a text retrieval enhancement generation device provided in the second embodiment of the present invention is shown;

[0063] Fig.10 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0065] Embodiment 1

[0066] To facilitate understanding of this application, Figure 1 The flowchart of a text retrieval enhancement generation method provided in the first embodiment of the present invention is shown to describe the content of the first embodiment of the present application in detail.

[0067] See also Figure 1 As shown, Figure 1 A flowchart of a text retrieval enhancement generation method provided in the first embodiment of the present invention is shown, wherein the method comprises steps S101 to S104:

[0068] S101: Acquire original document data, and store the original document data in a first database.

[0069] Specifically, in order to improve the flexibility and diversity of text retrieval, this application provides an innovative dual database storage strategy. First, the original document data is acquired through a third-party system platform or uploaded and input by users, and the original document data is stored in the first database to form a knowledge base, providing data semantic retrieval support for subsequent text retrieval.

[0070] S102: Perform text vectorization processing on the original document data to obtain document vector data, and store the document vector data in a second database.

[0071] Specifically, the original document data is discretely segmented and then mapped to the vector space to obtain document vector data, and the processed vector data is stored in the second database to provide keyword retrieval support for subsequent text retrieval. This strategy not only ensures fast data retrieval, but also provides strong support for complex queries and analysis.

[0072] S103: Retrieving a number of document-related data from the first database and the second database respectively based on the question information input by the user.

[0073] Specifically, the user asks questions through natural language, and the first database is searched based on the question information input by the user, and the second database is searched. The search results of these two databases are saved as document-related data, laying the foundation for subsequent retrieval and generation processes.

[0074] S104: Generate target keywords based on relevant data of each document, and parse the target keywords through a large model to obtain a reply text for the question information.

[0075] Specifically, the correlation between each document-related data and the question information is calculated, and several document-related data are screened out according to the size of the correlation to form the target keywords. Then the target keywords are input into a powerful big model. The big model uses its deep learning ability to parse the information in the target keywords and generate accurate, comprehensive and insightful reply texts to the user's question information.

[0076] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flowchart of a method for generating document vector data provided by the first embodiment of the present invention is shown, wherein the process of performing text vectorization processing on the original document data to obtain the document vector data comprises steps S201 to S203:

[0077] S201: Convert the original document data into a document character string.

[0078] Specifically, it uses file loading technology and integrates various Python dependency libraries such as PDFLoader, TextLoader, and CSVLoader to support multiple file formats including PDF, text, and CSV. These libraries work together to seamlessly convert the original document data uploaded by users into a unified string format, laying a solid foundation for subsequent processing.

[0079] S202: Segment the document string based on a preset separator to obtain a plurality of discrete strings.

[0080] Specifically, the document string obtained in step S201 is accurately segmented into a number of discrete strings using the user-defined delimiter as the segmentation standard and the built-in split function of Python, so as to optimize the readability and retrieval efficiency of the document.

[0081] S203: Map each discrete character string to a vector space through an embedding model to obtain the document vector data.

[0082] Specifically, through embedding model technology, discrete text strings are mapped into a continuous vector space to obtain document vector data, achieving deeper semantic understanding and information retrieval.

[0083] In an optional embodiment, the embedded model includes an M3E model, a GTE model and a BGE model.

[0084] Specifically, three embedding models are provided, namely, M3E model, GTE model, and BGE model. Each of them has unique advantages and characteristics. In this application, they are selected according to the actual needs and preferences of the user to obtain the best embedding effect.

[0085] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a document-related data retrieval method provided in Embodiment 1 of the present invention is shown, wherein the method retrieves a plurality of document-related data from the first database and the second database based on the question information input by the user, including steps S301 to S302:

[0086] S301: performing keyword analysis on the question information to obtain a number of question keywords, and performing keyword search on the original document data in the first database according to the question keywords to obtain first document related data.

[0087] Specifically, the first database is an ElasticSearch database, and keyword search is performed on the first database. First, keyword analysis is performed on the question information to obtain a number of question keywords. The full-text search capability of ElasticSearch is used to construct a complex query grammar to perform keyword search on the user's question information to obtain the first document related data. Including but not limited to: Boolean query to identify documents containing all or part of the keywords; phrase query to find document fragments that accurately match the user's query phrase; fuzzy query to capture spelling errors or synonyms. This multi-dimensional keyword matching strategy ensures the accuracy and relevance of the search results.

[0088] S302: Map the question information to a vector space to obtain question vector data, and perform semantic retrieval on the document vector data in the second database according to the question vector data to obtain second document related data.

[0089] Specifically, the second database is the Milvus vector database, and semantic retrieval is performed on the second database. Using the semantic search capability of the Milvus vector database, the user's question information is converted into points in a high-dimensional vector space to form question vector data. By calculating the similarity between the question vector data and the document vector data in the second database, Milvus can retrieve the second document related data that is closely related to the user's question information at the semantic level. This method is particularly suitable for processing queries where the keywords do not completely match but have similar meanings in the context.

[0090] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flowchart of a second document related data retrieval method provided in the first embodiment of the present invention is shown, wherein the semantic retrieval of the document vector data in the second database according to the question vector data to obtain the second document related data includes steps S401 to S402:

[0091] S401: Calculate the similarity between the question vector data and each document vector data in the second database respectively.

[0092] Specifically, the similarity includes but is not limited to cosine similarity and Euclidean distance.

[0093] S402: Filtering out the second document related data from the second database according to the similarity between the question vector data and each document vector data in the second database.

[0094] Specifically, the document vector data in the second database whose similarity with the question vector data exceeds a preset threshold or which ranks in a preset top position is selected as the second document related data.

[0095] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flow chart of a method for generating target keywords provided by the first embodiment of the present invention is shown, wherein the method for generating target keywords based on the relevant data of each document includes steps S501 to S503:

[0096] S501: Calculate the correlation between each document-related data and the question information.

[0097] Specifically, the Rerank service of the BGE model is used to calculate the relevance between the relevant data of each document and the user's question information, and the search result set is sorted from high to low according to the relevance. BGE is a text similarity calculation model based on deep learning, and its ReRank module can calculate text similarity.

[0098] S502: Filter out a preset number of target-related data with the highest relevance to the question information from each document-related data.

[0099] Specifically, a preset number of document-related data with the highest relevance are selected as target-related data.

[0100] S503: Fusing the target-related data to obtain the target keyword.

[0101] Specifically, each target-related data is deeply integrated, that is, each target-related data is sequentially spliced ​​into a target keyword, which provides a solid foundation for generating accurate answers.

[0102] In an optional implementation manner, the first database is an ElasticSearch database, and the second database is a Milvus vector database.

[0103] Specifically, the combination of Milvus and ElasticSearch databases can provide a comprehensive search strategy that can not only use ES's keyword matching capabilities to quickly find documents containing specific keywords, but also use Milvus's semantic similarity search to find documents that are semantically related to the query but do not completely match the keywords. This two-pronged strategy not only improves the diversity and comprehensiveness of search results, but also better meets the needs of different types of queries, thereby providing users with richer and more accurate search results.

[0104] The text retrieval enhancement generation method provided in this application can be applied to multiple application scenarios, and the text retrieval enhancement generation platform constructed can realize the generation of text retrieval enhancement in various application scenarios. Figure 6 As shown, Figure 6A schematic diagram of a text retrieval enhancement generation platform provided in Example 1 of the present invention is shown, wherein application scenarios include agent assistant, intelligent customer service, office assistant, document question and answer, and knowledge search, and the platform architecture is divided into the following levels, including basic model, service component, service orchestration, and service scenario.

[0105] The basic model layer is centered on the general large model base, and uses its powerful language generation capabilities to organize, expand and optimize the final answer. The service component layer includes modules such as reading comprehension, dialogue question and answer, and text tracing, supporting a variety of text processing functions.

[0106] The service component layer includes components such as reading comprehension, conversational question and answer, and text tracing. Users can select components according to their needs to implement their corresponding service functions.

[0107] The service orchestration layer consists of two parts: knowledge storage and retrieval enhancement. Figure 7 As shown, Figure 7 The schematic diagram of a knowledge storage process provided by the first embodiment of the present invention is shown, wherein the knowledge storage includes two steps: file parsing and text segmentation. First, multi-format files are loaded, then the text is segmented by custom, then word vectors are embedded, and finally the text vectors are stored to ensure the systematic and structured storage of information. Figure 8 As shown, Figure 8 A schematic diagram of an enhanced search generation process provided by the first embodiment of the present invention is shown, wherein the search enhancement includes multiple steps such as search planning, content optimization, distributed search and content traceability, and sequentially receives questions, embeds word vectors, searches ES (ElasticSearch database), searches Milvus data, performs relevance sorting, integrates prompt keywords and LLM large language model question and answer, aiming to improve the search efficiency and the relevance of the results.

[0108] The service scenario layer is dedicated to retrieving and enhancing large models to ensure that they can be applied to a variety of practical application scenarios, such as agent assistants, intelligent customer service, office assistants, document Q&A, and knowledge search, significantly improving users' interactive experience and work efficiency.

[0109] The text retrieval enhancement generation platform integrates multiple word embedding technology models to enhance the ability to represent original keywords and the expression of the relationship between words. At the same time, by adopting a distributed storage and retrieval mechanism, the flexibility and comprehensiveness of retrieval are improved, thereby significantly improving the generation effect and efficiency of large models when processing private knowledge bases. These innovations not only break through the limitations of traditional models, but also provide strong technical support for enterprises on the road to intelligent and data-driven decision-making, accelerating the pace of digital and intelligent transformation of enterprises.

[0110] Embodiment 2

[0111] See also Fig. 9 As shown, Fig. 9 A schematic diagram of the structure of a text retrieval enhancement generation device provided in the second embodiment of the present invention is shown, wherein the device comprises:

[0112] The first database storage module 901 is used to obtain original document data and store the original document data in a first database;

[0113] A second database storage module 902 is used to perform text vectorization processing on the original document data to obtain document vector data, and store the document vector data in a second database;

[0114] A text retrieval module 903, configured to retrieve a number of document-related data from the first database and the second database respectively based on the question information input by the user;

[0115] The reply text generation module 904 is used to generate target keywords based on the relevant data of each document, and parse the target keywords through a large model to obtain a reply text for the question information.

[0116] In an optional implementation, the performing text vectorization processing on the original document data to obtain document vector data includes:

[0117] Converting the original document data into a document string;

[0118] Segment the document string based on a preset delimiter to obtain a plurality of discrete strings;

[0119] Each discrete character string is mapped to a vector space through an embedding model to obtain the document vector data.

[0120] In an optional embodiment, the embedded model includes an M3E model, a GTE model and a BGE model.

[0121] In an optional implementation manner, the step of retrieving a number of document-related data from the first database and the second database based on the question information input by the user includes:

[0122] Perform keyword analysis on the question information to obtain a number of question keywords, and perform keyword search on the original document data in the first database according to the question keywords to obtain first document related data;

[0123] The question information is mapped to a vector space to obtain question vector data, and a semantic search is performed on the document vector data in the second database according to the question vector data to obtain second document related data.

[0124] In an optional implementation manner, performing semantic retrieval on the document vector data in the second database according to the question vector data to obtain the second document related data includes:

[0125] Calculating the similarity between the question vector data and each document vector data in the second database respectively;

[0126] The second document related data is screened out from the second database according to the similarity between the question vector data and each document vector data in the second database.

[0127] In an optional implementation, generating target keywords based on the relevant data of each document includes:

[0128] Calculate the correlation between the relevant data of each document and the question information;

[0129] Filtering out a preset number of target related data with the highest relevance to the question information from each document related data;

[0130] The target keywords are obtained by fusing the target-related data.

[0131] In an optional implementation manner, the first database is an ElasticSearch database, and the second database is a Milvus vector database.

[0132] Embodiment 3

[0133] Based on the same application concept, see Fig.10 As shown, Fig.10 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Fig.10 As shown, a computer device 1000 provided in Embodiment 3 of the present application includes:

[0134] A processor 1001, a memory 1002 and a bus 1003, wherein the memory 1002 stores machine-readable instructions executable by the processor 1001. When the computer device 1000 is running, the processor 1001 communicates with the memory 1002 via the bus 1003. When the processor 1001 is running, the machine-readable instructions execute the steps of the text retrieval enhancement generation method shown in the above-mentioned embodiment 1.

[0135] Embodiment 4

[0136] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the text retrieval enhancement generation method described in any one of the above embodiments are executed.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0138] The computer program product for text retrieval enhancement generation provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0139] The text retrieval enhancement generation device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0140] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0143] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0144] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0145] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A text retrieval enhancement generation method, characterized in that: The method comprises: Acquire original document data, and store the original document data in a first database; Performing text vectorization processing on the original document data to obtain document vector data, and storing the document vector data in a second database; Retrieving a number of document-related data from the first database and the second database respectively based on the question information input by the user; Target keywords are generated based on relevant data of each document, and the target keywords are parsed through a large model to obtain a reply text for the question information.

2. The method according to claim 1, characterized in that The step of performing text vectorization processing on the original document data to obtain document vector data includes: Converting the original document data into a document string; Segment the document string based on a preset delimiter to obtain a plurality of discrete strings; Each discrete character string is mapped to a vector space through an embedding model to obtain the document vector data.

3. The method according to claim 2, characterized in that The embedded models include M3E model, GTE model and BGE model.

4. The method according to claim 1, characterized in that: The step of retrieving a number of document-related data from the first database and the second database based on the question information input by the user includes: Perform keyword analysis on the question information to obtain a number of question keywords, and perform keyword search on the original document data in the first database according to the question keywords to obtain first document related data; The question information is mapped to a vector space to obtain question vector data, and a semantic search is performed on the document vector data in the second database according to the question vector data to obtain second document related data.

5. The method according to claim 4, characterized in that The performing semantic retrieval on the document vector data in the second database according to the question vector data to obtain the second document related data includes: Calculating the similarity between the question vector data and each document vector data in the second database respectively; The second document related data is screened out from the second database according to the similarity between the question vector data and each document vector data in the second database.

6. The method according to claim 1, characterized in that The generating target keywords based on the relevant data of each document includes: Calculate the correlation between the relevant data of each document and the question information; Filtering out a preset number of target related data with the highest relevance to the question information from each document related data; The target keywords are obtained by fusing the target-related data.

7. The method according to claim 1, characterized in that The first database is an ElasticSearch database, and the second database is a Milvus vector database.

8. A text retrieval enhancement generation device, characterized in that: The device comprises: A first database storage module, used to obtain original document data and store the original document data in a first database; A second database storage module is used to perform text vectorization processing on the original document data to obtain document vector data, and store the document vector data in a second database; A text retrieval module, used to retrieve a number of document-related data from the first database and the second database respectively based on the question information input by the user; The reply text generation module is used to generate target keywords based on the relevant data of each document, and parse the target keywords through a large model to obtain a reply text for the question information.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the text retrieval enhancement generation method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the text retrieval enhancement generation method as claimed in any one of claims 1 to 7.

Images