Retrieval enhancement generation method, device and equipment based on LSH and storage medium thereof

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an LSH-based retrieval enhancement generation method and device, equipment and a storage medium thereof. Carrying out serialization processing; sampling processing is carried out through the vector space model, and high-dimensional embedded vector representation is generated; inputting the high-dimensional embedded vector representation into a vector database, and carrying out matching calculation; carrying out binary code valuation conversion on a matching calculation result by adopting an LSH algorithm to obtain binary code values corresponding to all the retrieved texts after conversion; according to the binary code value, an expected retrieved text is screened out; combining and sorting the retrieval input text and the expected retrieved text to generate a to-be-enhanced text; and inputting the text to be enhanced into the text enhancement generator to generate an enhanced retrieval feedback text. The method is applied to webpage search, question and answer systems and recommendation system scenes in the field of financial services or medical services, and a faster and more accurate retrieval feedback result is provided for retrieval personnel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, applied to the text retrieval scene, and relates to an LSH-based retrieval enhancement generation method, device, equipment and storage medium thereof. BACKGROUND

[0002] Text embedding is a core component of many natural language processing and information retrieval applications, and is widely used in web search, question answering systems, recommendation systems and other scenarios. In particular, in the web search, question answering systems, recommendation systems and other scenarios corresponding to the financial business or medical business field with a large amount of data, high-quality embedding can help the model capture the semantic relationship between texts. Although the emergence of large language models has significantly promoted the development of text embedding models, from early methods based on BERT and other encoder pre-training models to new architectures that utilize the rich knowledge and reasoning capabilities of LLMs.

[0003] However, in emerging retrieval augmentation generation (RAG) and agent system application paradigms, especially in text retrieval of RAG systems, since the three steps of retrieval, augmentation and generation need to be processed, the retrieval efficiency and accuracy are poor when the data content in the text corpus is complex. SUMMARY

[0004] The purpose of the embodiments of the present application is to propose an LSH-based retrieval augmentation generation method, device, equipment and storage medium to solve the problem of poor retrieval efficiency and accuracy in text retrieval of RAG systems.

[0005] In a first aspect, the embodiments of the present application provide an LSH-based retrieval augmentation generation method, which adopts the technical solution as follows:

[0006] An LSH-based retrieval augmentation generation method includes the following steps:

[0007] Obtain a retrieval input text;

[0008] Serializing the retrieval input text to obtain a serialized input text;

[0009] Sampling the serialized input text by a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text;

[0010] Input the high-dimensional embedding vector representation into a preset vector database, and perform matching calculation with the high-dimensional vector representation corresponding to each piece of retrieved text in the vector database;

[0011] The LSH algorithm is used for binary code value conversion of the matching calculation result, to obtain binary code values corresponding to all the searched texts after conversion, wherein the LSH algorithm includes a local sensitive hash algorithm.

[0012] According to the binary code values, the searched text expected by the search input text is screened out.

[0013] The search input text and the searched text expected by the search input text are merged and arranged to generate a to-be-enhanced processing text.

[0014] The to-be-enhanced processing text is input into a preset text enhancement generator to generate an enhanced search feedback text, wherein the preset text enhancement generator includes an LLM-based text enhancement generator.

[0015] In a second aspect, the embodiments of the present application further provide a search enhancement generation device based on LSH, which adopts the technical scheme as follows:

[0016] A search enhancement generation device based on LSH, comprising:

[0017] A search input text acquisition module is configured to acquire a search input text.

[0018] A serialization processing module is configured to perform serialization processing on the search input text to obtain a serialized input text.

[0019] A high-dimensional embedding vector representation generation module is configured to perform sampling processing on the serialized input text by using a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text.

[0020] A matching calculation module is configured to input the high-dimensional embedding vector representation into a preset vector database to perform matching calculation on a high-dimensional vector representation corresponding to each searched text in the vector database.

[0021] An LSH algorithm processing module is configured to use an LSH algorithm to perform binary code value conversion on the matching calculation result to obtain binary code values corresponding to all the searched texts after conversion, wherein the LSH algorithm includes a local sensitive hash algorithm.

[0022] An expected search result screening module is configured to screen out the searched text expected by the search input text according to the binary code values.

[0023] A to-be-enhanced processing text generation module is configured to merge and arrange the search input text and the searched text expected by the search input text to generate a to-be-enhanced processing text.

[0024] The retrieval feedback text generation module is configured to input the to-be-enhanced processing text into a preset text enhancement generator to generate an enhanced retrieval feedback text.

[0025] In a third aspect, the embodiments of the present application further provide a computer device, which adopts the technical solutions described as follows:

[0026] A computer device includes a memory and a processor, the memory stores computer readable instructions, and the processor implements the steps of the LSH-based retrieval enhancement generation method described above when executing the computer readable instructions.

[0027] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which adopts the technical solutions described as follows:

[0028] A computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the LSH-based retrieval enhancement generation method described above.

[0029] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0030] The LSH-based retrieval enhancement generation method described in the present application acquires a retrieval input text, performs serialization processing, generates a high-dimensional embedding vector representation through vector space model sampling processing, inputs the high-dimensional embedding vector representation into a vector database to perform matching calculation, converts the matching calculation result into binary code values by using an LSH algorithm to obtain binary code values corresponding to all the searched texts after conversion, filters out the desired searched text according to the binary code values, merges and arranges the retrieval input text and the desired searched text to generate a to-be-enhanced processing text, inputs the to-be-enhanced processing text into a text enhancement generator to generate an enhanced retrieval feedback text. In essence, the LSH algorithm is applied to the RAG system, the RAG system has a retriever and an enhancement generator, the LSH algorithm is integrated into the retriever of the RAG system, so that the text-to-text retrieval of the retrieval input text can be realized when the retriever performs retrieval processing, the retrieval is converted into vector-to-vector retrieval, and then converted into binary code value processing, which is more in line with the processing specification of the computer, and the retrieval result can be obtained more quickly and accurately. Finally, the retrieval result is input into the enhancement generator for enhancement processing to obtain the final retrieval feedback text, which is fed back to the retrieval initiator. The method is applied to the web search, question and answer system, recommendation system and other scenes in the field of financial business or medical business, so as to provide more quickly and accurately retrieval feedback results for the retrieval personnel. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the solutions in the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0032] Figure 1 is an exemplary system architecture diagram in which the present application can be applied;

[0033] Figure 2 is a flow chart of one embodiment of a LSH-based retrieval enhancement generation method according to the present application;

[0034] Figure 3 is a flow chart of one specific embodiment of cleaning processing of a serialized input text in the LSH-based retrieval enhancement generation method described in the present application;

[0035] Figure 4 is a flow chart of one specific embodiment of step 203 shown in Figure 2 ;

[0036] Figure 5 is a flow chart of one specific embodiment of preparing a vector database in the LSH-based retrieval enhancement generation method described in the present application;

[0037] Figure 6 is a flow chart of one specific embodiment of step 204 shown in Figure 2 ;

[0038] Figure 7 is a flow chart of one specific embodiment of step 205 shown in Figure 2 ;

[0039] Figure 8 is a flow chart of one specific embodiment of step 206 shown in Figure 2 ;

[0040] Figure 9 is a structural schematic diagram of one embodiment of a LSH-based retrieval enhancement generation device according to the present application;

[0041] Figure 10 is a structural schematic diagram of one embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terms used in the specification are intended to describe the particular embodiments and are not intended to limit the application; the terms "include" and "have" and any variations thereof used in the specification and the claims and the above description of the drawings are intended to cover the non-exclusive inclusion; the terms "first", "second" and the like used in the specification and the claims and the above description of the drawings are intended to distinguish different objects, not to describe a particular order.

[0043] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive or alternative embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with each other.

[0044] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0045] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0046] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0047] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.

[0048] The server 103 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101.

[0049] It should be noted that the LSH-based retrieval enhancement generation method provided in the embodiments of the present application is generally executed by a server, and accordingly, the LSH-based retrieval enhancement generation apparatus is generally arranged in the server.

[0050] It should be understood that, Figure 1 The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0051] With reference to Figure 2 , a flow chart of one embodiment of the LSH-based retrieval enhancement generation method according to the present application is shown. The LSH-based retrieval enhancement generation method comprises the following steps:

[0052] Step 201, obtaining a retrieval input text.

[0053] In the embodiment, the retrieval input text includes a business consultation text input by a user through a business consultation window, and also includes a script search text provided by an intelligent customer service for answering a question of the user;

[0054] The LSH-based retrieval enhancement generation method is applied to web search, question and answer system, recommendation system and other scenarios, so as to provide more accurate retrieval feedback results for retrieval personnel. Specifically, for example, in the intelligent customer service system in the financial business or medical business scenario, more accurate script answer texts can be retrieved for the retrieval input text provided by the intelligent customer service.

[0055] Step 202, performing a serialization processing on the retrieval input text to obtain a serialized input text.

[0056] Specifically, based on the position information of each character in the search input text, the search input text is serialized to obtain serialized input text.

[0057] Step 203: The serialized input text is sampled using a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text.

[0058] Specifically, taking insurance business in financial business as an example, the preset vector space model assumes that the search input text is a text for consulting about insurance business. Here, the preset vector space model contains vectorized representations of various professional terms, proper nouns, and characters in the insurance industry.

[0059] The serialized input text is sampled using a preset vector space model to obtain the search terms or search characters related to specific business contained in the serialized input text. Then, based on the vector representations corresponding to the search terms or search characters in the vector space model, a high-dimensional embedding vector representation of the serialized input text is generated.

[0060] Step 204: Input the high-dimensional embedded vector representation into a preset vector database and perform matching calculations with the high-dimensional vector representation corresponding to each retrieved text in the vector database.

[0061] In this embodiment, the preset vector database is the vector database corresponding to the text corpus database corresponding to the input text, which realizes the transformation of text-to-text retrieval into vector-to-vector retrieval, thereby improving the retrieval efficiency to a certain extent.

[0062] Step 205: The LSH algorithm is used to convert the matching calculation results into binary code values ​​to obtain the binary code values ​​corresponding to all the retrieved texts after conversion. The LSH algorithm includes the Locality Sensitive Hash algorithm.

[0063] By employing the LSH (Local Sensitive Hash) algorithm, the retrieval and comparison of vectors is further transformed into a comparison of binary code values, providing a new comparison method for vector similarity measurement in text retrieval. This method is more in line with the underlying encoding and processing mode of computers and further improves retrieval efficiency based on step 204.

[0064] Step 206: Based on the binary code value, filter out the expected search text of the search input text.

[0065] Specifically, the desired search results are filtered directly using the binary code value generated in step 205.

[0066] Step 207: Merge and organize the search input text and the expected searched text to generate the text to be enhanced.

[0067] Step 208: Input the text to be enhanced into a preset text enhancement generator to generate enhanced search feedback text, wherein the preset text enhancement generator includes an LLM-based text enhancement generator.

[0068] Specifically, a text enhancement generator based on LLM (Large Language Model) is used to enhance the text to be enhanced, so that the final search feedback text better meets the search requirements of the search terminal.

[0069] This embodiment essentially applies LSH (Locality Sensitive Hash) to the RAG (Retrieval Enhancement Generation) system. The RAG system has two parts: a retriever and an enhancement generator. By integrating the LSH algorithm into the retriever of the RAG system, the retriever can obtain search results more quickly and accurately based on the search input text during the search process. Then, the search results are input into the enhancement generator for enhancement processing, and the enhanced output is fed back to the original search initiator.

[0070] In this embodiment, the following steps are taken: First, the input text is acquired and serialized. Then, a high-dimensional embedding vector representation is generated through sampling using a vector space model. This high-dimensional embedding vector representation is input into a vector database for matching calculations. Next, the LSH algorithm is used to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts. Based on these binary code values, the desired retrieved text is selected. The input text and the desired retrieved text are then merged and organized to generate the text to be enhanced. Finally, the text to be enhanced is input into a text enhancement generator to generate the enhanced search feedback text. Essentially, the LSH algorithm is applied to the RAG system. The RAG system has two parts: a search engine and an enhancement generator. By integrating the LSH algorithm into the RAG system's search engine, the search results can be obtained more quickly and accurately based on the input text during the search engine's retrieval process. The search results are then input into the enhancement generator for enhancement processing, resulting in the final search feedback text fed back to the search initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0071] Continue to refer to Figure 3 In some specific embodiments, a step of cleaning the serialized input text is included before step 203. Figure 3This is a flowchart of a specific embodiment of the LSH-based retrieval enhancement generation method described in this application, which involves cleaning the serialized input text, including:

[0072] Step 301: Use a preset cleaning component to clean the serialized input text of meaningless characters or words;

[0073] Specifically, when acquiring the search input text and serializing it, since the search input text may be entered by the user and may contain certain meaningless characters or words, such as greeting words, tone words or characters, a preset cleaning component is used to clean the serialized input text of meaningless characters or words to obtain a simplified serialized input text to be vectorized.

[0074] Step 302: Obtain the serialized input text after cleaning up meaningless characters or words, and update it to the serialized input text to be vectorized.

[0075] By updating the serialized input text after cleaning up meaningless words or phrases to the serialized input text to be vectorized, the interference of meaningless words or phrases on the retrieval can be avoided, thereby improving retrieval efficiency and accuracy.

[0076] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes:

[0077] Step 401: According to the word segmentation strategy in the vector space model, the serialized input text to be vectorized is segmented by word sampling.

[0078] Specifically, assuming the serialized input text is converted from a piece of text content, which often contains multiple words or characters, the serialized input text to be vectorized is sampled and segmented according to the word segmentation strategy in the vector space model, thereby converting the serialized input text to be vectorized into a text sequence composed of a series of characters or words.

[0079] Step 402: Based on the vector representations of the sampled and segmented words in the vector space model and the order of the sampled and segmented words in the serialized input text, generate a high-dimensional embedding vector representation of the serialized input text.

[0080] Specifically, from the vector space model, the vector representations corresponding to a series of characters or words after sampling and segmentation are obtained. Then, according to the order of the positions of the characters and words after sampling and segmentation in the serialized input text, the vector representations corresponding to all characters or words are concatenated and organized to obtain the high-dimensional embedding vector representation corresponding to the serialized input text.

[0081] It should be understood that the term "high-dimensional" in the high-dimensional embedding vector representation here is relative. It refers to the fact that, relative to the binary code value conversion result of the LSH algorithm in the subsequent processing steps, it contains only "0" and "1" encoded values, and is therefore high-dimensional.

[0082] Continue to refer to Figure 5 In some specific embodiments, a step of preparing the vector database is included before step 204. Figure 5 This is a flowchart of a specific embodiment of the LSH-based retrieval enhancement generation method described in this application, which involves preparing a vector database, including:

[0083] Step 501: Connect to the corpus of the retrieved text;

[0084] Specifically, the retrieved text corpus refers to the database where the actual search results of the retrieved input text are located.

[0085] Step 502: Sequentially obtain each retrieved text from the retrieved text corpus;

[0086] Step 503: Serialize each of the retrieved texts to obtain serialized retrieved text;

[0087] Step 504: Sample all serialized retrieved texts using the vector space model to generate high-dimensional vector representations of all serialized input texts;

[0088] Step 505: Add the high-dimensional vector representations corresponding to all serialized input texts to the preset vector database.

[0089] By pre-vectorizing each retrieved text in the retrieved text corpus, a high-dimensional vector representation corresponding to each retrieved text is obtained. Then, the high-dimensional vector representation corresponding to each retrieved text is added to the preset vector database to support the conversion of text-to-text retrieval mode into vector representation-to-vector representation retrieval mode, thereby improving the retrieval efficiency during actual retrieval.

[0090] Continue to refer to Figure 6 , Figure 6 yes Figure 2A flowchart of a specific embodiment of step 204 shown includes:

[0091] Step 601: Sequentially obtain the high-dimensional vector representation corresponding to each retrieved text in the vector database as the current high-dimensional vector representation to be matched;

[0092] Step 602: Compare the high-dimensional embedding vector representation with the current high-dimensional vector representation to be matched, and identify whether the current high-dimensional vector representation to be matched completely contains the high-dimensional embedding vector representation at least once;

[0093] Step 603: If the current high-dimensional vector representation to be matched completely contains the high-dimensional embedded vector representation at least once, then the current high-dimensional vector representation to be matched is marked with a preset first distinguishing identifier;

[0094] Specifically, if the current high-dimensional vector representation to be matched completely includes the high-dimensional embedding vector representation at least once, it indicates that the retrieved text corresponding to the current high-dimensional vector representation to be matched has a certain correlation with the retrieved input text.

[0095] Step 604: If the current high-dimensional vector representation to be matched does not fully contain or does not contain the high-dimensional embedded vector representation, then the current high-dimensional vector representation to be matched is marked with a preset second distinguishing identifier.

[0096] Specifically, if the current high-dimensional vector representation to be matched does not fully contain or does not contain the high-dimensional embedding vector table, it means that the searched text corresponding to the current high-dimensional vector representation to be matched has low relevance to the search input text, and the searched text can be excluded first.

[0097] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 205 shown includes:

[0098] Step 701: Based on the matching calculation results, select the high-dimensional vector representation marked by the second distinguishing identifier;

[0099] Step 702: Set the binary code value of the retrieved text corresponding to the high-dimensional vector representation marked by the second distinguishing identifier to 0.

[0100] Specifically, for the high-dimensional vector representation of the retrieved text marked by the second distinguishing identifier, its relevance to the retrieved input text is low. Therefore, its binary code value is set to 0. Here, setting its binary code value to 0 can be achieved by using the sign function in the LSH algorithm to output a binary code value of 0.

[0101] Step 703: Based on the matching calculation results, select the high-dimensional vector representation marked by the first distinguishing identifier;

[0102] Step 704: Count the number of times each high-dimensional vector representation marked by the first distinguishing identifier completely contains the high-dimensional embedded vector representation, and use this as the number of times it contains the vector representation;

[0103] Specifically, the searched texts that are related to the search input text are selected. Since the high-dimensional vector representation corresponding to the searched text may not be limited to containing the high-dimensional embedding vector representation only once, the number of times each high-dimensional vector representation marked by the first distinguishing identifier completely contains the high-dimensional embedding vector representation is counted as the number of inclusions. That is, the more inclusions, the stronger the correlation between the searched text and the search input text.

[0104] Step 705: For the retrieved text corresponding to the high-dimensional vector representation marked by the first distinguishing identifier, generate a target binary code value according to the number of inclusions and a preset binary code value conversion format. Specifically, the step of generating the target binary code value according to the number of inclusions and the preset binary code value conversion format includes: if the number of inclusions is N, then generate an N-bit binary code value consisting entirely of 1s, where N is a positive integer.

[0105] In this embodiment, the target binary code value is generated according to the number of inclusions and the preset binary code value conversion format. That is, if the number of inclusions is 5, the binary code value "11111" is generated, and if the number of inclusions is 2, the binary code value "11" is generated. This realizes the use of binary code values ​​to reflect the strength of the relevance between the retrieved text and the retrieved input text. Here, the binary code value 1 can also be output progressively by the sign function in the LSH algorithm.

[0106] Continue to refer to Figure 8 , Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 206 shown includes:

[0107] Step 801: Filter out the text to be retrieved when the number of 1s in the binary code value is the maximum value;

[0108] Specifically, assuming the generated binary code values ​​are "111", "1111", and "111111", the text corresponding to the binary code value "111111" is selected as the expected text to be retrieved as the search input text, that is, the text with the highest relative relevance is selected as the expected text.

[0109] Step 802: Set the retrieved text to the desired retrieved text of the retrieval input text.

[0110] By selecting the most relevant retrieved text as the desired text, the algorithm aims to provide the most relevant and higher-quality retrieval feedback text for the input retrieval text. Compared to traditional relevance calculation methods, such as the cosine similarity algorithm, the LSH algorithm has the following significant advantages: First, LSH maps high-dimensional continuous vectors to low-dimensional binary code values, significantly reducing storage space requirements and computational complexity, making large-scale vector retrieval possible; second, the LSH algorithm ensures that similar or nearly related texts are mapped to the same binary code value; finally, LSH naturally supports approximate nearest neighbor search, avoiding the computational burden of accurately calculating the similarity of all candidate documents, making it particularly suitable for real-time retrieval applications.

[0111] In this embodiment, the step of merging and organizing the search input text and the expected searched text to generate the text to be enhanced specifically includes: merging and organizing the search input text and the expected searched text according to a preset generation template, wherein the generation template includes a search field filling area and a search result filling area. Specifically, the search input text is filled into the search field filling area and all the expected searched texts corresponding to the search input text are filled into the search result filling area to generate the text to be enhanced.

[0112] In essence, when generating the text to be enhanced, the generation template is combined with a combination of prompt content and main content. Here, the prompt content refers to the search input text, while the main content is the text that the search input text expects to be retrieved. Then, the enhancement generator based on the LLM large language model is used to optimize the search results to obtain higher quality search feedback text.

[0113] In this embodiment, the following steps are taken: First, the input text is acquired and serialized. Then, a high-dimensional embedding vector representation is generated through sampling using a vector space model. This high-dimensional embedding vector representation is input into a vector database for matching calculations. Next, the LSH algorithm is used to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts. Based on these binary code values, the desired retrieved text is selected. The input text and the desired retrieved text are then merged and organized to generate the text to be enhanced. Finally, the text to be enhanced is input into a text enhancement generator to generate the enhanced search feedback text. Essentially, the LSH algorithm is applied to the RAG system. The RAG system has two parts: a search engine and an enhancement generator. By integrating the LSH algorithm into the RAG system's search engine, the search results can be obtained more quickly and accurately based on the input text during the search engine's retrieval process. The search results are then input into the enhancement generator for enhancement processing, resulting in the final search feedback text fed back to the search initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0114] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0115] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0116] In this embodiment, the following steps are taken: First, the input text is acquired and serialized. Then, a high-dimensional embedding vector representation is generated through sampling using a vector space model. This high-dimensional embedding vector representation is input into a vector database for matching calculations. Next, the LSH algorithm is used to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts. Based on these binary code values, the desired retrieved text is selected. The input text and the desired retrieved text are then merged and organized to generate the text to be enhanced. Finally, the text to be enhanced is input into a text enhancement generator to generate the enhanced search feedback text. Essentially, the LSH algorithm is applied to the RAG system. The RAG system has two parts: a search engine and an enhancement generator. By integrating the LSH algorithm into the RAG system's search engine, the search results can be obtained more quickly and accurately based on the input text during the search engine's retrieval process. The search results are then input into the enhancement generator for enhancement processing, resulting in the final search feedback text fed back to the search initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0117] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an LSH-based retrieval enhancement generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0118] like Figure 9 As shown, the LSH-based retrieval enhancement generation device 900 described in this embodiment includes: a retrieval input text acquisition module 901, a serialization processing module 902, a high-dimensional embedding vector representation generation module 903, a matching calculation module 904, an LSH algorithm processing module 905, a desired retrieval result filtering module 906, a text generation module 907 to be enhanced, and a retrieval feedback text generation module 908.

[0119] in:

[0120] The input text acquisition module 901 is used to acquire the input text for retrieval.

[0121] The serialization processing module 902 is used to serialize the search input text to obtain serialized input text.

[0122] The high-dimensional embedding vector representation generation module 903 is used to sample the serialized input text through a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text.

[0123] The matching calculation module 904 is used to input the high-dimensional embedded vector representation into a preset vector database and perform matching calculations with the high-dimensional vector representation corresponding to each retrieved text in the vector database.

[0124] LSH algorithm processing module 905 is used to perform binary code value conversion on the matching calculation result using the LSH algorithm to obtain the binary code value corresponding to all the retrieved text after conversion. The LSH algorithm includes the Locality Sensitive Hash algorithm.

[0125] The expected search result filtering module 906 is used to filter out the expected search text of the search input text based on the binary code value.

[0126] The text to be enhanced processing generation module 907 is used to merge and organize the search input text and the expected searched text of the search input text to generate the text to be enhanced processing;

[0127] The retrieval feedback text generation module 908 is used to input the text to be enhanced into a preset text enhancement generator to generate enhanced retrieval feedback text, wherein the preset text enhancement generator includes an LLM-based text enhancement generator.

[0128] This application involves: acquiring the input text for retrieval; performing serialization processing; sampling processing using a vector space model to generate a high-dimensional embedding vector representation; inputting the high-dimensional embedding vector representation into a vector database for matching calculations; using the LSH algorithm to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts after conversion; filtering the desired retrieved text based on the binary code values; merging and organizing the input text and the desired retrieved text to generate the text to be enhanced; and inputting the text to be enhanced into a text enhancement generator to generate the enhanced retrieval feedback text. Essentially, it applies the LSH algorithm to the RAG system, which has two parts: a retrieval unit and an enhancement generator. By integrating the LSH algorithm into the retrieval unit of the RAG system, the retrieval results can be obtained more quickly and accurately based on the input text during retrieval processing. The retrieval results are then input into the enhancement generator for enhancement processing, obtaining the final retrieval feedback text to be fed back to the retrieval initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0129] In this embodiment, the LSH-based retrieval enhancement generation device 900 further includes: a text cleaning module and a text updating module. Wherein:

[0130] The text cleaning module is used to clean the serialized input text of meaningless characters or words using preset cleaning components;

[0131] The text update module is used to obtain the serialized input text after the meaningless characters or words have been cleaned, and update it to the serialized input text to be vectorized.

[0132] In this embodiment, the high-dimensional embedding vector representation generation module 903 includes: a sampling segmentation unit and a high-dimensional embedding vector representation generation unit. Wherein:

[0133] The sampling and segmentation unit is used to perform word sampling and segmentation on the serialized input text to be vectorized according to the word segmentation strategy in the vector space model;

[0134] The high-dimensional embedding vector representation generation unit is used to generate a high-dimensional embedding vector representation of the serialized input text based on the vector representations of the sampled and segmented words in the vector space model and the order of the sampled and segmented words in the serialized input text.

[0135] In this embodiment, the LSH-based retrieval enhancement generation device 900 further includes: a text corpus connection module, a retrieved text acquisition module, a retrieved text serialization module, a retrieved text vectorization module, and a vectorization result storage module. Wherein:

[0136] The text corpus connection module is used to connect to the retrieved text corpus;

[0137] The retrieved text acquisition module is used to sequentially acquire each retrieved text from the retrieved text corpus;

[0138] The retrieved text serialization module is used to serialize each retrieved text to obtain serialized retrieved text.

[0139] The retrieved text vectorization module is used to sample all serialized retrieved texts through the vector space model to generate high-dimensional vector representations corresponding to all serialized input texts;

[0140] The vectorization result storage module is used to add the high-dimensional vector representations corresponding to all serialized input texts to the preset vector database.

[0141] In this embodiment, the matching calculation module 904 includes: a high-dimensional vector representation sequential acquisition unit, a matching recognition unit, a first distinguishing marker unit, and a second distinguishing marker unit. Wherein:

[0142] The high-dimensional vector representation sequential acquisition unit is used to sequentially acquire the high-dimensional vector representation corresponding to each retrieved text in the vector database as the current high-dimensional vector representation to be matched.

[0143] The matching and recognition unit is used to compare the high-dimensional embedding vector representation with the current high-dimensional vector representation to be matched, and to identify whether the current high-dimensional vector representation to be matched contains the high-dimensional embedding vector representation at least once.

[0144] The first distinguishing labeling unit is used to mark the current high-dimensional vector representation to be matched with a preset first distinguishing identifier if the current high-dimensional vector representation to be matched contains the high-dimensional embedded vector representation at least once.

[0145] The second distinguishing labeling unit is used to mark the current high-dimensional vector representation to be matched with a preset second distinguishing identifier if the current high-dimensional vector representation to be matched does not fully contain or does not contain the high-dimensional embedded vector representation.

[0146] In this embodiment, the LSH algorithm processing module 905 includes: a first filtering unit for high-dimensional vector representation, a first setting unit for binary code values, a second filtering unit for high-dimensional vector representation, a unit for counting frequency, and a second setting unit for binary code values. Wherein:

[0147] The high-dimensional vector representation first filtering unit is used to filter out the high-dimensional vector representation marked by the second distinguishing identifier based on the matching calculation result;

[0148] The first binary code value setting unit is used to set the binary code value of the retrieved text corresponding to the high-dimensional vector representation marked by the second distinguishing identifier to 0.

[0149] The high-dimensional vector representation second filtering unit is used to filter out the high-dimensional vector representation marked by the first distinguishing identifier based on the matching calculation result;

[0150] The system includes a count unit, which counts the number of times each high-dimensional vector representation marked by the first distinguishing identifier completely contains the high-dimensional embedded vector representation, and uses this count as the number of times the vector representation is contained.

[0151] The second binary code value setting unit is used to generate a target binary code value for the retrieved text corresponding to the high-dimensional vector representation marked by the first distinguishing identifier, according to the number of inclusions and a preset binary code value conversion format. Specifically, the step of generating the target binary code value according to the number of inclusions and the preset binary code value conversion format includes: if the number of inclusions is N, then generating an N-bit binary code value consisting entirely of 1s, where N is a positive integer.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0153] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0154] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0155] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c, which are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 10 with component memory 10a, processor 10b, and network interface 10c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0156] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0157] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10. Of course, the memory 10a may also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for a search enhancement generation method based on LSH. Furthermore, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.

[0158] In some embodiments, the processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions of the LSH-based retrieval enhancement generation method.

[0159] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0160] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in text retrieval scenarios. This application obtains the retrieval input text; performs serialization processing; performs sampling processing through a vector space model to generate a high-dimensional embedding vector representation; inputs the high-dimensional embedding vector representation into a vector database for matching calculation; uses the LSH algorithm to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts after conversion; filters the desired retrieved text based on the binary code values; merges and organizes the retrieval input text and the desired retrieved text to generate the text to be enhanced; and inputs the text to be enhanced into a text enhancement generator to generate the enhanced retrieval feedback text. Essentially, it applies the LSH algorithm to the RAG system. The RAG system has two parts: a retrieval unit and an enhancement generator. By integrating the LSH algorithm into the retrieval unit of the RAG system, it enables faster and more accurate retrieval results based on the retrieval input text during retrieval processing. Then, the retrieval results are input into the enhancement generator for enhancement processing to obtain the final retrieval feedback text, which is then fed back to the retrieval initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0161] This application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the LSH-based retrieval enhancement generation method described above.

[0162] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in text retrieval scenarios. This application obtains the retrieval input text; performs serialization processing; performs sampling processing through a vector space model to generate a high-dimensional embedding vector representation; inputs the high-dimensional embedding vector representation into a vector database for matching calculation; uses the LSH algorithm to convert the matching calculation results into binary code values, obtaining the binary code values ​​corresponding to all retrieved texts after conversion; filters the desired retrieved text based on the binary code values; merges and organizes the retrieval input text and the desired retrieved text to generate the text to be enhanced; and inputs the text to be enhanced into a text enhancement generator to generate the enhanced retrieval feedback text. Essentially, it applies the LSH algorithm to the RAG system. The RAG system has two parts: a retrieval unit and an enhancement generator. By integrating the LSH algorithm into the retrieval unit of the RAG system, it enables faster and more accurate retrieval results based on the retrieval input text during retrieval processing. Then, the retrieval results are input into the enhancement generator for enhancement processing to obtain the final retrieval feedback text, which is then fed back to the retrieval initiator. The method can be applied to scenarios such as web search, question-and-answer systems, and recommendation systems in the fields of financial or medical services, so as to provide searchers with faster and more accurate search results.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0164] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A method for LSH-based retrieval augmentation generation, the method comprising: The method comprises the following steps: obtaining a search input text; serializing the search input text to obtain a serialized input text; sampling the serialized input text by using a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text; inputting the high-dimensional embedding vector representation into a preset vector database and performing matching calculation on the high-dimensional vector representation corresponding to each searched text in the vector database; converting the matching calculation result into binary code values by using an LSH algorithm, wherein the LSH algorithm comprises a local sensitive hashing algorithm; screening the searched text expected by the search input text according to the binary code values; merging and arranging the search input text and the searched text expected by the search input text to generate a text to be enhanced; inputting the text to be enhanced into a preset text enhancement generator to generate an enhanced search feedback text, wherein the preset text enhancement generator comprises an LLM-based text enhancement generator.

2. The LSH-based retrieval boost generation method of claim 1, wherein, Before the step of sampling the serialized input text by using a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text, the method further comprises: cleaning meaningless words or meaningless phrases from the serialized input text by using a preset cleaning component; updating the serialized input text after the cleaning of meaningless words or meaningless phrases to a serialized input text to be vectorized; the step of sampling the serialized input text by using a preset vector space model to generate a high-dimensional embedding vector representation corresponding to the serialized input text specifically comprises: sampling and dividing the serialized input text to be vectorized according to the word segmentation strategy in the vector space model; generating the high-dimensional embedding vector representation corresponding to the serialized input text according to the vector representation corresponding to the sampled and divided words in the vector space model and the order of the sampled and divided words in the serialized input text.

3. The LSH-based retrieval boost generation method of claim 1, wherein, Before the step of inputting the high-dimensional embedding vector representation into a preset vector database and performing matching calculation on the high-dimensional vector representation corresponding to each searched text in the vector database, the method further comprises: connecting a searched text corpus; sequentially obtaining each searched text in the searched text corpus; serializing each searched text to obtain a serialized searched text; sampling all serialized searched texts by using the vector space model to generate high-dimensional vector representations corresponding to all serialized input texts; adding the high-dimensional vector representations corresponding to all serialized input texts to the preset vector database.

4. The LSH-based retrieval boost generation method according to any one of claims 1 or 3, characterized in that, the step of inputting the high-dimensional embedding vector representation into a preset vector database and performing matching calculation on the high-dimensional vector representation corresponding to each searched text in the vector database specifically comprises: sequentially obtaining each high-dimensional vector representation corresponding to each piece of searched text in the vector database as a current high-dimensional vector representation to be matched; matching the high-dimensional embedding vector representation with the current high-dimensional vector representation to be matched, and identifying whether the current high-dimensional vector representation to be matched at least completely contains the high-dimensional embedding vector representation once; if the current high-dimensional vector representation to be matched at least completely contains the high-dimensional embedding vector representation once, marking the current high-dimensional vector representation to be matched with a preset first distinguishing identifier; if the current high-dimensional vector representation to be matched does not completely contain or does not contain the high-dimensional embedding vector representation, marking the current high-dimensional vector representation to be matched with a preset second distinguishing identifier.

5. The LSH-based retrieval boost generation method of claim 4, wherein, The step of converting the matching calculation result into a binary code value by using the LSH algorithm to obtain the binary code value corresponding to all searched texts after conversion, specifically includes: According to the matching calculation result, filtering out the high-dimensional vector representation marked with the second distinguishing identifier; setting the binary code value of the searched text corresponding to the high-dimensional vector representation marked with the second distinguishing identifier to 0; According to the matching calculation result, filtering out the high-dimensional vector representation marked with the first distinguishing identifier; counting the number of times that each high-dimensional vector representation marked with the first distinguishing identifier completely contains the high-dimensional embedding vector representation as a containing number; According to the containing number and a preset binary code value conversion format, generating a target binary code value for the searched text corresponding to the high-dimensional vector representation marked with the first distinguishing identifier, wherein the step of generating a target binary code value according to the containing number and a preset binary code value conversion format specifically includes: if the containing number is N, generating an N-bit binary code value composed of 1s, and N is a positive integer.

6. The LSH-based retrieval enhancement generation method of claim 1, wherein, The step of filtering out the searched text expected by the search input text according to the binary code value, specifically includes: filtering out the searched text when the number of code values 1 in the binary code value is the maximum value; setting the searched text as the searched text expected by the search input text.

7. The LSH-based retrieval enhancement generation method of claim 1, wherein, The step of merging and arranging the search input text and the searched text expected by the search input text to generate a text to be enhanced, specifically includes: merging and arranging the search input text and the searched text expected by the search input text according to a preset generation template, wherein the generation template includes a search field filling area and a search result filling area, and specifically, the search input text is filled into the search field filling area, and all expected searched texts corresponding to the search input text are filled into the search result filling area to generate the text to be enhanced.

8. An LSH-based retrieval augmentation generation device, characterized by, It includes: a search input text acquisition module for acquiring a search input text; a serialization processing module for serializing the search input text to obtain a serialized input text; a high-dimensional embedding vector representation generation module for generating a high-dimensional embedding vector representation corresponding to the serialized input text by sampling the serialized input text through a preset vector space model; The matching calculation module is configured to input the high-dimensional embedding vector representation into a preset vector database, and perform matching calculation on a high-dimensional vector representation corresponding to each piece of retrieved text in the vector database respectively. The LSH algorithm processing module is configured to perform binary code value conversion on the matching calculation result by using an LSH algorithm to obtain binary code values corresponding to all pieces of retrieved text after conversion, wherein the LSH algorithm includes a local sensitive hashing algorithm. The expected search result screening module is configured to screen out the retrieved text expected by the search input text according to the binary code values. The text to be enhanced processing module is configured to merge and arrange the search input text and the retrieved text expected by the search input text to generate a text to be enhanced processing. The search feedback text generation module is configured to input the text to be enhanced processing into a preset text enhancement generator to generate an enhanced search feedback text, wherein the preset text enhancement generator includes an LLM-based text enhancement generator.

9. A computer device, comprising: A computer readable storage medium has computer readable instructions stored thereon, and the computer readable instructions are executed by a processor to implement the steps of the LSH-based search enhancement generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has computer readable instructions stored thereon, and the computer readable instructions are executed by a processor to implement the steps of the LSH-based search enhancement generation method according to any one of claims 1 to 7.

Images