Method and apparatus for performing multi-path recall retrieval-augmented generation on basis of real-time data indexing

Abstract

A method and apparatus for performing multi-path recall retrieval-augmented generation on the basis of real-time data indexing, which are applied to the technical field of data processing. The solution of the present application comprises: in response to a retrieval requirement, setting at least one data capture method to acquire real-time data; on the basis of the real-time data, constructing an index database; setting at least one type of retrieval scheme on the basis of the index database, and re-ranking retrieval results obtained by means of the retrieval scheme; and generating a context prompt on the basis of the retrieval results, and generating retrieval text on the basis of the context prompt by means of a large language model. In the solution of the embodiments of the present application, efficient retrieval and utilization of information are realized by means of real-time data indexing and a multi-path recall mechanism, and the content generation quality and correlation of a large language model in specific fields and real-time scenarios are significantly improved, thereby enhancing the accuracy and reliability of a generative task.

Description

A method and apparatus for enhancing multi-path recall retrieval based on real-time data indexing

[0001] This application claims priority to Chinese Patent Application No. 202410955855.3, filed on July 17, 2024, entitled "Multi-path Recall Retrieval Enhancement Generation Method and Apparatus Based on Real-time Data Indexing", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data processing, and in particular to a method and apparatus for enhancing multi-way recall retrieval based on real-time data indexing. Background Technology

[0003] With the rapid development of artificial intelligence technology, large language models have become a core technology in the field of natural language processing. While large language models have achieved remarkable success in processing static data and general knowledge, they still face challenges when dealing with real-time data and domain-specific knowledge. To overcome these challenges, Retrieval Augmentation (RAG) technology has emerged. RAG technology combines a retrieval system with a large language model, enabling the model to retrieve relevant information from a large number of external data sources before generating a response. This not only utilizes the latest data but also incorporates domain-specific knowledge, thereby significantly improving the performance of large language models in specific tasks.

[0004] Existing RAG systems still have shortcomings in terms of real-time performance and data security. The lack of real-time data indexing and multiple-path recall mechanisms limits the system's ability to respond quickly to the latest information. At the same time, existing systems often lack sufficient security measures when processing sensitive data, failing to meet the high standards of data privacy required by enterprises and individuals. Summary of the Invention

[0005] This application provides a multi-path recall retrieval enhancement generation method and apparatus based on real-time data indexing, which can improve the accuracy and reliability of generative tasks.

[0006] According to one aspect of this application, a multi-way recall retrieval enhancement generation method based on real-time data indexing is provided, comprising:

[0007] In response to retrieval needs, at least one data crawling method is set to obtain real-time data;

[0008] An index database is constructed based on the real-time data;

[0009] Based on the index database, at least one type of retrieval scheme is set, and the retrieval results obtained through the retrieval scheme are reordered;

[0010] Contextual hints are generated based on the search results, and the search text is generated based on the contextual hints using a large language model.

[0011] Optionally, the data crawling methods include: web crawling technology, RSS subscription mechanism, API call interface, web page monitoring algorithm and business data push.

[0012] Optionally, after setting at least one data capture method to obtain real-time data, the method further includes:

[0013] Perform one or more of the following operations on the real-time data:

[0014] The real-time data is cleaned.

[0015] The real-time data is deduplicated;

[0016] The real-time data is formatted and deduplicated of special characters;

[0017] Meta-information is extracted from the real-time data, and the meta-information is then reviewed.

[0018] Optionally, the step of building an index database based on the real-time data includes:

[0019] The real-time text data is converted into a high-dimensional vector using text embedding technology.

[0020] Construct an index database based on the high-dimensional vector and incrementally update and construct the index.

[0021] Optionally, the retrieval scheme includes: sparse retrieval, dense retrieval, querying through a public domain search engine, and querying through a private domain search engine.

[0022] Optionally, the reordering of the search results obtained through the search scheme includes:

[0023] Retrieve information related to the search request from at least one data source or index;

[0024] Remove duplicates from the search results, merge and reorder the recalled information, and sort it according to the relevance results.

[0025] Optionally, generating contextual hints based on the search results and generating search text based on the contextual hints using a large language model includes:

[0026] The search results and the search requirements are used to generate the contextual prompts.

[0027] The search text is generated based on the contextual prompts using at least one sampling and decoding strategy.

[0028] According to another aspect of this application, a multi-way recall retrieval enhancement generation apparatus based on real-time data indexing is provided, comprising:

[0029] The crawling unit is used to respond to retrieval needs by setting at least one data crawling method to obtain real-time data;

[0030] A construction unit is used to build an index database based on the real-time data;

[0031] The sorting unit is used to set at least one type of retrieval scheme based on the index database and reorder the retrieval results obtained by the retrieval scheme.

[0032] The generation unit is used to generate contextual prompts based on the search results, and to generate search text based on the contextual prompts using a large language model.

[0033] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0034] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the multi-path recall retrieval enhancement generation method based on real-time data indexing as described in any embodiment of this application.

[0035] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multi-way recall retrieval enhancement generation method based on real-time data indexing as described in any embodiment of this application.

[0036] The solution of this application embodiment includes: responding to retrieval needs, setting at least one data crawling method to acquire real-time data; constructing an index database based on the real-time data; setting at least one type of retrieval scheme based on the index database, and reordering the retrieval results obtained through the retrieval scheme; generating contextual prompts based on the retrieval results, and generating retrieval text based on the contextual prompts using a large language model. The solution of this application embodiment, through real-time data indexing and a multi-path recall mechanism, achieves efficient retrieval and utilization of information, significantly improving the content generation quality and relevance of the large language model in specific domains and real-time scenarios, thereby improving the accuracy and reliability of generative tasks.

[0037] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 is a flowchart of a multi-way recall retrieval enhancement generation method based on real-time data indexing provided in an embodiment of this application;

[0040] Figure 2 is a schematic diagram of an incremental index construction provided in an embodiment of this application;

[0041] Figure 3 is an architecture diagram of retrieval enhancement generation based on real-time data indexing provided in an embodiment of this application;

[0042] Figure 4 is a schematic diagram of a multi-path recall retrieval enhancement generation device based on real-time data indexing provided in an embodiment of this application;

[0043] Figure 5 is a schematic diagram of the structure of an electronic device that implements the multi-way recall retrieval enhancement generation method based on real-time data indexing according to the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] Figure 1 is a flowchart of a multi-way recall retrieval enhancement generation method based on real-time data indexing according to an embodiment of this application. As shown in Figure 1, the method includes the following steps:

[0047] S110. In response to retrieval needs, set at least one data capture method to obtain real-time data.

[0048] In this application embodiment, the data crawling methods include: web crawler technology, RSS subscription mechanism, API call interface, web page monitoring algorithm and business data push.

[0049] Figure 2 is a schematic diagram of incremental index construction provided in an embodiment of this application. As shown in Figure 2, real-time data crawling and collection refers to the real-time crawling and collection of data from the Internet and diverse data sources, supporting different protocols to meet common data sources in the real world:

[0050] Web crawling technology: This module integrates efficient and advanced web crawling technology, which can intelligently traverse and index web page content; 1) Based on Python's asyncio module, it implements asynchronous I / O technology to process multiple network requests in parallel; 2) It follows the robots.txt protocol. Before crawling, the crawler will check and parse the target website's robots.txt file to ensure the legality of data crawling, while following the crawling frequency and path set in the protocol to reduce the impact on the target website's service performance.

[0051] RSS Subscription and Update Mechanism: This module uses Python's BeautifulSoup library to parse HTML, attempts to find the rss field in the rel attribute of the link tag, and uses the feedparser library to parse the RSS source, thereby automatically discovering the website's RSS source and subscribing to it. Combined with a periodic polling scheme, it implements an automatic update mechanism to quickly respond to and collect updates of key information and obtain updated information in a timely manner.

[0052] API call functionality: This allows the system to directly interface with third-party data providers to obtain structured and unstructured data. This includes partnered social media platforms, news websites, professional forums, and industry databases, ensuring a wide range of information sources and rich content.

[0053] Webpage monitoring capabilities: By setting monitoring based on keywords or specific events (such as news releases, stock price fluctuations, etc.), the data capture process can be triggered immediately when relevant information is released, greatly shortening the time difference between information release and system acquisition.

[0054] Business Data Push: To meet specific business needs, this module also supports real-time data push solutions, allowing the system to proactively extract and import the latest data from business systems based on preset business rules and data models. This ensures the system can quickly obtain the latest information and data.

[0055] In this embodiment of the application, after setting at least one data capture method to obtain real-time data, the method further includes:

[0056] Perform one or more of the following operations on the real-time data:

[0057] The real-time data is cleaned.

[0058] The real-time data is deduplicated;

[0059] The real-time data is formatted and deduplicated of special characters;

[0060] Meta-information is extracted from the real-time data, and the meta-information is then reviewed.

[0061] A general-purpose, pluggable data processing pipeline can be built for data processing, offering high scalability and customizability, and can be adjusted according to business needs and actual conditions.

[0062] Data cleaning: This includes removing useless information, such as HTML tags, advertisements, scripts, etc.

[0063] Data deduplication: Duplicate content is checked using SimHash and cosine or L2 distance metrics, and deduplication is performed at three granularities: sentence, paragraph, and chapter.

[0064] Data formatting: Convert the data into a uniform format, such as field information including crawling date, title, body, data category, and URL, and remove duplicate special characters, and de-identify sensitive information containing user personal information, such as mobile phone number and ID card number.

[0065] Meta-information extraction: Utilizing mature NLP technologies such as Named Entity Recognition (NER), keyword extraction algorithms (TF-IDF, TeXRank, etc.), key entities in the text, such as names of people, locations, organization names, and important keywords, are extracted. Furthermore, the semantic sentiment analysis capabilities of NLP are used to determine whether the text contains content that threatens social security or violates human values, in order to achieve governance of the crawled data.

[0066] The above functions are implemented through the construction of sub-modules to achieve a plug-in pipeline, which allows for flexible addition, deletion, or modification of configurations to ensure compatibility between plugins and the continuity of data flow. By using big data technologies, combined with Spark, MySQL, and HDFS, data is managed and ETL (Extract Transform Load) is performed, and monitoring and management of data categories, quantity, and quality are added.

[0067] S120. Construct an index database based on the real-time data.

[0068] In this embodiment of the application, the step of constructing an index database based on the real-time data includes:

[0069] The real-time text data is converted into a high-dimensional vector using text embedding technology.

[0070] Construct an index database based on the high-dimensional vector and incrementally update and construct the index.

[0071] The processed data is transformed into an efficient index format, supporting incremental updates. It employs advanced text embedding techniques, such as the BGE-M3 model from Zhiyuan Open Source, which combines multilingual and multi-granular characteristics to encode text fragments into high-dimensional representations after character length and semantic segmentation identification. Based on the Facebook AI Research open-source index building engine FAISS system (with the built-in function `index.add`), data is updated to the existing index library for efficient similarity retrieval. Simultaneously, NLP techniques are used to extract keywords, and the open-source ElasticSearch (ES) engine is used for sparse retrieval and incremental index building.

[0072] S130. Based on the index database, set at least one type of retrieval scheme, and reorder the retrieval results obtained through the retrieval scheme.

[0073] In this application embodiment, the retrieval scheme includes: sparse retrieval, dense retrieval, querying through a public domain search engine, and querying through a private domain search engine.

[0074] Figure 3 is an architecture diagram of a retrieval enhancement generation based on real-time data indexing provided in an embodiment of this application.

[0075] This application's embodiments support multiple types of search schemes, including:

[0076] 1) Sparse retrieval: such as retrieval based on the ElasticSearch (ES) engine, and inverted index schemes built based on keywords;

[0077] 2) Dense retrieval: such as retrieval based on FAISS, or retrieval based on open-source vector databases such as Milvus;

[0078] 3) Search through public search engines, such as calling the APIs of search engines like Baidu, Bing, and Google;

[0079] 4) Search through a private domain search engine, such as by calling the API of a self-built search engine within the business.

[0080] Each retrieval process supports multiple indexes or data sources. Asynchronous concurrent recall of content related to the user's query enables retrieval requests, improving system response speed and throughput. Simultaneously, to enhance system response performance, a combination of memory-based or Redis-based high-speed caching with the LRU (Least Recently Used) algorithm and expiration mechanisms can be used to cache retrieval results for the user's query, reducing duplicate queries to backend data sources and decreasing retrieval time. This allows the system to retrieve relevant information from different data sources, improving recall and accuracy while ensuring system performance and availability.

[0081] In this embodiment of the application, the search results obtained through the search scheme are reordered, including:

[0082] Retrieve information related to the search request from at least one data source or index;

[0083] Remove duplicates from the search results, merge and reorder the recalled information, and sort it according to the relevance results.

[0084] Each retrieval scheme supports recall from multiple data sources or indexes, enabling the retrieval of information relevant to the user's query from multiple perspectives. Duplicates may exist between different retrieval schemes. After removing duplicates from the search results, a reranker is used to merge and re-rank the content retrieved from multiple data sources. The ranking is based on relevance, with the most relevant information (using top_k=4 in practice) prioritized for the enhancement module. This information is directly concatenated with a newline character ("\n") and combined with the prompt word template as input to the larger model.

[0085] S140. Generate contextual hints based on the search results, and generate search text based on the contextual hints using a large language model.

[0086] In this embodiment of the application, a contextual prompt is generated based on the search results, and a search text is generated based on the contextual prompt using a large language model, including:

[0087] The search results and the search requirements are used to generate the contextual prompts.

[0088] The search text is generated based on the contextual prompts using at least one sampling and decoding strategy.

[0089] By leveraging a Large Language Model (LLM), retrieved information is combined with the user's original query to form a rich contextual prompt, guiding the LLM's generation process. By combining the retrieved information with the powerful generation capabilities of the LLM, different sampling and decoding strategies, such as greedy search, beam serach, top-k sampling, and top-p sampling, are employed during generation to optimize the quality and diversity of the generated content. These strategies aim to balance the fluency, relevance, and creativity of the generated text, enabling the system to produce accurate, reliable, and high-quality content.

[0090] For the large models used here, LLMs may need to be fine-tuned for specific tasks to improve the accuracy and reliability of the generated content. This includes adjusting the model's parameters using a small amount of labeled data (fine-tuning), and using full-parameter fine-tuning or efficient parameter fine-tuning schemes to better adapt to the language habits and knowledge requirements of the specific domain.

[0091] Compared with the prior art, this application has the following beneficial effects:

[0092] (1) More real-time information content: By using RSS subscription, API calls, web page monitoring and business-customized push, the speed of acquiring and updating external knowledge bases has been significantly improved. At the same time, the real-time performance and accuracy of knowledge base index construction have also been improved, enabling the system to quickly adapt to changes in information.

[0093] (2) More comprehensive search capabilities: Through sparse search, dense search, and multi-channel search queries from public and private search engines, more relevant and comprehensive information about the user's questions can be obtained; the use of public and private search engines fully utilizes the powerful capabilities of modern search engine systems while also ensuring the real-time nature of the data.

[0094] (3) More secure private domain data: By proactively pushing data through the business and using the private domain search engine, while ensuring the timeliness of information and utilizing domain knowledge, users can also customize data privacy and security policies, thereby better protecting user privacy and confidentiality, which is in line with the current growing attention to data security and regulatory requirements.

[0095] (4) Better scalability: Support for multi-path concurrent retrieval makes it easier for users to configure and specify new index data sources;

[0096] In summary, the system designed in this application can acquire domain-related data in more real-time, protect user privacy and data security, and improve the accuracy and reliability of generative tasks. It can adapt to more diverse application scenarios and needs and has broad application prospects.

[0097] Figure 4 is a schematic diagram of a multi-path recall retrieval enhancement generation device based on real-time data indexing according to an embodiment of this application. As shown in Figure 4, the device includes:

[0098] The crawling unit 410 is used to respond to retrieval needs by setting at least one data crawling method to obtain real-time data;

[0099] Construction unit 420 is used to construct an index database based on the real-time data;

[0100] The sorting unit 430 is used to set at least one type of retrieval scheme based on the index database and reorder the retrieval results obtained by the retrieval scheme.

[0101] The generation unit 440 is used to generate contextual prompts based on the search results, and generate search text based on the contextual prompts using a large language model.

[0102] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the multi-path recall retrieval enhancement generation device based on real-time data indexing. In other embodiments of this application, the multi-path recall retrieval enhancement generation device based on real-time data indexing may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0103] The information interaction and execution process between the various units in the above-mentioned device are based on the same concept as the method embodiments of this application, and the specific details can be found in the descriptions in the method embodiments of this application, and will not be repeated here.

[0104] Figure 5 illustrates a schematic diagram of an electronic device 10 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0105] As shown in Figure 5, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a multi-way recall retrieval enhancement generation method based on real-time data indexing.

[0108] In some embodiments, the multi-way recall retrieval enhancement generation method based on real-time data indexing can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-way recall retrieval enhancement generation method based on real-time data indexing described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the multi-way recall retrieval enhancement generation method based on real-time data indexing by any other suitable means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0114] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0115] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating enhanced multi-pass recall search results based on real-time data indexing, the method comprising: The method comprises the following steps: in response to a search requirement, setting at least one data crawling method to obtain real-time data; constructing an index database based on the real-time data; based on the index database, setting at least one type of search scheme, and reordering the search results obtained through the search scheme; generating a context prompt based on the search results, and generating search text based on the context prompt through a large language model.

2. The method of claim 1, wherein, The data crawling method includes: web crawler technology, RSS subscription mechanism, API calling interface, web page monitoring algorithm and business data push.

3. The method of claim 1, wherein, After setting at least one data crawling method to obtain real-time data, further comprising: performing one or more of the following on the real-time data: data cleaning on the real-time data; data deduplication on the real-time data; data formatting and special character removal on the real-time data; extracting meta information from the real-time data, and auditing the meta information.

4. The method of claim 1, wherein, The method comprises the following steps: convert the text real-time data into a high-dimensional vector through a text embedding technology; constructing an index database based on the high-dimensional vector and incrementally updating and constructing the index.

5. The method of claim 1, wherein, The search scheme includes: sparse search, dense search, public domain search engine query, and private domain search engine query.

6. The method of claim 1, wherein, The reordering of the search results obtained through the search scheme includes: recall information related to the search requirement from at least one data source or index library; remove duplicates in the search results, fuse and reorder the recalled information, and sort according to the relevance result.

7. The method of claim 1, wherein, The method comprises the following steps: generate the context prompt based on the search results and the search requirement; generate the search text based on the context prompt through at least one sampling and decoding strategy.

8. A multi-path recall retrieval enhancement generation device based on real-time data indexing, characterized in that, The method comprises the following steps: a crawling unit for setting at least one data crawling method to obtain real-time data in response to a search requirement; a construction unit for constructing an index database based on the real-time data; an ordering unit for setting at least one type of search scheme based on the index database, and reordering the search results obtained through the search scheme; a generation unit for generating a context prompt based on the search results, and generating search text based on the context prompt through a large language model.

9. An electronic device, characterized by The method comprises the following steps: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the real-time data index-based multi-path recall search enhancement generation method of any one of claims 1-7.

10. A computer readable medium characterized by The computer readable storage medium stores computer instructions for enabling the processor to execute the real-time data index-based multi-path recall search enhancement generation method of any one of claims 1-7 when executed.

Images