Data retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides a data retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, large models and generative models. According to the specific implementation scheme, in response to received query information, strategy parameters corresponding to the query information are determined, and the strategy parameters comprise a recall strategy and a sorting strategy; determining candidate data subsets from the data set according to a recall strategy; and sorting the candidate data in the candidate data subset according to a sorting strategy to obtain target data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of deep learning, large models, and generative models. More specifically, this disclosure provides a data retrieval method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the development of the internet, enterprises rely on automation and intelligent technologies to optimize user experience and advertising, giving rise to the RAG (Retrieval-Augmented Generation) ecosystem. In the RAG system, the retrieval module becomes a crucial component, aiming to enhance the knowledge of large models and thereby improve the accuracy and efficiency of dialogue generation. Summary of the Invention

[0003] This disclosure provides a data retrieval method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to the first aspect, a data retrieval method is provided, the method comprising: in response to receiving query information, determining strategy parameters corresponding to the query information, wherein the strategy parameters include a recall strategy and a ranking strategy; determining a subset of candidate data from the dataset according to the recall strategy; and ranking the candidate data in the subset of candidate data according to the ranking strategy to obtain target data.

[0005] According to a second aspect, a data retrieval apparatus is provided, comprising: a strategy determination module, configured to determine strategy parameters corresponding to the query information in response to receiving query information, wherein the strategy parameters include a recall strategy and a ranking strategy; a recall module, configured to determine a subset of candidate data from a dataset according to the recall strategy; and a ranking module, configured to rank the candidate data in the subset of candidate data according to the ranking strategy to obtain target data.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the methods provided in this disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program, when executed by a processor, implements the method provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is an exemplary system architecture diagram of a data retrieval method and apparatus applicable according to an embodiment of the present disclosure;

[0012] Figure 2 This is a flowchart of a data retrieval method according to an embodiment of the present disclosure;

[0013] Figure 3 This is a system schematic diagram of a data retrieval method according to an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram of the retrieval module of a RAG system according to an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the change in data volume during a data retrieval process according to an embodiment of the present disclosure;

[0016] Figure 6 This is a block diagram of a data retrieval apparatus according to an embodiment of the present disclosure; and

[0017] Figure 7 This is a block diagram of an electronic device for a data retrieval method according to an embodiment of the present disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] To meet the needs of multiple fields and business scenarios, building a unified search system faces the following technical challenges:

[0020] Multi-domain heterogeneity: The advertising alliance covers a wide range of businesses, including education, entertainment, military, and novels, among others. These sectors have significantly different data formats and needs. Each industry's data includes images, text, and various fields, and each has its own unique business logic and objectives. Traditional search systems cannot adapt to all industries at low cost, resulting in long development cycles and high maintenance difficulty.

[0021] High optimization and upgrade costs: Building a dedicated search system to serve a single industry may negatively impact performance in other areas. For example, optimizing a recall strategy for novels might negatively affect search performance for education-related categories. Therefore, it's necessary to achieve adaptive and highly customizable search strategies while maintaining business isolation.

[0022] Rapid response to demands: As business needs constantly change, search strategies need to be adjusted flexibly. Search systems developed with fixed code are unresponsive, increasing development and maintenance costs.

[0023] To address the aforementioned issues, embodiments of this disclosure provide a data retrieval method that achieves cross-domain retrieval by constructing an adaptive, general-purpose retrieval system. This system employs a wide data table design and flexible retrieval strategies to support intelligent retrieval needs across multiple domains and scenarios. By providing a combination of basic and vertically specialized strategies, the system ensures universality while allowing each business domain to optimize its strategies as needed.

[0024] The application scenarios of the embodiments disclosed herein cover multi-business intelligent agents in advertising networks, such as novel recommendations, travel guides, and educational dialogue content retrieval. With this retrieval system, business departments in various industries can easily achieve dynamic switching of recall, ranking, filtering, and multi-path recall strategies, reducing development costs and improving retrieval performance.

[0025] The embodiments of this disclosure can be applied to the following fields and projects:

[0026] Lead matching: Improve ad matching efficiency through intelligent retrieval, thereby enhancing user experience and conversion rates.

[0027] Educational marketing dialogue: Retrieve relevant content based on user queries to improve dialogue quality.

[0028] Tourist navigation system: Supports intelligent retrieval of multimedia information, such as tourist information queries based on images and text, enhancing user experience.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0031] Figure 1 This is a schematic diagram of an exemplary system architecture for applying data retrieval methods and apparatus according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, etc.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as search results) to the terminal devices.

[0035] The data retrieval method provided in this disclosure can generally be executed by server 105. Correspondingly, the data retrieval device provided in this disclosure can generally be located in server 105. The data retrieval method provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the data retrieval device provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0036] Figure 2 This is a flowchart of a data retrieval method according to an embodiment of the present disclosure.

[0037] like Figure 2As shown, the data retrieval method 200 includes operations S210 to S230.

[0038] In operation S210, in response to receiving query information, the strategy parameters corresponding to the query information are determined, wherein the strategy parameters include recall strategy and ranking strategy.

[0039] The query information can be a question entered by the user. Based on the query information, the type of business the user wants to query can be determined. Business types can include multiple industry sectors, such as education, entertainment, tourism, military, and novels. For different business types, corresponding strategy parameters can be preset, which can include recall strategies and ranking strategies.

[0040] After determining the business type based on the query information, the corresponding recall and ranking strategies can be determined.

[0041] Recall strategies can include recall path priorities, and recall paths can include inverted index recall paths and vector index recall paths. For example, in an education business, where business data includes textbook content and teaching videos, the recall strategy could include prioritizing inverted index recall followed by vector index recall. In a tourism business, the recall strategy could include prioritizing vector index recall followed by inverted index recall.

[0042] Vector index recall can also include image recall (query->image), text recall (query->text), and similar question recall (query->query). When the business type is tourism, the recall path priority from high to low can be: image recall, text recall, inverted index recall, etc.

[0043] The sorting strategy can include sorting rule priority, and sorting rules can include semantic relevance sorting rules, keyword relevance sorting rules, tag relevance sorting rules, business type custom sorting rules, etc.

[0044] For example, when the business type is novel writing, the priority of the sorting rules from high to low can be as follows: keyword relevance sorting rule, semantic relevance sorting rule, tag relevance sorting rule, etc.

[0045] In operation S220, a subset of candidate data is determined from the dataset according to the recall strategy.

[0046] Based on the recall path priority in the recall strategy, multiple recall paths with the highest priority (e.g., the top 3) can be selected for parallel recall. Mixing multiple recall paths for parallel recall can improve the coverage of recall data.

[0047] For each recall path, a corresponding subset of candidate data can be retrieved from the dataset. For example, when the business type is tourism, image recall, text recall, and inverted index recall can be selected for recall, resulting in a subset of candidate image data, a subset of candidate text data, and a subset of candidate keyword data.

[0048] In operation S230, the candidate data in the candidate data subset is sorted according to the sorting strategy to obtain the target data.

[0049] For example, when the business type is novel business, the sorting can be done first according to the keyword relevance sorting rule. If there are tied data in the keyword relevance sorting results, the tied data can be sorted according to the semantic relevance sorting rule, and so on. Then, the sorting can be done according to the tag relevance, etc., to obtain the final sorting result.

[0050] You can select the top few (e.g., the top 10) candidate data from the final sorting results as the target data. The target data can be returned to the user, or it can be provided to a large language model to generate response information, and then the target data and response information can be returned to the user together.

[0051] In addition, the sorting strategy also includes custom sorting rules for different business types. For example, in the novel business type, sorting can be customized according to reading time, while in the education business type, sorting can be customized according to content difficulty, etc.

[0052] According to embodiments of this disclosure, in response to receiving query information, corresponding strategy parameters are determined. The strategy parameters include a recall strategy and a ranking strategy. Candidate data is recalled according to the recall strategy, and the candidate data is ranked according to the ranking strategy to obtain target data. This can improve the accuracy and personalization of search results and enhance user experience.

[0053] According to embodiments of this disclosure, different strategy parameters are configured for different business scenarios in different fields. Different strategy parameters are used to control the recall strategy and ranking strategy in the corresponding scenario. Therefore, cross-domain and business scenario retrieval can be achieved in one retrieval system without the need for independent development of each field's business, thus reducing development and maintenance costs.

[0054] Figure 3 This is a system schematic diagram of a data retrieval method according to an embodiment of the present disclosure.

[0055] like Figure 3As shown, the system in this embodiment can be an adaptive general retrieval system tailored to the needs of multiple domains and business scenarios. These multiple domains and business scenarios include, for example, novels, tourism, and education, with business data in each domain including questions, answers, text content, images, and summary content. The system in this embodiment includes a database construction module 310, a retrieval module 320, and a strategy parameter configuration module 330.

[0056] According to embodiments of this disclosure, the database construction module 310 is configured to acquire business data of multiple business types; extract content from the business data and map it to fields of a data table; convert the business data into vectors and store them in a vector library; generate inverted indexes and vector indexes for specific fields in the data table; and extract at least one of the summaries and tags of the business data.

[0057] In multi-domain business operations, the data formats and structures vary significantly across different industries. To address the issue of data inconsistency, the database construction module 310 in this embodiment uses a unified data table, namely a wide table, to build an index database, ensuring that data from different business domains can be efficiently integrated and retrieved.

[0058] Wide tables have a unified field structure that can cover common data types in various business areas (such as text, images, metadata, etc.) and cover multiple commonly used fields, such as title, description, category tags, images, time, etc. It also reserves custom fields for specific industries, enabling the integration of core fields from various industries into a single wide table, thus achieving integrated management of data from multiple fields.

[0059] For example, for a novel business, fields such as title, summary, Q&A, and chapter content are included. For a tourism business, tourist destinations can be mapped to titles, attraction descriptions can be mapped to summaries, and so on.

[0060] The database creation module 310 supports the construction and sharing of cross-domain indexes. For example, it can generate various index structures based on specific fields in a wide table, including inverted indexes, vector indexes, and forward indexes.

[0061] An inverted index maps each word in a field to a collection of documents containing that word. It is suitable for text and keyword retrieval and can improve the speed of text queries.

[0062] Forward sorting information refers to storing detailed information for each document. This detailed information can be attributes stored in a wide table, and personalized sorting and result display can be achieved by filtering by attributes.

[0063] Vector indexing refers to transforming business data into vectors and storing them as a vector library to support cross-domain semantic retrieval. The vector library supports the storage of multimodal data, such as text, images, and audio, ensuring sufficient flexibility for the system when processing unstructured data. Text and image vectorization can be achieved by calling model services, for example, by using pre-trained models to convert text and image data into vectors. Vectors in the vector library can include question vectors, answer vectors, image vectors, and text vectors, among others.

[0064] The construction of a multi-layered index structure ensures fast response times across different retrieval scenarios. For example, when processing a query->content request, the inverted index is used first to match keyword results. When processing a query->image request, the vector index is used to achieve semantic-level image matching.

[0065] The database creation module 310 can also generate summaries and tags for business data. The summaries and tags can be used by the subsequent retrieval module 320 to assist in the display and sorting of retrieval results.

[0066] Summary generation refers to the process of extracting key information from text and generating a short summary based on a natural language processing model.

[0067] Tag generation refers to automatically identifying key information in text or images and assigning them category tags.

[0068] During the database construction process, the database construction module 310 can be configured to perform data preprocessing tasks, including deduplication and tag generation, to improve data quality.

[0069] To ensure data security and privacy across multiple business scenarios, the data tables and databases built by the database creation module 310 can employ strict data isolation and access control strategies. Business data isolation means that data from different business modules are logically stored separately to avoid data contamination and conflicts. Access control refers to ensuring that business departments can only access their authorized data through API (Application Programming Interface) keys and permission management mechanisms.

[0070] According to embodiments of this disclosure, a service type is determined based on query information; and strategy parameters are determined based on the service type.

[0071] After a user inputs query information, the system can identify the domain, industry, or business type to which the query belongs. Based on the domain, industry, or business type, pre-configured strategy parameters corresponding to the query information can be determined. These strategy parameters can be configured in the strategy parameter configuration module 330. The strategy parameters include recall and ranking strategies, used to control the recall and ranking processes of the retrieval module 320.

[0072] The retrieval module 320 includes a recall submodule and a sorting submodule. The recall submodule includes streaming recall and multi-channel hybrid recall. Streaming recall refers to real-time recall of newly generated data from the database creation module. Hybrid recall refers to multi-channel mixed recall.

[0073] Recall methods (recall paths) can include inverted index recall and vector index recall. Vector index recall can include image recall, text recall, question recall, etc.

[0074] Inverted index recall refers to quickly locating a set of candidate data containing keywords based on the query fields entered by the user.

[0075] Vector index recall refers to semantic retrieval achieved through vector indexes. For example, the ANN (Approximate Nearest Neighbor) algorithm is used to match the potential relationship between user query information and data to achieve efficient semantic retrieval.

[0076] The multi-path hybrid recall strategy refers to supporting multiple recall paths to work in parallel, and the priority of each recall path can be controlled by parameters.

[0077] According to embodiments of this disclosure, multiple recall paths are determined based on recall path priority; for inverted index recall paths, a first candidate data subset is recalled from a data table based on the keywords of the query information; for vector index recall paths, a second candidate data subset is recalled from a vector database based on the feature vectors of the query information.

[0078] In different business scenarios, the strategy parameter configuration module 330 can configure different strategy parameters to control the selection and priority of the recall path. For example, the education business prioritizes inverted index recall, while the entertainment business prioritizes vector index recall. Vector index recall includes image recall, text recall, question recall, etc. The recall paths for the entertainment business, in descending order of priority, can be: image recall, text recall, and inverted index recall.

[0079] Multiple recall paths with the highest priority (e.g., the top 3) can be selected for parallel recall. Mixing multiple recall paths for parallel recall can improve the coverage of recall data.

[0080] For each recall path, a corresponding subset of candidate data can be recalled from the dataset.

[0081] According to embodiments of this disclosure, the strategy parameters further include a model strategy, which determines a neural network model corresponding to the vector index recall path; and uses the neural network model corresponding to the vector index recall path to recall a second subset of candidate data from the vector database.

[0082] Model strategy specifies the vector model used in the vector indexing recall path. For example, in image recall (query->image), a multimodal model, such as the CLIP (Contrastive Language-Image Pre-Training) model, is used. In text recall, a natural language processing model, such as the ERNIE (Enhanced Language Representation with Informative Entities) model, is used.

[0083] The sorting submodule includes coarse sorting and fine sorting. Coarse sorting includes real-time filtering and coarse sorting. The recall results of the recall submodule are filtered in real-time and coarse sorted to ensure that the final returned candidate results have high business relevance.

[0084] The filtering function performs real-time filtering on the recalled data, removing redundant and low-relevance data. It can perform initial filtering based on rules such as keyword matching and domain relevance. It also supports blacklist filtering and content sensitivity detection to ensure that the returned results comply with business specifications. Coarse ranking is a basic relevance ranking, which can quickly sort the initial recalled results based on rules such as keyword matching degree and tag weight. The above filtering and coarse ranking can employ streaming processing technology to sort the real-time generated data, improving system response speed.

[0085] After the initial ranking, the system performs a fine ranking of the candidate results to ensure the business accuracy and personalization of the final results.

[0086] Refined ranking includes multi-level refined ranking strategies, which further rank the recall results based on business domain-specific rules and ranking models. When merging results from multiple recall paths, they can be sorted according to predefined priorities to ensure that high-value content is displayed first.

[0087] Fine-grained ranking also includes personalized sorting strategies, which can use machine learning models to achieve personalized sorting based on user behavior data and preferences. Different business scenarios can customize the sorting logic; for example, novels can be sorted by reading time, while educational content can be sorted by content difficulty.

[0088] According to embodiments of this disclosure, candidate data in multiple candidate data subsets corresponding to multiple recall paths are sorted according to the priority of sorting rules to obtain target data.

[0089] The ranking rules include business domain-specific rules and relevance ranking rules. Relevance ranking rules include semantic relevance rules, tag relevance rules, etc. For each business type, the ranking strategy includes a ranking priority corresponding to that business type. When merging the retrieved data from multiple recall paths, the merging and ranking are performed according to the priority of the ranking rules.

[0090] For example, when the business type is novel reading, the priority of the sorting rules from high to low can be as follows: keyword relevance sorting rule, then reading time sorting rule. During fine-tuning, the multi-way recall results can be merged and sorted first according to the keyword relevance sorting rule. If there are tied data in the merged sorting results, they can then be sorted according to reading time.

[0091] From the sorted results obtained after fine sorting, the top N (e.g., the top 10) data points can be selected as target data. Target data can be returned to the user or provided as RAG reference information to the large model, which will then generate response information. Alternatively, both the target data and the response information generated by the large model can be returned to the user.

[0092] When no reference content can be retrieved, new data can be generated using a large model and some domain knowledge. The generated data can then be stored for future use.

[0093] The strategy parameter configuration module 330 is used to configure corresponding strategy parameters for different business types, adapting to the diverse needs of businesses across multiple domains. The strategy parameter configuration module 330 supports dynamic parameterized control and open interfaces, allowing businesses to flexibly adjust their search strategies.

[0094] Dynamic parameter control refers to the ability to adjust recall, filtering, and ranking strategies by passing parameters via API, thereby dynamically optimizing system behavior. For example, based on the strategy parameters in the request, the system can dynamically switch to the corresponding recall and ranking strategies.

[0095] Open API support means the system provides open API interfaces, allowing business users to implement custom logic by calling these interfaces. It also supports adding custom filters and sorting rules to ensure that specific business needs are met.

[0096] The strategy parameter configuration module 330 ensures flexible switching of search strategies across different scenarios and reduces regression costs after system optimization. By combining basic and specialized strategies, it meets the needs of various scenarios and provides highly precise and recall-based search services.

[0097] The combination of basic strategies and vertical specialization strategies refers to the system's design that combines basic strategies with vertical specialization strategies to support multi-domain retrieval needs.

[0098] The basic strategy refers to a set of universal recall, ranking, and filtering strategies built into the system, covering the basic needs of most business scenarios. These strategies are used as default configurations for various business verticals.

[0099] Vertical specialization strategies refer to systems that allow business units to customize personalized strategies through open interfaces for certain fields or specific needs. For example, education businesses can adjust the model selection for recall strategies, while tourism businesses can optimize sorting logic to highlight popular attractions.

[0100] This combination approach not only ensures the system's versatility but also enhances its adaptability to various business scenarios. When using basic strategies, business users can flexibly add specialized strategies to achieve rapid optimization without developing new code.

[0101] The strategy parameter configuration module 330 supports dynamic parameter configuration and automated strategy switching.

[0102] Dynamic parameter configuration refers to allowing business users to pass parameters in real time via API, thereby dynamically controlling the execution of retrieval strategies. The system supports dynamic parameter configuration across different dimensions; for example, model selection parameters: different vector models are selected based on the specific business type of the query.

[0103] Business users can adjust the recall path and sorting priority by passing parameters, such as increasing the sorting priority of specific products during promotional periods. Dynamic parameter configuration enables rapid response to business needs, ensuring that business departments can flexibly adjust search strategies at different times or under special circumstances.

[0104] The strategy parameter configuration module 330 provides a strategy control interface, supports dynamic parameter distribution, and enables the optimization and updating of the retrieval system without code development.

[0105] The strategy control interface allows business units to define their own search strategies via open APIs, including logic for recall, ranking, and filtering. It controls the priority of strategies across different business modules through parameterization, avoiding conflicts and ensuring a smooth transition when switching strategies.

[0106] Dynamic parameter delivery means the system supports dynamically loading different search strategies upon query request. Appropriate search paths and models are selected through strategy parameters. Strategies can be shared across multiple business modules, and parameters can be dynamically adjusted based on business objectives.

[0107] According to embodiments of this disclosure, the data retrieval method system is an adaptive retrieval system that supports automatic adjustment of retrieval strategies based on different business scenarios and domains, achieving integrated cross-domain retrieval services. The system employs a modular architecture, decoupling modules such as database creation, recall, sorting, and filtering. Each module can be independently updated and optimized, ensuring system flexibility. The system supports data and model isolation through parameter control, improving system stability and scalability. Business departments can optimize the system through parameter adjustments and strategy deployment, requiring no additional code development and resulting in low development costs.

[0108] Figure 4 This is a schematic diagram of the retrieval module of a RAG system according to an embodiment of the present disclosure.

[0109] like Figure 4 As shown, this embodiment includes a search entry point 410, a recall module 420, and a sorting module 430. The search entry point 410 can provide search entry points for multiple business areas, such as search entry points for novels, tourism, and education.

[0110] The recall module 420 provides multiple recall paths, such as inverted index recall paths, question-article recall paths, question-image recall paths, and similar question recall paths. These recall paths are shared across different domains. For each business type, a recall strategy can be pre-configured, including the recall path and its priority for that business type. Therefore, for each business type, when a query request is generated, by loading the strategy parameters corresponding to that business type, the system can automatically switch to the recall strategy for that business type and select the recall path indicated in the strategy to recall candidate data.

[0111] The sorting module 430 provides multiple sorting rules, such as semantic relevance, tag relevance, and keyword relevance. These sorting rules are shared across different domains. Furthermore, specific proprietary sorting rules are configured for each business type. A sorting strategy can be pre-configured for each business type, including the sorting rules and their priorities. Therefore, for each business type, after obtaining the candidate data retrieved by the recall path for that business type, the candidate data can be sorted according to the sorting rules and priorities indicated by the sorting strategy to obtain the target data.

[0112] The embodiments disclosed herein, through a modular retrieval framework and entry point, ensure that each business module can dynamically load and customize retrieval strategies, avoiding the problem of high system regression costs. Simultaneously, it supports complete isolation of data, models, and business logic, ensuring that the system achieves optimal scalability, flexibility, and stability.

[0113] Figure 5 This is a schematic diagram illustrating the change in data volume during a data retrieval process according to an embodiment of the present disclosure.

[0114] like Figure 5 As shown, data retrieval is a data filtering process, selecting target novels from a dataset of tens of millions of data points. The data changes in a funnel shape.

[0115] Dataset 510 includes data from various business types, such as novels, tourism, and education. Data from each business type can be isolated and stored in different storage spaces through parameters (such as type parameters) to form datasets for each business type, such as novel dataset 520 and tourism dataset.

[0116] Upon receiving a query request for a specific business type (such as novels), a search can be performed on the novel dataset 520. A candidate novel set 530 is then retrieved from the novel dataset 520 according to a recall strategy corresponding to the novel business type. Next, the candidate novels in the candidate novel set 530 are sorted according to a ranking strategy corresponding to the novel business type, resulting in a novel ranking result 540. Based on business requirements, the top N (e.g., 10) novels from the novel ranking result can be selected as the target novel 550.

[0117] The target novel 550 can be returned to the user, or it can be used as RAG reference information and passed to the large model, which then generates a response. Both the response generated by the large model and the target novel 550 can be returned to the user.

[0118] The embodiments disclosed herein can meet the retrieval needs of multiple fields and multiple business scenarios, improve the accuracy and personalization of retrieval results, and enhance user experience.

[0119] According to embodiments of this disclosure, this disclosure also provides a data retrieval device.

[0120] Figure 6 This is a block diagram of a data retrieval apparatus according to an embodiment of the present disclosure.

[0121] like Figure 6 As shown, the data retrieval device 600 includes a strategy determination module 610, a recall module 620, and a sorting module 630.

[0122] The strategy determination module 610 is used to determine the strategy parameters corresponding to the query information in response to the received query information. The strategy parameters include a recall strategy and a ranking strategy.

[0123] The recall module 620 is used to determine a subset of candidate data from the dataset according to the recall strategy.

[0124] The sorting module 630 is used to sort the candidate data in the candidate data subset according to the sorting strategy to obtain the target data.

[0125] According to embodiments of this disclosure, the recall strategy includes a recall path priority. The recall module 620 includes a recall path determination unit and a recall unit.

[0126] The recall path determination unit is used to determine multiple recall paths based on the recall path priority.

[0127] The recall unit is used to determine a subset of candidate data from the dataset for each recall path.

[0128] The dataset includes data tables and a vector database, and the recall paths include inverted index recall paths and vector index recall paths. The recall unit includes a first recall subunit and a second recall subunit.

[0129] The first recall subunit is used to recall the first candidate data subset from the data table based on the keywords of the query information for the inverted index recall path.

[0130] The second recall subunit is used to recall a second subset of candidate data from the vector database based on the feature vector of the query information for the vector index recall path.

[0131] The strategy parameters also include the model strategy. The second recall subunit is used to determine the neural network model corresponding to the vector index recall path based on the model strategy; and to recall a second subset of candidate data from the vector database using the neural network model corresponding to the vector index recall path.

[0132] The data retrieval device 600 also includes an acquisition module, a mapping module, a vectorization module, an index generation module, and an extraction module.

[0133] The acquisition module is used to acquire business data for multiple business types.

[0134] The mapping module is used to extract content from business data and map it to fields in a data table.

[0135] The vectorization module is used to convert business data into vectors and store them in a vector library.

[0136] The index generation module is used to generate inverted indexes and vector indexes for specific fields in a data table.

[0137] The extraction module is used to extract at least one of the summary and tags from the business data.

[0138] The sorting strategy includes sorting rule priorities. The sorting module 630 is used to sort the candidate data in multiple candidate data subsets corresponding to multiple recall paths according to the sorting rule priorities to obtain the target data; wherein, the sorting rules include at least one of relevance sorting rules and sorting rules corresponding to business types.

[0139] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0142] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as data retrieval methods. For example, in some embodiments, the data retrieval method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data retrieval method by any other suitable means (e.g., by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0148] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0149] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0150] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data retrieval method, comprising: In response to receiving the query information, determining a strategy parameter corresponding to the query information, wherein the strategy parameter includes a recall strategy and a sorting strategy; Determining a candidate data subset from the data set according to the recall strategy; and According to the sorting strategy, the candidate data in the candidate data subset are sorted to obtain target data.

2. The method according to claim 1, wherein: The recall strategy includes a recall path priority; and determining a candidate data subset from a data set according to the recall strategy includes: Determining a plurality of recall paths according to the recall path priorities; and For each recall path, a candidate data subset is determined from the data set through the recall path.

3. The method according to claim 2, wherein: The data set includes a data table and a vector database, and the recall path includes an inverted index recall path and a vector index recall path; For each recall path, determining a candidate data subset from the data set through the recall path includes: For the inverted index recall path, recall a first candidate data subset from the data table according to the keyword of the query information; For the vector index recall path, a second candidate data subset is recalled from the vector database according to the feature vector of the query information.

4. The method according to claim 3, wherein: The strategy parameters also include a model strategy; and recalling the second candidate data subset from the vector database through the vector index recall path includes: Determining a neural network model corresponding to the vector index recall path according to the model strategy; and A second candidate data subset is recalled from the vector database using a neural network model corresponding to the vector index recall path.

5. The method according to claim 3, further comprising: Obtain business data for multiple business types; Extracting content from the business data and mapping it to fields in the data table; Convert the business data into vectors and store them in the vector library; Generate an inverted index and a vector index for a specific field in the data table; and At least one of a summary and a tag of the business data is extracted.

6. The method according to claim 5, wherein: The sorting strategy includes a sorting rule priority; and the candidate data in the candidate data subset are sorted according to the sorting strategy to obtain the target data, including: sorting the candidate data in the plurality of candidate data subsets corresponding to the plurality of recall paths respectively according to the sorting rule priority to obtain target data; The sorting rule includes at least one of a relevance sorting rule and a sorting rule corresponding to the business type.

7. The method according to any one of claims 1 to 6, wherein: In response to receiving the query information, determining a policy parameter corresponding to the query information includes: Determine the business type according to the query information; and The policy parameters are determined according to the service type.

8. A data retrieval device, comprising: A strategy determination module, configured to determine, in response to receiving query information, strategy parameters corresponding to the query information, wherein the strategy parameters include a recall strategy and a sorting strategy; A recall module, configured to determine a candidate data subset from the data set according to the recall strategy; and A sorting module is used to sort the candidate data in the candidate data subset according to the sorting strategy to obtain target data.

9. The device according to claim 8, wherein: The recall strategy includes a recall path priority; the recall module includes: a recall path determining unit, configured to determine a plurality of recall paths according to the recall path priorities; and The recall unit is used to determine, for each recall path, a candidate data subset from the data set through the recall path.

10. The device according to claim 9, wherein: The data set includes a data table and a vector database, the recall path includes an inverted index recall path and a vector index recall path; the recall unit includes: A first recall subunit, configured to recall a first candidate data subset from the data table according to a keyword of the query information based on the inverted index recall path; The second recall subunit is used to recall the path of the vector index and recall a second candidate data subset from the vector database according to the feature vector of the query information.

11. The device according to claim 10, wherein: The strategy parameters also include a model strategy; the second recall subunit is used to determine the neural network model corresponding to the vector index recall path according to the model strategy; and use the neural network model corresponding to the vector index recall path to recall a second candidate data subset from the vector database.

12. The device according to claim 10, wherein: The device also includes: An acquisition module, used to acquire business data of multiple business types; A mapping module, used to extract content from the business data and map it to the fields of the data table; A vectorization module, used for converting the business data into vectors and storing them in the vector library; An index generation module, used to generate an inverted index and a vector index for a specific field in the data table; and The extraction module is used to extract at least one of the summary and the label of the business data.

13. The device according to claim 12, wherein: The sorting strategy includes a sorting rule priority; the sorting module is used to sort the candidate data in the multiple candidate data subsets corresponding to the multiple recall paths respectively according to the sorting rule priority to obtain the target data; wherein the sorting rule includes a relevance sorting rule and at least one of the sorting rules corresponding to the business type.

14. The device according to any one of claims 8 to 13, wherein: The strategy determination module includes: a service type determination unit, configured to determine the service type according to the query information; and A policy determination unit is used to determine the policy parameters according to the service type.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images