A distributed retrieval method and apparatus

Abstract

This invention discloses a distributed retrieval method and apparatus. The distributed retrieval method includes: obtaining an additional dataset and two or more sub-datasets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more sub-datasets are obtained based on the dataset to be retrieved; based on a query request input by a user, performing searches in the additional dataset and the two or more sub-datasets respectively, obtaining a first retrieval result corresponding to the additional dataset and a second retrieval result corresponding to each sub-dataset; wherein the first retrieval result is obtained before the second retrieval result. This invention can improve retrieval accuracy and retrieval response speed.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, and in particular to a distributed retrieval method and apparatus. Background Technology

[0002] With the development of information technology, new technologies such as 5G networks, cloud services, the Internet of Things, and big data are gradually being widely applied, bringing with them massive amounts of data and various challenges. In the context of massive amounts of customer data, conventional retrieval technologies are insufficient to meet the needs of processing such data. For example, when retrieving specific data from massive datasets, slow response times and high error rates can occur, severely impacting user experience and query efficiency.

[0003] Therefore, current methods for retrieving large amounts of data suffer from slow response times and high error rates. Summary of the Invention

[0004] In view of the above problems, the present invention proposes a distributed retrieval method and apparatus, which can improve the accuracy of data retrieval and the retrieval response speed.

[0005] In a first aspect, this application provides the following technical solution through an embodiment:

[0006] A distributed retrieval method includes: obtaining an additional dataset and two or more sub-datasets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more sub-datasets are obtained based on the dataset to be retrieved; based on a query request input by a user, performing searches in the additional dataset and the two or more sub-datasets respectively, to obtain a first retrieval result corresponding to the additional dataset and a second retrieval result corresponding to each of the sub-datasets; wherein the first retrieval result is obtained before the second retrieval result.

[0007] Optionally, obtaining the additional dataset includes:

[0008] Based on the historical search information corresponding to the dataset to be searched, one or more historical search items are obtained; the historical search items are data items in the dataset to be searched; for each historical search item, related data items associated with the historical search item are found in the dataset to be searched; based on the historical search items and the related data items, an additional dataset is obtained.

[0009] Optionally, the step of searching for associated data items related to the historical search item in the dataset to be searched includes:

[0010] Based on the attribute information of each data item in the dataset to be retrieved and the attribute information of the historical search items, the similarity between the historical search items and each data item in the dataset to be retrieved is obtained; data items with similarity greater than a preset similarity threshold are taken as the associated data items.

[0011] Optionally, the attribute information of the data item includes the query time interval with the historical search item; wherein the query time interval is negatively correlated with similarity.

[0012] Optionally, obtaining the similarity between the historical search items and each data item in the dataset to be retrieved, based on the attribute information of each data item in the dataset to be retrieved and the attribute information of the historical search items, includes:

[0013] The attribute information of each data item in the dataset to be retrieved is input into a preset neural network model to obtain the first encoded data corresponding to each data item; the attribute information of the historical retrieval items is input into the neural network model to obtain the second encoded data; based on the first encoded data and the second encoded data, the similarity between the historical retrieval items and each data item in the dataset to be retrieved is determined.

[0014] Optionally, obtaining the additional dataset based on the historical search items and the associated data items further includes:

[0015] The index order of the associated data items is configured based on the similarity score; wherein, for the same query request, the index order of the associated data items is after the historical search items; the additional dataset is obtained based on the historical search items and the associated data items for which the index order configuration has been completed.

[0016] Optionally, obtain more than two subsets of the dataset, including:

[0017] Obtain two or more historical sub-datasets; based on the index order of the associated data items in the supplementary dataset, update the two or more historical sub-datasets to obtain the two or more sub-datasets.

[0018] Optionally, updating the two or more historical sub-datasets based on the index order of related data items in the supplementary dataset to obtain the two or more sub-datasets includes:

[0019] In the two or more historical subsets, determine the first target subset where the historical retrieval item is located, and one or more second target subsets where the associated data item corresponding to the historical retrieval item is located; divide the associated data item from the second target subset to the first target subset, and configure the index order.

[0020] Secondly, based on the same inventive concept, this application provides the following technical solution through an embodiment:

[0021] A distributed retrieval device, comprising:

[0022] A subset acquisition module is used to acquire an additional dataset and two or more subsets; wherein the data volume of the additional dataset is smaller than that of the dataset to be retrieved, and both the additional dataset and the two or more subsets are obtained based on the dataset to be retrieved; a retrieval module is used to perform retrievals in the additional dataset and the two or more subsets respectively based on a query request input by the user, to obtain a first retrieval result corresponding to the additional dataset and a second retrieval result corresponding to each subset; wherein the first retrieval result is obtained before the second retrieval result.

[0023] Thirdly, based on the same inventive concept, this application provides the following technical solution through an embodiment:

[0024] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0025] This invention provides a distributed retrieval method and apparatus. By dividing the dataset to be retrieved into two or more subsets, retrieval is performed on different subsets within a time limit, resulting in distributed retrieval and improved retrieval efficiency and accuracy. Furthermore, this embodiment also constructs a corresponding supplementary dataset and performs retrieval within it. Since the supplementary dataset is smaller than the dataset to be retrieved, it prioritizes providing users with partial retrieval results (the first retrieval result), enabling rapid retrieval and response. The second retrieval result obtained from retrieving from two or more subsets provides a complete and comprehensive retrieval result, ensuring accuracy. Therefore, this invention improves retrieval accuracy and response speed.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0028] Figure 1 A flowchart of a distributed retrieval method according to an embodiment of the present invention is shown;

[0029] Figure 2 A schematic diagram of a distributed retrieval device according to an embodiment of the present invention is shown. Detailed Implementation

[0030] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0031] Please see Figure 1 This invention provides a distributed retrieval method, which includes the following steps:

[0032] Step S10: Obtain an additional dataset and two or more sub-datasets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more sub-datasets are obtained based on the dataset to be retrieved;

[0033] Step S20: Based on the query request input by the user, perform searches in the additional dataset and the two or more sub-datasets respectively to obtain a first search result corresponding to the additional dataset and a second search result corresponding to each sub-dataset; wherein, the first search result is obtained before the second search result.

[0034] In this embodiment, steps S10-S20 described above can divide the dataset to be retrieved into multiple smaller subsets during the retrieval process. This allows the retrieval process to be distributed to different processing units. Simultaneously, each subset has a smaller data volume, improving retrieval accuracy and obtaining a more accurate second retrieval result. Furthermore, this embodiment adds an additional dataset, which is also constructed based on the dataset to be retrieved, but its data volume is significantly smaller. This enables a faster response during the retrieval process, resulting in a first retrieval result. Therefore, the first and second retrieval results can be output sequentially and presented to the user, effectively improving the query efficiency and accuracy of the dataset to be retrieved. The specific implementation of each step in this embodiment will be explained in more detail below.

[0035] Step S10: Obtain an additional dataset and two or more sub-datasets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more sub-datasets are obtained based on the dataset to be retrieved.

[0036] In step S10, the dataset to be retrieved is the total dataset, which can be one or more of the following: industry terminology data, news data, video data, academic paper data, etc., where each piece of data can be recorded as a data item. For example, if the dataset to be retrieved is industry terminology data, then each piece of data can specifically be the explanation of each term in that industry; if the dataset to be retrieved is news data, then each data item can specifically be the news title and the corresponding news content; if the data to be retrieved is video data, then each data item can specifically be the video title, the corresponding video content, and video parameter information, such as resolution, refresh rate, playback address, etc.; if the data to be retrieved is academic paper data, then each data item can specifically be the paper title, the corresponding paper content, and paper information. Of course, the implementation of the dataset to be retrieved is only a limited number of implementation methods listed, and can be referred to in specific implementations, which will not be elaborated further.

[0037] Furthermore, each data item in the dataset to be retrieved can be configured with a corresponding title and attribute information. The title can be used to represent a summary of the data item's content, while the attribute information can be used to represent some basic information about the data item, such as the data item's inclusion time, retrieval time, number of retrievals, industry sector, and data type. This can improve processing efficiency in subsequent processing. For example, when using a preset neural network model to process each data item, processing can be performed based on the title and title attributes of each data item, thereby improving processing efficiency and reducing computational load.

[0038] Each piece of data in the supplementary dataset comes from the dataset to be retrieved. Specifically, the supplementary dataset can be determined based on the user's historical queries in the dataset to be retrieved, thus ensuring that the supplementary dataset can be continuously updated and that its data volume is much smaller than that of the dataset to be retrieved, enabling a fast response during the retrieval process.

[0039] Understandably, one way to obtain additional datasets is as follows:

[0040] If the dataset to be retrieved has not been retrieved before, a certain number of data items can be randomly selected from the dataset to be retrieved to form the dataset to be retrieved.

[0041] If the dataset to be retrieved has been retrieved multiple times, an additional dataset can be constructed as follows to ensure that the additional dataset is more accurate and reliable.

[0042] First, based on the historical search information corresponding to the search dataset, one or more historical search terms are obtained; historical search terms are data items in the dataset to be searched. Historical search information may include the searched data items and their corresponding attribute information. Since historical search terms can express the user's recent search habits and may implicitly imply the relationships between data items, the supplementary dataset constructed from historical search information can more accurately represent changes in the user's search habits and current search habits over a long period of time, while also extracting the relationships between various data items. In other words, one or more historical search terms can be added to the supplementary dataset as data items, thus making the supplementary dataset more accurate and reliable.

[0043] Then, for each historical search term, the associated data items are searched in the dataset to be searched. Historical search terms are the data items retrieved from the dataset after a user entered a query in the past. These historical search terms can be defined as data items retrieved within a fixed time period, such as within a week, a month, three months, etc. Of course, this fixed time period can be freely set by the user. Associated data items are data items that have a relationship with historical search terms; a historical search term can have one or more associated data items. The associated data items corresponding to historical search terms can also be added to the supplementary dataset, thereby further improving the completeness and accuracy of the supplementary dataset and improving search efficiency. The relationship between historical search terms and associated data items can be implemented in various ways.

[0044] For example, this can be achieved manually or automatically when constructing the database to be searched. One or more related data items can be labeled for a historical search item. In the process of labeling related data items, the strength of the correlation can also be labeled. This way, if a historical data item is retrieved during the search process, the related data items corresponding to that historical data item can be indexed from the strongest to the weakest correlation, thereby improving the indexing efficiency and accuracy.

[0045] For example, by calculating the similarity between historical search terms and other data terms, related data terms similar to historical search terms can be identified. Understandably, firstly, based on the attribute information of each data term in the dataset to be retrieved and the attribute information of historical search terms, the similarity between historical search terms and each data term in the dataset to be retrieved can be obtained. Then, data terms with a similarity greater than a preset similarity threshold are considered related data terms, which avoids too many related data terms or the construction of excessively long index chains. Here, similarity calculation can be performed using the encoded data between two data terms; during the database construction process, corresponding encoded data can be matched for each data term according to preset rules; the difference between the encoded data of two data terms can characterize the similarity between the two data terms. The encoded data should include those unchanging attributes in the attribute information of the data terms, and may also include attributes that change based on user search time and search frequency. Among them, the inclusion time, industry field, and data type are the unchanging attributes of the data items, while the search time and number of searches are the changing attributes of the data items. During the search process, the user's search operation will change the changing attributes of the data items, thereby changing the relationship between the data items. However, each data item still has unchanging attributes. Therefore, the basic relationship between the data items will not change significantly. Through the continuous search of the dataset by the user, the relationship can be continuously fine-tuned and corrected at different times, thereby avoiding over-correction.

[0046] Furthermore, to more accurately characterize the correlation between two data items, the similarity between them can be determined by comprehensively considering the retrieval time. That is, in some implementations, the attribute information of the data items can also include the query time interval between the data item and historical retrieval items, which is negatively correlated with similarity. By jointly determining the similarity using the query time interval and the encoded data, the similarity between historical retrieval items and other data items can be guaranteed. In implementation, the first similarity obtained based on the encoded data and the second similarity obtained based on the query time interval can be normalized, and then summed to obtain the final similarity between the historical retrieval item and a certain data item.

[0047] In addition, this embodiment also provides another method for determining encoded data and similarity. A preset neural network model can be used for encoding. Specifically, the attribute information of each data item in the dataset to be retrieved is input into the preset neural network model to obtain the first encoded data corresponding to each data item; the attribute information of historical search items is input into the neural network model to obtain the second encoded data. Finally, based on the first and second encoded data, the similarity between the historical search items and each data item in the dataset to be retrieved is determined.

[0048] The pre-trained neural network model can be obtained through training, and the specific training process can be implemented with reference to existing technologies. The type of neural network model is not limited. The input to the neural network model can be the attribute information of data items in the dataset to be retrieved. The attribute information can be converted into attribute vectors as input, and the output encoded data can be a feature vector representing the characteristics of that data item. That is, the first encoded data, i.e., the first feature vector, can be obtained in this way; the second encoded data, i.e., the second feature vector, can be obtained. Then, the similarity between the first feature vector and the second feature vector can be calculated to represent the similarity between historical retrieval items and other data items. Of course, the similarity between the first feature vector and the second feature vector can also be calculated using a trained neural network model.

[0049] For each historical search term, the similarity between the historical search term and each data term in the dataset to be searched can be obtained through the above method. Then, data terms with high similarity to the historical search term can be selected from the dataset to be searched as associated data terms of the historical search term. Finally, an additional dataset is constructed using the historical search term and its corresponding associated data terms.

[0050] In some implementations, to further improve the retrieval efficiency for users when searching additional databases, an index order can be configured for historical search terms and their corresponding related data terms. Specifically, the steps for obtaining additional datasets based on historical search terms and related data terms can include the following processes:

[0051] First, the index order of related data items is configured based on their similarity. For the same query request, the index order of related data items follows that of historical search terms. This means that when a user enters a query and retrieves the same historical search term again, it will be indexed in the supplementary database according to the configured index order, thus improving retrieval efficiency. When configuring the index order, two data items with similar indexes can be stored in a connected storage location, thereby improving retrieval efficiency during computer reading or searching.

[0052] Then, based on historical search terms and associated data items configured with the index order, an additional dataset is obtained. Understandably, as time increases and user queries rise, the similarity between data items in this additional dataset will be corrected and become more accurate. Therefore, over time, the historical search terms in the additional dataset will better match users' search habits, the associated data items corresponding to historical search terms will become more accurate and reliable, and the configured index order will be continuously modified for greater accuracy.

[0053] In this embodiment, two or more subsets are obtained based on the dataset to be retrieved, which is the initial segmentation of the dataset. Specifically, the dataset to be retrieved can be randomly divided according to a preset size to obtain two or more subsets. Alternatively, subsets can be divided based on the domain or type of each data item in the dataset to be retrieved, thereby achieving more accurate retrieval. In this embodiment, two or more subsets can also be obtained by partitioning using the current MapReduce platform. MapReduce is a computing model, framework, and platform for parallel processing of big data. It is a high-performance parallel computing platform based on a cluster. It allows the construction of a distributed and parallel computing cluster containing tens, hundreds, or even thousands of nodes using ordinary commercial servers. It provides a large but well-designed parallel computing software framework that can automatically complete the parallel processing of computing tasks, automatically divide computing data and computing tasks, automatically allocate and execute tasks on cluster nodes, and collect computing results.

[0054] In some implementations, the corresponding index order can be configured within a subset to improve retrieval efficiency within that subset. That is, within each subset, the similarity between each data item and other data items can be calculated.

[0055] The other data items here can be any of the data items in the current subset being processed. After the calculation is complete, a complete index order can be established for each data item. For example, when a subset contains data items A, B, C, D, and E; data item A can correspond to the index order A, C, D, E, B; data item B can correspond to the index order B, C, D, A, E; similarly, data items C, D, and E also have their corresponding index orders. When data item B is retrieved for the first time in this subset, it can be indexed according to data items B, C, D, A, and E; similarly, if data item A is retrieved for the first time in this subset, it can be indexed according to data items A, C, D, E, and B.

[0056] The other data items here can refer to all data items in the dataset to be retrieved. After the calculation is completed, a complete index order can be established for each data item. The index order can be constructed by traversing each subset sequentially; for example, in the first subset, the index order can be constructed for each data item within the scope of the dataset to be retrieved, thereby improving indexing efficiency.

[0057] In some implementations, the subset can be updated after each retrieval or at regular time intervals. If the subset was updated once before the current retrieval, the update process can be as follows:

[0058] Before the current retrieval, a subset of data from the previous retrieval (referred to here as the historical subset for ease of description) can be obtained. Then, based on the index order of related data items in the supplementary dataset, two or more historical subsets are updated to obtain two or more subsets. Specifically, a first target subset containing the historical retrieval item and one or more second target subsets containing the related data items corresponding to the historical retrieval item can be determined from the two or more historical subsets. Then, the related data items are moved from the second target subsets to the first target subset, and the index order is configured. The configuration process for forming two or more subsets can be referred to above when configuring the index order.

[0059] Understandably, when initially splitting the data items to be indexed, no data items were moved between the various subsets. At this point, adjusting the data items between subsets based on the supplementary database can more accurately reflect the correlation between data items and users' search habits, improving search efficiency and accuracy; at the same time, it also avoids a large number of data items being too concentrated in a single subset, which would reduce the efficiency of distributed retrieval.

[0060] Step S30: Based on the query request input by the user, perform searches in the additional dataset and the two or more sub-datasets respectively to obtain the first search result corresponding to the additional dataset and the second search result corresponding to each sub-dataset; wherein the first search result is obtained before the second search result.

[0061] In step S30, the user's query request is input. The system applying this method can then search within the supplementary dataset and two or more subsets to obtain search results. The search logic can be implemented using existing technologies, such as the MapReduce platform, provided that the rules for the subsets and supplementary datasets described above are met. Alternatively, data items can be stored in a "keyword-data" structure to implement indexed queries. In this case, the user's query request should include keywords; there can be one or more keywords, without restriction.

[0062] When outputting search results, the first search result can be prioritized for user viewing. When a second search result is obtained during the viewing process, a prompt message can be generated to ask the user whether to refresh the search results. If the user finds the desired data item in the first search result, the second search result can be left unrefreshed in response to a user's input (no refresh command) or without such a command. In other words, the second search result may not be merged with the first search result, or it may not be displayed at all, avoiding wasted computing resources. If the user does not find the desired data item in the first search result, the first and second search results can be merged in response to a user's input (refresh command); alternatively, the first search result can be refreshed to become the second search result, providing only the second result to the user for viewing, thereby improving viewing efficiency.

[0063] In some implementations, the MapReduce platform can be used for distributed encrypted computation and the creation of more secure distributed indexes, i.e., indexing encrypted data. MapReduce-based data encryption and indexing methods distribute traditional data encryption, querying, and analysis, effectively supporting the encryption and indexing of massive amounts of data in big data environments. Furthermore, based on data sensitivity and importance, different data items can be selectively encrypted, encrypting only sensitive, important, and / or core data as needed, while omitting encryption of insensitive data. This reduces the performance impact of encrypted storage and positively contributes to maintaining high system performance, eliminating the need to decrypt every data attribute during queries and improving query efficiency. Additionally, data items that are not frequently read or retrieved can be encrypted, ensuring security without affecting retrieval efficiency.

[0064] In summary, this embodiment provides a distributed retrieval method. By dividing the dataset to be retrieved into two or more subsets, retrieval is performed on different subsets within a time limit, resulting in distributed retrieval and improved retrieval efficiency and accuracy. Furthermore, this embodiment also constructs a corresponding supplementary dataset and performs retrieval within it. Since the supplementary dataset is smaller than the dataset to be retrieved, it can prioritize providing users with partial retrieval results (the first retrieval result), enabling rapid retrieval and response. The second retrieval result obtained from retrieving from two or more subsets provides complete and comprehensive results, ensuring accuracy. Therefore, the method of this invention improves retrieval accuracy and response speed.

[0065] Please see Figure 2 Based on the same inventive concept, another embodiment of the present invention also provides a distributed retrieval device 300, comprising:

[0066] The subset acquisition module 301 is used to obtain an additional dataset and two or more subsets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and the additional dataset and the two or more subsets are all obtained based on the dataset to be retrieved;

[0067] The retrieval module 302 is used to perform retrievals in the supplementary dataset and the two or more sub-datasets based on the query request input by the user, and obtain a first retrieval result corresponding to the supplementary dataset and a second retrieval result corresponding to each of the sub-datasets; wherein the first retrieval result is obtained before the second retrieval result.

[0068] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0069] Based on the historical search information corresponding to the dataset to be searched, one or more historical search items are obtained; the historical search items are data items in the dataset to be searched; for each historical search item, related data items associated with the historical search item are found in the dataset to be searched; based on the historical search items and the related data items, an additional dataset is obtained.

[0070] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0071] Based on the attribute information of each data item in the dataset to be retrieved and the attribute information of the historical search items, the similarity between the historical search items and each data item in the dataset to be retrieved is obtained; data items with similarity greater than a preset similarity threshold are taken as the associated data items.

[0072] As an optional implementation, the attribute information of the data item includes the query time interval with the historical search item; wherein the query time interval is negatively correlated with similarity.

[0073] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0074] The attribute information of each data item in the dataset to be retrieved is input into a preset neural network model to obtain the first encoded data corresponding to each data item; the attribute information of the historical retrieval items is input into the neural network model to obtain the second encoded data; based on the first encoded data and the second encoded data, the similarity between the historical retrieval items and each data item in the dataset to be retrieved is determined.

[0075] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0076] The index order of the associated data items is configured based on the similarity score; wherein, for the same query request, the index order of the associated data items is after the historical search items; the additional dataset is obtained based on the historical search items and the associated data items for which the index order configuration has been completed.

[0077] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0078] Obtain two or more historical sub-datasets; based on the index order of the associated data items in the supplementary dataset, update the two or more historical sub-datasets to obtain the two or more sub-datasets.

[0079] As an optional implementation, the subset acquisition module 301 is specifically used for:

[0080] In the two or more historical subsets, determine the first target subset where the historical retrieval item is located, and one or more second target subsets where the associated data item corresponding to the historical retrieval item is located; divide the associated data item from the second target subset to the first target subset, and configure the index order.

[0081] Based on the same inventive concept, another embodiment of the present invention also provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in the foregoing method embodiments.

[0082] It should be noted that the readable storage medium provided in this embodiment of the invention, in which the program is executed by the processor, the specific implementation of each step and the resulting technical effects are the same as those in the foregoing method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the foregoing method embodiment.

[0083] The term "and / or" as used herein is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship; the word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of multiple such elements. This invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several of these means can be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A distributed retrieval method, characterized in that, include: Obtain an additional dataset and two or more sub-datasets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more sub-datasets are obtained based on the dataset to be retrieved; The process of obtaining the supplementary dataset includes: obtaining one or more historical search items based on the historical search information corresponding to the dataset to be searched; the historical search items are data items in the dataset to be searched; for each historical search item, searching for associated data items in the dataset to be searched that are related to the historical search item; and obtaining the supplementary dataset based on the historical search items and the associated data items. Obtaining two or more subsets of data includes: obtaining two or more historical subsets of data; determining a first target subset containing the historical search item and one or more second target subsets containing the associated data items corresponding to the historical search item in the two or more historical subsets of data; dividing the associated data items from the second target subsets to the first target subsets and configuring the index order; Based on the user's query request, searches are performed in the additional dataset and the two or more sub-datasets respectively to obtain a first search result corresponding to the additional dataset and a second search result corresponding to each sub-dataset; wherein the first search result is obtained before the second search result.

2. The method according to claim 1, characterized in that, The step of searching for associated data items related to the historical search items in the dataset to be searched includes: Based on the attribute information of each data item in the dataset to be retrieved and the attribute information of the historical search items, the similarity between the historical search items and each data item in the dataset to be retrieved is obtained. Data items with a similarity greater than a preset similarity threshold are designated as associated data items.

3. The method according to claim 2, characterized in that, The attribute information of the data item includes the query time interval with the historical search item; wherein the query time interval is negatively correlated with similarity.

4. The method according to claim 2, characterized in that, The step of obtaining the similarity between the historical search items and each data item in the dataset to be retrieved, based on the attribute information of each data item in the dataset to be retrieved and the attribute information of the historical search items, includes: The attribute information of each data item in the dataset to be retrieved is input into a preset neural network model to obtain the first encoded data corresponding to each data item; The attribute information of the historical search items is input into the neural network model to obtain the second encoded data; Based on the first encoded data and the second encoded data, the similarity between the historical retrieval item and each data item in the dataset to be retrieved is determined.

5. The method according to claim 2, characterized in that, The step of obtaining the additional dataset based on the historical search items and the associated data items further includes: The index order of the associated data items is configured based on the similarity score; wherein, for the same query request, the index order of the associated data items is after the historical search items; The additional dataset is obtained based on the historical search items and the associated data items that have completed the index order configuration.

6. A distributed retrieval device, characterized in that, include: A subset acquisition module is used to obtain an additional dataset and two or more subsets; wherein the amount of data in the additional dataset is smaller than the dataset to be retrieved, and both the additional dataset and the two or more subsets are obtained based on the dataset to be retrieved; The process of obtaining the supplementary dataset includes: obtaining one or more historical search items based on the historical search information corresponding to the dataset to be searched; the historical search items are data items in the dataset to be searched; for each historical search item, searching for associated data items in the dataset to be searched that are related to the historical search item; and obtaining the supplementary dataset based on the historical search items and the associated data items. Obtaining two or more subsets of data includes: obtaining two or more historical subsets of data; determining a first target subset containing the historical search item and one or more second target subsets containing the associated data items corresponding to the historical search item in the two or more historical subsets of data; dividing the associated data items from the second target subsets to the first target subsets and configuring the index order; The retrieval module is used to perform searches in the supplementary dataset and the two or more sub-datasets based on the query request input by the user, and obtain a first retrieval result corresponding to the supplementary dataset and a second retrieval result corresponding to each of the sub-datasets; wherein the first retrieval result is obtained before the second retrieval result.

7. A readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

Images