Method for searching data, related device and computer program product

Abstract

The invention provides a data searching method, a related device and a computer program product, and the method comprises the steps: respectively processing a first hash value corresponding to a search word and a second hash value of to-be-searched data through employing a target hash modulus rule, and obtaining a search word fragment and a to-be-searched data fragment corresponding to the search word fragment; executing data search operation in the to-be-searched data fragments corresponding to the search word fragments by using the search word fragments in parallel to obtain search results corresponding to the to-be-searched data fragments; and summarizing the search results to generate a total search result. Therefore, the search efficiency of the to-be-searched data can be improved by splitting the search term and the to-be-searched data into the fragments and processing the search term and the to-be-searched data fragments in parallel, so that good search efficiency and search performance can also be achieved under the scene of large-scale to-be-searched data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, computer-readable medium, and computer program product for searching data. Background Technology

[0002] In the context of modern internet technology, content delivery has become an important component of many online platforms and applications. Content delivery is no longer simply about advertising or information dissemination; it is increasingly becoming personalized and intelligent to meet the needs and interests of different users. For example, based on users' interests and preferences, content can be provided that they might find interesting, thereby reducing the cost of searching and accessing content for users.

[0003] Against this backdrop, with the massive increase in data content and the gradual refinement of user and interest tags, traditional data matching and search technologies are struggling to meet the search demands under these circumstances. Therefore, how to conduct efficient and high-quality data searches in this "big data" era is a matter of concern and an urgent need. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for searching data, which can improve the search efficiency for the data to be searched by splitting the search terms and the data to be searched into fragments and processing the fragments of the search terms and the data to be searched in parallel, so as to have good search efficiency and search performance even in scenarios with large-scale data to be searched.

[0005] One aspect of this application provides a method for searching data, comprising: processing the first hash value corresponding to the search term and the second hash value of the data to be searched respectively using a target hash modulo rule to obtain search term shards and data shards to be searched corresponding to the search term shards; performing data search operations in parallel using the search term shards in the data shards to be searched corresponding to the search term shards to obtain search results corresponding to each data shard to be searched; and summarizing the search results to generate a total search result.

[0006] Another aspect of this application provides an apparatus for searching data, comprising: a sharding generation module configured to process a first hash value corresponding to a search term and a second hash value of the data to be searched using a target hash modulo rule, to obtain search term shards and data shards to be searched corresponding to the search term shards; a search execution module configured to perform data search operations in parallel using the search term shards in the data shards to be searched corresponding to the search term shards, to obtain search results corresponding to each data shard to be searched; and a result aggregation module configured to aggregate the search results to generate a total search result.

[0007] In another aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of searching data as provided above.

[0008] Another aspect of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the method for searching data as provided above.

[0009] Another aspect of this application is a computer program product that includes a computer program having computer program instructions stored thereon, which, when executed by a processor, enables the implementation of the method for searching data as provided above.

[0010] The solution provided in this application firstly uses a target hash modulo rule to process the first hash value corresponding to the search term and the second hash value of the data to be searched, respectively, to obtain search term shards and corresponding data shards to be searched. Then, using the search term shards in parallel, data search operations are performed in the corresponding data shards to obtain search results for each data shard. Finally, the search results are aggregated to generate a total search result. Therefore, by splitting the search term and the data to be searched into shards and processing them in parallel, the search efficiency for the data to be searched can be improved, resulting in good search efficiency and performance even in scenarios with large-scale data to be searched. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0013] Figure 1 A flowchart illustrating a data search process provided in an embodiment of this application;

[0014] Figure 2 A flowchart illustrating a process for determining search terms, provided as an embodiment of this application;

[0015] Figure 3 A flowchart illustrating the process of searching data in a specific application scenario, as provided in an embodiment of this application.

[0016] Figure 4 This is a schematic diagram of the structure of a device for searching data provided in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application.

[0018] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0021] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0022] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0023] As discussed above, how to conduct data searches efficiently and with high quality in this "big data" era is a matter of concern and urgent need.

[0024] In some traditional solutions, data search is often performed by repeatedly executing single-pattern search and search term matching. This limits the processing and search capabilities of these solutions for "big data" and "large-scale" data, making it difficult to efficiently complete searches in "big data".

[0025] To address this situation, this application provides a technical solution. First, using a target hash modulo rule, the first hash value corresponding to the search term and the second hash value of the data to be searched are processed separately to obtain search term shards and corresponding data shards to be searched. Then, the search term shards are used in parallel to perform data search operations within the corresponding data shards to obtain search results for each data shard. Finally, the search results are aggregated to generate a total search result. Therefore, by splitting the search term and the data to be searched into shards and processing them in parallel, the search efficiency for the data to be searched can be improved, resulting in good search efficiency and performance even in scenarios with large-scale data to be searched.

[0026] In practical scenarios, the execution entity of this method can be a user device, a device formed by integrating a user device and a network device through a network, or an application running on the aforementioned devices. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. Network devices include, but are not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0027] When the executing entity is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules, or as a single software program or software module, without specific limitations.

[0028] Figure 1 The present application illustrates a data search process 100, which includes at least the following processing steps:

[0029] (Step) S101, using the target hash modulo rule, process the first hash value corresponding to the search term and the second hash value of the data to be searched respectively to obtain the search term fragment and the data fragment to be searched corresponding to the search term fragment;

[0030] In the embodiments of this application, a server, cloud server, or other server capable of providing data search services to users can be used as the execution entity for the data search process.

[0031] After obtaining the search terms and the data to be searched (this process can be understood as using the search terms to search the data to extract the data and content containing the search terms), the executing entity can process the search terms and the data to be searched separately using preset hash processing rules, such as a hash function, to obtain the first search term corresponding to the search term and the second hash value corresponding to the data to be searched.

[0032] Search terms can be "keywords" used to achieve the search purpose. The data to be searched can include pre-maintained static data (e.g., Spark offline data, i.e., data under the Apache Spark framework) or online real-time data (e.g., Flink real-time data).

[0033] In some embodiments, after obtaining the search terms, the executing entity can not only improve the data quality by deduplication and removing useless symbols, but also preprocess the search terms by regularization, standardization and other methods so that they can be processed in a form that the executing entity can understand to complete subsequent steps.

[0034] For example, the executing entity can process and convert search terms and search data of arbitrary length, expressed in the form of strings, file content, numbers, etc., into a fixed-length hash value.

[0035] For example, given the search terms a and b (e.g., specific content that a user might be interested in) and the data A and B to be searched (e.g., online information published on the Internet that can be accessed by a user), the executing entity can process the search terms a and b using a hash function to obtain their respective first hash values, and process the data A and B to be searched using a hash function to obtain their respective second hash values.

[0036] It should be understood that a "search term" can be represented as a single character or multiple characters, and this application is not intended to limit its number of characters.

[0037] In the embodiments of this application, since hash values ​​will be used to complete the sharding of search terms and data to be searched, and it is expected that the two different types of shards (i.e., search term shards and data to be searched allocation) can be aligned, that is, the corresponding and aligned search term and data to be searched shards are consistent results obtained based on the same hash allocation and hash value strategy, in the embodiments of this application, the same hash processing rules, such as the same hash function, are used in the two processes of generating the first hash value and the second hash value.

[0038] Accordingly, in order to achieve this objective, in some embodiments, after obtaining a search term, the executing entity can first detect whether it is a newly obtained search term for the first time. If so, the executing entity can use the same target hash rule and target hash function (e.g., a target hash rule selected from a pre-configured hash rule library and hash function library based on data type and scenario) to process the search term based on the target hash rule, generate a first hash value, and process the data to be searched based on the target hash rule to generate a second hash value.

[0039] This allows both to achieve "fragment alignment" during subsequent fragmentation processing.

[0040] Similarly, in obtaining the "first hash value" and the "second hash value," the rules used by the executing entity should also be the same. For example, hash values ​​should be obtained for "prefixes" that are in the same position and have the same meaning. This ensures that when they are partitioned using the same modulo rule, they can be partitioned using "alignment" logic. For example, for both search terms and the information to be searched, hash operations are performed on the "prefixes" that represent the content, semantics, source, and context of the data to obtain their respective hash values ​​(i.e., the first hash value and the second hash value).

[0041] In some embodiments, if the search term is not being obtained for the first time, for example, if the search term is a historical search term repeatedly submitted and uploaded by the user, then in such cases, because it has already been used, the executing entity can directly read the historical hash value corresponding to the search term as the first hash value, in order to save computing resources and avoid processing it repeatedly.

[0042] Accordingly, in such a scenario, the executing entity can retrieve the historical hashing rule used to generate the historical hash value as the aforementioned target hashing rule, and use this historical hashing rule to process the data to be searched to generate a second hash value for the data to be searched. This allows the executing entity to reduce the computational resources used to determine the hash value while maintaining "shard alignment" in scenarios where different, new data to be searched using the same search terms.

[0043] It should be understood that before performing the above-described process of using the historical hash rule to process the data to be searched in order to generate a second hash value for the data to be searched, the executing entity can similarly determine whether the "data to be searched" corresponds to the historical search term used by the user. If so, the executing entity can similarly directly call the second hash value that was previously generated, thereby avoiding repeated processing of the data to be searched.

[0044] It should be understood that, because in scenarios where the same search terms and search data are used, the executing entity can often achieve the purpose of providing search results to the user by calling and reusing previous search results. Therefore, for the sake of ease of understanding, we will directly discuss the implementation examples of different scenarios of "search data". However, for situations where previous search results cannot be reused, or where the user expects to re-execute such a search process and provides the same search terms and search data, the executing entity can also achieve the search purpose through such an implementation method, which will not be repeated here.

[0045] After obtaining the first hash value and the second hash value, the executing entity can use the first hash value to perform sharding on the search terms to obtain (one or more) search term shards, and use the second hash value to perform sharding on the data to be searched to obtain (one or more) data shards.

[0046] In this step, as discussed above, the executing entity can use the same target hash modulo rule to shard the search terms and the data to be searched.

[0047] For example, the executing entity can first determine the number of shards, such as N, where N is a positive integer. That is, the number of search term shards and the number of data shards to be searched are both "N".

[0048] Then, the executing entity can use the modulo operation, or the remainder with respect to N, based on the "N", to use the first hash value and the second hash value to allocate the search terms and the data to be searched to one of the N search term shards and one of the N data to be searched shards, so as to form N search term shards and data to be searched shards.

[0049] Accordingly, in this process, as discussed above, because the same target hash processing rules and target hash modulo rules are used, the search term shards with corresponding order can be matched with the data shards to be searched in the sharding results, so that they can be processed in pairs subsequently. For example, the first search term shard can correspond to the first data shard to be searched, and the Nth search term shard can correspond to the Nth data shard to be searched.

[0050] Therefore, by using such shard pairs (i.e., a "pair" consisting of a search term shard and its corresponding data shard to be searched), not only can a large amount of data be searched simultaneously by parallelizing the various shard pairs, but network transmission between shards (pairs) can also be avoided, thereby improving search efficiency while reducing resource usage requirements.

[0051] In some embodiments, when the search term is multi-character, in order to save computing resources and enable search terms with the same prefix to be assigned to the same shard, the execution entity may also choose to select only the first character of the search term to generate the first hash value (e.g., using the ASCII code of the first character) and perform the above-mentioned modulo operation.

[0052] In some embodiments, if the total number of search terms is small, the executing entity may choose to shard only the data to be searched and use the same "search terms" for searching on each task node described below (i.e., when parallelizing on each task node, each task node uses the same search term shard, which is a "search term shard" composed of all the complete search terms). This allows for a more comprehensive and accurate search task when there are fewer search terms, less search resource usage, and search efficiency can be guaranteed. Thus, both search efficiency and accuracy can be balanced.

[0053] In such cases, in some embodiments, before this step is executed, the executing entity can first determine the number of search terms, and if the number of search terms is greater than or equal to a predetermined threshold, it can respond and simultaneously shard the search terms and the data to be searched. If the number of search terms is less than the threshold, the executing entity can directly use the same "search term sharding" on each task node to execute the "parallel search" discussed below.

[0054] S102, in parallel, the search term sharding is used to perform data search operations in the data shards to be searched corresponding to the search term shards, and the search results corresponding to each data shard to be searched are obtained.

[0055] In the embodiments of this application, after determining and obtaining the search term fragments and the corresponding data fragments to be searched based on the above S101, the executing entity can use these fragment pairs to perform searches in parallel to obtain the search results corresponding to each data fragment to be searched. That is, data search operations are performed in the data fragments to be searched corresponding to the search term fragments to obtain the search results corresponding to each data fragment to be searched.

[0056] In some embodiments, the executing entity may construct multiple task nodes and task processes for performing search tasks, and then use these task nodes to process a shard pair accordingly, so as to execute the search tasks for each shard pair in parallel through parallel task nodes.

[0057] In some embodiments, when determining the number of shard pairs, i.e., the aforementioned "N", the executing entity may also refer to the number of currently available task nodes. For example, the executing entity can use the number of currently available task nodes as "N". This allows the executing entity to fully utilize task nodes to perform parallel search tasks on shard pairs, thereby improving resource utilization while avoiding situations where an unreasonable number of shard pairs (e.g., too many) leads to some search tasks not being able to be parallelized, thus slowing down the overall search efficiency.

[0058] In some embodiments, during the parallel execution of data search operations to obtain search results corresponding to each data fragment to be searched, the executing entity of the search operation (e.g., search operations or actions executed in task nodes) can choose to utilize a trie to improve the efficiency and quality of the specific search operation. For example, a double-array trie can be used for searching.

[0059] For example, the executing entity can use AC-DAT (Aho-Corasick Double Array Trie), an improved version of the multi-pattern string matching algorithm, under a double array trie to complete the search process in the slice pair.

[0060] AC-DAT is a hybrid structure that combines Aho-Corasick automata and Double Array Trie (DAT). It adds a failure pointer (pointing to the node corresponding to the longest common suffix) and an output pointer (pointing to the list of matched words) to each node on top of the DAT, enabling parallel matching of multiple pattern strings during single-text scanning. Therefore, AC-DAT allows for faster and more efficient search operations while maintaining search quality during the search of each piece of data.

[0061] Accordingly, in such a scenario, the executing entity can, during the parallel execution of various search operations, first utilize the target search terms included in each search term shard to generate prefix trees and failed links (or, failure pointers) in parallel. For example, when utilizing multiple task nodes, the executing entity can (in parallel) instruct each task node to perform the actions of generating prefix trees and failed links based on the search terms in the assigned search term shards.

[0062] In the process of constructing a prefix tree, after obtaining the search terms included in the search term fragments, the task node can use a tree structure (Trie) to assemble the strings included in them to form a character sequence, i.e., the "prefix tree". For example, for the search terms "he", "she", "his", and "hers", the tree structure can be a tree starting from the root node, with the first level being h, the second level being e and i, the third level being s and r after e, i after i, s after i, and so on.

[0063] Then, the executing entity can use this prefix tree to traverse the corresponding data to be searched in order to complete the retrieval operation of the data to be searched based on the search terms.

[0064] Accordingly, while building and generating the prefix tree, task nodes also need to be configured with failure links, and a failure link needs to be added to each node. Failure links are used to point to another node that can continue matching when a node in the tree cannot continue matching, avoiding backtracking and improving matching efficiency. For example, if the current character fails to match, the failure link can be used to backtrack to the parent node and continue trying to match the failed link. Typically, each node in the tree has a failure link pointing to a sibling node or parent node to avoid repeated comparisons.

[0065] For example, in the example above, for "he", if the match fails at the end of "he", the jump to "his" or "hers" via the failure link is used to avoid rematching from the root.

[0066] Accordingly, in this step, the execution entity can, as an alternative, use the prefix tree and failed links corresponding to the search term shards in parallel to perform data search operations in the data shards to be searched corresponding to the search term shards, thereby obtaining the search results corresponding to each data shard to be searched, so as to improve search efficiency.

[0067] In some embodiments, if prefix trees and failed links are used, after determining the search term fragments, the executing entity can first assign search terms to task nodes to establish a placeholder and build a search framework. This allows the entity to directly utilize and reuse the search framework after acquiring and fragmenting the data to be searched. For example, after receiving a search term, the executing entity can determine whether it is a previously received historical search term. If so, the executing entity can reuse the historical search term for new data by invoking the "search framework".

[0068] Accordingly, in such a scenario, the executing entity can first allocate search term shards to target task nodes. Furthermore, when generating prefix trees and failed links in parallel using the target search terms included in each search term shard, it can further select which target task node will use the target search terms included in the allocated target search term shards to generate the prefix trees and failed links used in that target task node.

[0069] Accordingly, in some embodiments, given such a "search framework," if the subsequent execution entity obtains the data to be searched, it performs data search operations in the data shards corresponding to the search terms in parallel using the prefix tree and failed links corresponding to the search term shards to obtain the search results corresponding to each data shard. In the process, the execution entity further chooses to allocate the target data shards corresponding to the target search terms to the target node based on the correspondence between the data shards to be searched and the search term shards after sharding the data to be searched using the target hash modulo rule and obtaining the data shards to be searched, according to the correspondence between the data shards to be searched and the search term shards.

[0070] Then, the execution entity uses the prefix tree and failed links corresponding to the target search term shards in parallel at each target task node to perform data search operations in the target data shards to be searched, and obtains the search results corresponding to each target data shard to be searched.

[0071] Therefore, by allowing asynchronous sharding and task node division, users can flexibly and efficiently complete the configuration of search operations by reusing the framework when there are different needs for search data and search data sources, thereby reducing their operating costs.

[0072] S103, aggregate all search results and generate the total search results.

[0073] In the embodiments of this application, after obtaining and searching the search results corresponding to each data segment to be searched based on the above S102 (that is, each data segment to be searched includes the results of the search terms in the corresponding search term segment), the executing entity can summarize the various search results by concatenation to generate the total search results.

[0074] In some embodiments, during the generation of total search results, since each search result comes from different segments, the executing entity can also adjust the duplicate information included in the results through operations such as deduplication and data processing to improve the value of the data.

[0075] After obtaining the total search results, the executing entity can choose to store the total search results and then generate an interface for accessing and retrieving the total search structure, so that users can obtain the total search results by calling the interface.

[0076] The method for searching data provided in this application firstly processes the first hash value corresponding to the search term and the second hash value of the data to be searched using the target hash modulo rule, obtaining search term shards and corresponding data shards to be searched. Then, it performs data search operations in parallel within the corresponding data shards to obtain search results for each data shard. Finally, it aggregates all search results to generate a total search result. Thus, by splitting search terms and data to be searched into shards and processing them in parallel, the search efficiency for the data to be searched is improved, enabling good search efficiency and performance even in scenarios with large-scale data to be searched.

[0077] In some embodiments, to facilitate the user's use of such search data, the executing entity can also provide and encapsulate user-defined functions (UDFs) so that users can directly call the UDFs to more conveniently access search services that search for data based on the search terms provided by the user.

[0078] In some embodiments, the UDF can allow users to invoke it using Structured Query Language (SQL), enabling users to initiate a search process directly through SQL statements with low comprehension costs, thereby improving the user experience.

[0079] Accordingly, to better understand the process by which users provide search terms via SQL, we can also refer to... Figure 2 . Figure 2 This application illustrates a process 200 for determining search terms according to an embodiment of the present application.

[0080] Process 200 includes at least the following processing steps:

[0081] S201, in response to receiving the search term file from the target device, the search term file is stored and a first storage path for the search term file is generated;

[0082] Specifically, if a user uploads a search term file (e.g., a file in orc, txt, or parquet format that records the search terms the user intends to use) to the executing entity using the target device (e.g., a terminal device), the executing entity can respond by storing the search term file and generating a first storage path for the search term file.

[0083] Typically, the executing entity can store the search term file in its local memory or persistent storage for later use.

[0084] In some embodiments, the implementing entity may also choose to integrate it with a "big data" scenario, utilizing a distributed storage system, such as HDFS (Hadoop Distributed File Systems), to store the search term file. This leverages the storage capabilities of HDFS to store the search terms (files) and the data to be searched, providing storage capacity for "big data" search scenarios.

[0085] Accordingly, after storing the search term file, the executing entity can generate a first storage path for the search term file, so that the user can subsequently call the search term file through the first storage path, instructing the executing entity to use the search terms in the search term file to perform a search.

[0086] S202, Provide the first storage path to the target device;

[0087] Specifically, after completing the storage and generating the first storage path, the executing entity can return the first storage path to the target device for subsequent use by the user.

[0088] S203, in response to receiving a first search instruction from the target device to read search terms through a first storage path by calling a user-defined function, the search term file is obtained by accessing the first storage path;

[0089] Specifically, as discussed above, the executing entity can allow users to interact with it in the form of SQL to obtain search services.

[0090] In such a scenario, a user can send a first search instruction in SQL form by invoking a UDF, instructing the execution entity to retrieve the search term file using the first storage path specified in the first search instruction, and to perform a search process based on the search terms specified in the search term file. For example, a user can write their desired storage path (e.g., the first storage path) in the first search instruction in SQL form to indicate their desired search terms (files).

[0091] Accordingly, if the executing entity receives a first search instruction from the user using a search term file by calling a user-defined function, it can respond by accessing the search term file through the first storage path.

[0092] Then, by continuing to execute the next step S204, the search term file is obtained, and the search terms in the search term file are extracted by reading the search term file, which will be used as the search terms described above.

[0093] S204, Use the search term file to determine the search terms.

[0094] In some embodiments, to further reduce the user's operating costs, the executing entity may also choose to pre-maintain candidate search term files that can be used by the user, so that the user can efficiently and cost-effectively provide and indicate search terms to the executing entity and start the search process by specifying the candidate search term files.

[0095] For ease of understanding, we will use them together. Figure 2 The process 200 shown will be discussed in this paper. For example, process 200 may also include at least S205.

[0096] S205, in response to receiving a selection instruction from the target device for a candidate search term file, determine the search term file corresponding to the selection instruction;

[0097] Specifically, as discussed above, the executing entity can pre-maintain one or more candidate search term files for user use, for example, by using a separate storage to maintain these candidate search term files. Correspondingly, the executing entity can also allow users to specify the desired search term file to the executing entity using SQL-style selection instructions, for example, by invoking a UDF to provide such SQL-style selection instructions.

[0098] Accordingly, if the executing entity receives such a selection instruction, it can respond by determining the search term file corresponding to the selection instruction.

[0099] In some embodiments, the selection instruction can further indicate whether the user wishes to use all of the candidate search terms, in addition to specifying the candidate search term file to be used. For example, the user can also use the selection instruction to specify that they only wish to use a portion of the indicated search term file. This not only allows users to flexibly specify and select the specific "search terms" they wish to use according to their specific needs, satisfying their diverse and personalized search requirements, but also enhances the usability of the search term file.

[0100] S206, Store the search term file and generate a second storage path for the search term file;

[0101] Specifically, after determining the search term file corresponding to the selection instruction (or a portion of the search term file that is specifically expected to be used), the executing entity can similarly store the search term file (e.g., to the HDFS that will actually be used, as discussed above) and generate a second storage path for the search term file.

[0102] S207, Provide a second storage path to the target device;

[0103] Specifically, the executing entity can similarly provide a second storage path to the target device for the user to use, and actually use the search term file by calling the second storage path.

[0104] S208, in response to receiving a second search instruction from the target device via a call to a user-defined function, using a search term file, the search term file is obtained by accessing the second storage path;

[0105] Specifically, similar to the discussion above, if the executing entity receives a second search instruction from the user, which is in the form of SQL passed through a UDF, to retrieve the search term file through the second storage path and to execute the search using the search term file, the executing entity can respond by retrieving the search term file through the second storage path.

[0106] Then, the executing entity can similarly continue to select to execute S204 above to determine the search terms as described above, which will not be repeated here.

[0107] To deepen understanding, this application also incorporates a specific application scenario, through... Figure 3 This diagram illustrates a specific implementation of the data search process in this particular application scenario. For further details, please refer to... Figure 3 The process shown is 300.

[0108] In process 300, server 310 can be used as an example of the "executing entity" (e.g., server 310 can be a server or cloud server that provides search services to the terminal device 313 used by the user).

[0109] In process 300, firstly, a user (not shown in the figure) can execute S301 using terminal device 313 to provide the server 310 with the search terms they wish to use. For example, the user provides a set 320 of search terms 321, 322, ..., 32N, where N is a positive integer.

[0110] Accordingly, after receiving the set of search terms 320, the server 310 can generate the hash value of each search term by executing S302, for example, the hash value 331 of search term 321, the hash value 332 of search term 322, ... the hash value 33N of search term 32N.

[0111] Next, server 310 can generate search term shards based on the target hash modulo rule by executing S303, for example, generating search term shard 341 and search term shard 342.

[0112] It should be understood that the number of search term segments shown here is merely illustrative and not intended to limit the number of generated segments.

[0113] For example, search term segment 341 may include search term 321 and search term 322, while search term segment 342 may include the remaining search terms in search term set 320 other than search term 321 and search term 322.

[0114] For ease of understanding, the process of searching data can be implemented in process 300 by building the "search framework" as described above.

[0115] Accordingly, in such a case, server 310 can first execute S304 to allocate search term shard 341 and search term shard 342 to task node 351 and task node 352 respectively (similarly, the number of task nodes is only an exemplary choice corresponding to the number of shards and is not intended to limit the number of task nodes).

[0116] After being assigned a corresponding search term shard, the task node can, as discussed above, use the actual search terms included in the search term shard to generate a prefix tree and failed links within it.

[0117] Taking task node 351 as an example, task node 351 can generate a prefix tree 361 and failed links 362 by executing S305 (for example, by being controlled by server 310 to execute S305).

[0118] Next, server 310 can process the search data as described in S302 via S306 to determine the hash value of each piece of data. For example, it can process the search data using the "target hash rule" used when processing search terms.

[0119] Here, because the process is similar to S302 above (the only difference is between the objects being processed), for ease of understanding and for the purpose of referring to the accompanying drawings, the process will not be discussed in detail here.

[0120] Similarly, after generating the hash value of the data to be searched based on S306, server 310 can also generate data fragment 371 and data fragment 372 to be searched using the hash value of the data to be searched, just like in S303, by using the target hash modulo rule. Again, because this process is similar to S303 above (only the objects being processed are different), for the purpose of easy understanding and the content of the attached figures, this process will not be discussed in detail here.

[0121] After generating the data shards 371 and 372 to be searched, the server 310 can execute S308 to allocate the data shards 371 and 372 to task nodes 351 and 352 respectively, based on the correspondence between the data shards 371 and 372 to be searched and the search term shards 341 and 342 (e.g., search term shard 341 corresponds to data shard 371, and search term shard 342 corresponds to data shard 372). For example, data shard 371 is allocated to task node 351 for processing, and data shard 372 is allocated to task node 352 for processing.

[0122] Accordingly, after completing the allocation, server 310 can execute S309 to perform parallel data search operations on task nodes 351 and 352 to obtain search results 381 and 382. That is, in task node 351, the search terms in search shard 341 (i.e., search terms 321 and 322) are used to perform a search on the data to be searched in search data shard 371 to obtain search result 381, and in task node 352, the search terms in search shard 342 (i.e., the remaining search terms other than search terms 321 and 322) are used to perform a search on the data to be searched in search data shard 372 to obtain search result 382.

[0123] After completing each task node, server 310 can execute S310 to aggregate search results 381 and 382 to obtain the final total search results 390.

[0124] This application also provides an apparatus for searching data, the structure of which is as follows: Figure 4The apparatus 400 shown includes: a shard generation module 410, configured to process the first hash value corresponding to the search term and the second hash value of the data to be searched using a target hash modulo rule, to obtain search term shards and data shards to be searched corresponding to the search term shards; a search execution module 420, configured to perform data search operations in parallel using the search term shards in the data shards to be searched corresponding to the search term shards, to obtain search results corresponding to each data shard to be searched; and a result aggregation module 430, configured to aggregate the search results to generate a total search result.

[0125] In some embodiments, the search execution module 420 includes: a parallel generation submodule configured to generate a prefix tree and failed links in parallel using the target search terms included in each search term shard; and a parallel execution submodule configured to perform data search operations in the search data shards corresponding to the search term shards in parallel using the prefix trees and failed links corresponding to the search term shards, and obtain the search results corresponding to each search data shard.

[0126] In some embodiments, the apparatus 400 further includes: a search term shard allocation module configured to allocate search term shards to target task nodes; and a parallel generation submodule further configured to generate a prefix tree and failed links at each target task node using the target search terms included in the allocated target search term shards.

[0127] In some embodiments, the parallel execution submodule includes: a search data shard allocation unit, configured to allocate target data shards corresponding to target search term shards to target nodes based on the correspondence between data shards to be searched and search term shards; and a parallel execution unit, configured to perform data search operations in parallel on each target task node using the prefix tree and failed links corresponding to the target search term shards, to obtain search results corresponding to each target data shard to be searched.

[0128] In some embodiments, the apparatus 400 further includes: a search term acquisition module configured to acquire search terms; a first hash value generation module configured to process the search term based on a target hash rule and generate a first hash value in response to the search term being a newly acquired search term; and a second hash value generation module configured to process the data to be searched based on the target hash rule and generate a second hash value.

[0129] In some embodiments, the apparatus 400 further includes: a historical hash value reading module, configured to read the historical hash value corresponding to the search term as a first hash value in response to the search term being a historical search term; and a third hash value generation module, configured to process the data to be searched and generate a second hash value by using the historical hash rule used when generating the historical hash value as the target hash rule.

[0130] In some embodiments, the apparatus 400 further includes: a first search term file storage module, configured to store the search term file and generate a first storage path for the search term file in response to receiving a search term file from a target device; a first path providing module, configured to provide the first storage path to the target device; a first search term file acquisition module, configured to acquire the search term file by accessing the first storage path in response to receiving a first search instruction from the target device to read search terms through the first storage path by calling a user-defined function, wherein the first search instruction is in the form of a structured query language; and a first search term determination module, configured to determine search terms using the search term file.

[0131] In some embodiments, the apparatus 400 further includes: a search term file determination module, configured to determine a search term file corresponding to a selection instruction for a candidate search term file in response to receiving such an instruction, wherein the selection instruction is in the form of a Structured Query Language; a second search term file storage module, configured to store the search term file and generate a second storage path for the search term file; a second path providing module, configured to provide the second storage path to a target device; a second search term file acquisition module, configured to acquire the search term file by accessing the second storage path in response to receiving a second search instruction using the search term file passed by the target device through a call to a user-defined function; and a second search term determination module, configured to determine a search term using the search term file.

[0132] This device embodiment corresponds to the method embodiment shown in the figures above. This device can improve the search efficiency for the search data by splitting the search terms and the data to be searched into fragments and processing the fragments of search terms and the data to be searched in parallel. This enables it to have good search efficiency and search performance even in scenarios with large-scale data to be searched.

[0133] Based on the same inventive concept, this application also provides an electronic device, a readable storage medium, and a computer program product. The method corresponding to the electronic device can be the data search method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0134] Electronic devices can be user devices, or devices composed of user devices and network devices integrated through a network, or applications running on the aforementioned devices. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. Network devices include, but are not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0135] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage portion 508 into a Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0136] The following components are connected to I / O interface 505: an input section 504 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 508 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0137] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0138] Another embodiment of this application provides a computer-readable storage medium and a computer program product having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0139] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, a system, apparatus, or device that is, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0141] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0142] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules and units is only a logical functional division, and in actual implementation, there may be other division methods. Taking units as examples, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional modules and units in the various embodiments of this application can be integrated into one processing module or unit, or each module or unit can exist physically separately, or two or more units can be integrated into one module or unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules and units.

[0148] The integrated modules and units implemented as software functional modules and units described above can be stored in a computer-readable storage medium. These software functional modules and units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0150] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for searching data, characterized in that, include: Using the target hash modulo rule, the first hash value corresponding to the search term and the second hash value of the data to be searched are processed respectively to obtain the search term fragment and the data fragment to be searched corresponding to the search term fragment; The search term segments are used in parallel to perform data search operations in the data segments to be searched corresponding to the search term segments, so as to obtain the search results corresponding to each data segment to be searched. Combine all the search results to generate the overall search results.

2. The method according to claim 1, characterized in that, The parallel utilization of the search term shards, performing data search operations in the data shards to be searched corresponding to the search term shards, and obtaining search results corresponding to each data shard to be searched includes: In parallel, the target search terms included in each search term segment are used to generate a prefix tree and failed links; The prefix tree and the failed links corresponding to the search term segments are used in parallel to perform data search operations in the data segments to be searched corresponding to the search term segments, so as to obtain the search results corresponding to each data segment to be searched.

3. The method according to claim 2, characterized in that, The method further includes: The search term segments are allocated to target task nodes; and The parallel generation of prefix trees and failed links using the target search terms included in each search term segment includes: At each of the target task nodes, a prefix tree and failed links are generated using the target search terms included in the assigned target search term shards.

4. The method according to claim 3, characterized in that, The method involves using the prefix tree and failed links corresponding to the search term shards in parallel to perform data search operations in the data shards to be searched corresponding to the search term shards, obtaining search results for each data shard to be searched, including: Based on the correspondence between the data shards to be searched and the search term shards, target data shards to be searched corresponding to the target search term shards are allocated to the target nodes; In each of the target task nodes, the prefix tree and the failed links corresponding to the target search term segment are used in parallel to perform data search operations in the target data segment to be searched, so as to obtain the search results corresponding to each target data segment to be searched.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the search term; In response to the fact that the search term is a newly acquired search term, the search term is processed based on the target hash rule to generate the first hash value; The search data is processed based on the target hash rule to generate the second hash value.

6. The method according to claim 5, characterized in that, The method further includes: In response to the fact that the search term is a historical search term, the historical hash value corresponding to the search term is read as the first hash value; The historical hashing rule used when generating the historical hash value is used as the target hashing rule to process the data to be searched and generate the second hash value.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to receiving a search term file from the target device, the search term file is stored, and a first storage path for the search term file is generated; Provide the first storage path to the target device; In response to receiving a first search instruction from the target device to read the search term through the first storage path by calling a user-defined function, the search term file is obtained by accessing the first storage path, wherein the first search instruction is in the form of a structured query language. The search terms are determined using the search term file.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to receiving a selection instruction from a target device for a candidate search term file, the search term file corresponding to the selection instruction is determined, wherein the selection instruction is in the form of a structured query language; Store the search term file and generate a second storage path for the search term file; Provide the second storage path to the target device; In response to receiving a second search instruction from the target device via a call to a user-defined function, utilizing the search term file, the search term file is obtained by accessing the second storage path, wherein the second search instruction is in the form of the structured query language; The search terms are determined using the search term file.

9. An apparatus for searching data, characterized in that, include: The sharding generation module is configured to use the target hash modulo rule to process the first hash value corresponding to the search term and the second hash value of the data to be searched, respectively, to obtain the search term shards and the data to be searched shards corresponding to the search term shards; The search execution module is configured to utilize the search term segments in parallel to perform data search operations in the data segments to be searched corresponding to the search term segments, and obtain the search results corresponding to each data segment to be searched. The results aggregation module is configured to aggregate the individual search results and generate a total search result.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

11. A computer-readable medium, characterized in that, It stores computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images