Data retrieval method and device, terminal and computer readable storage medium

Abstract

The embodiment of the invention discloses a data retrieval method and device, a terminal and a computer readable storage medium, and the method comprises the steps: building a mapping relation between sample data features of each piece of sample data in a data set and sample data position information, and building a data index set corresponding to each piece of sample data in the data set based on each mapping relation; in response to a target data feature of the target data, obtaining a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature; and based on the target mapping relationship, determining target data position information corresponding to the target data feature, and searching target data based on the target data position information. In the retrieval process, retrieval can be conducted through the input target retrieval information of the target data and the data index set, it is avoided that all sample data in the data set is traversed, the retrieval speed is greatly increased, and when the sample data in the data set is more and more, the influence on the size of the data index set is small.

Description

Technical Field

[0001] The present application relates to the field of computer network service retrieval, and in particular to a data retrieval method, device, terminal and computer-readable storage medium. Background Art

[0002] When searching for target data from a data set, people can use retrieval methods to find the target data. However, when the data set contains more and more data, the retrieval speed of the target data will drop significantly, resulting in a significant increase in the retrieval time of the target data. Summary of the invention

[0003] The embodiments of the present application provide a data retrieval method, device, terminal and computer-readable storage medium, which can solve the technical problem that when more and more data is included in a data set, the retrieval speed of target data will drop significantly, thereby causing the retrieval time of target data to increase significantly.

[0004] In a first aspect, an embodiment of the present application provides a data retrieval method, the method comprising:

[0005] Establishing a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and establishing a data index set corresponding to each sample data in the data set based on each mapping relationship;

[0006] In response to target search information of the target data input by the search operation, determining a target data feature corresponding to the target search information, and obtaining a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature;

[0007] Based on the target mapping relationship, target data location information corresponding to the target data feature is determined, and the target data is searched based on the target data location information.

[0008] Optionally, establishing a mapping relationship between sample data features of each sample data in the data set and sample data location information includes:

[0009] A mapping relationship between the sample data feature and the data location information of each sample data in the data set is established based on at least one preset hash function.

[0010] Optionally, establishing a mapping relationship between sample data features and data location information of each sample data in the data set based on at least one preset hash function includes:

[0011] Inputting the sample data features of each sample data in the data set into at least one preset hash function respectively, to obtain at least one sample hash value corresponding to the sample data features of each sample data in the data set;

[0012] Merging at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set;

[0013] Determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash combination value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0014] Optionally, obtaining a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature includes:

[0015] Inputting the target data feature into the at least one preset hash function to obtain at least one target hash value of the target data feature;

[0016] The at least one target hash value is merged to obtain a target hash merge value of the target data feature, a sample hash merge value matching the target hash merge value is determined in the data index set, and a target mapping relationship is determined based on a sample mapping relationship corresponding to the sample hash merge value.

[0017] Optionally, the sample data feature of each sample data in the data set is a feature that each sample data in the data set includes and has a differentiation feature.

[0018] Optionally, after the data index set corresponding to each sample data in the data set is established based on each mapping relationship, the method further includes:

[0019] Storing the data index set;

[0020] The obtaining of the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature includes:

[0021] The stored data index set is acquired, and the target data index corresponding to the target data feature is obtained according to the data index set and the target data feature.

[0022] Optionally, searching for the target data based on the target data location information includes:

[0023] Obtaining a storage path of the target data based on target data location information of the target data;

[0024] Search the target data according to the storage path of the target data

[0025] In a second aspect, an embodiment of the present application provides a data retrieval device, the device comprising:

[0026] An establishing module, adapted to establish a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and to establish a data index set corresponding to each sample data in the data set based on each mapping relationship;

[0027] A response module, adapted to respond to target search information of target data input by a search operation, determine a target data feature corresponding to the target search information, and obtain a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature;

[0028] The search module is adapted to determine the target data location information corresponding to the target data feature based on the target mapping relationship, and search for the target data based on the target data location information.

[0029] Optionally, the establishing module includes:

[0030] The establishing submodule is adapted to establish a mapping relationship between the sample data feature and the data location information of each sample data in the data set based on at least one preset hash function.

[0031] Optionally, the establishing submodule includes:

[0032] A first submodule, adapted to input the sample data features of each sample data in the data set into at least one preset hash function respectively, to obtain at least one sample hash value corresponding to the sample data features of each sample data in the data set;

[0033] The second submodule is adapted to merge at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set;

[0034] The third submodule is adapted to determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash merge value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0035] Optionally, the response module includes:

[0036] a target hash value acquisition submodule, adapted to input the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature;

[0037] The target mapping relationship determination submodule is suitable for merging the at least one target hash value to obtain the target hash merge value of the target data feature, determining a sample hash merge value matching the target hash merge value in the data index set, and determining the target mapping relationship based on the sample mapping relationship corresponding to the sample hash merge value.

[0038] Optionally, the response module includes:

[0039] a target hash value acquisition submodule, adapted to input the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature;

[0040] The target mapping relationship determination submodule is suitable for merging the at least one target hash value to obtain the target hash merge value of the target data feature, determining a sample hash merge value that matches the target hash merge value in the data index set, and determining the target mapping relationship based on the sample mapping relationship corresponding to the sample hash merge value.

[0041] Optionally, the sample data feature of each sample data in the data set is a feature that each sample data in the data set includes and has a differentiation feature.

[0042] Optionally, the data retrieval device comprises:

[0043] A storage module, adapted to store the data index set;

[0044] The response module comprises:

[0045] The acquisition submodule is adapted to acquire the stored data index set, and obtain the target data index corresponding to the target data feature according to the data index set and the target data feature.

[0046] Optionally, the search module includes:

[0047] A storage path determination submodule, adapted to obtain a storage path of the target data based on the target data location information of the target data;

[0048] The target data search submodule is adapted to search for the target data according to the storage path of the target data.

[0049] In a third aspect, an embodiment of the present application further provides a terminal, the terminal comprising:

[0050] Processor; and

[0051] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.

[0052] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, implement any of the methods described above.

[0053] The beneficial effects brought about by the technical solutions provided in some embodiments of the present application include at least:

[0054] The embodiment of the present application provides a data retrieval method, which first establishes a mapping relationship between the sample data features of each sample data in the data set and the sample data location information, and then obtains a data index set of each sample data in the data set based on the above mapping relationship. When searching, the target search information and data index set of the input target data can be used for searching without traversing each sample data in the data set, so that the search speed is greatly increased, and when the sample data in the data set increases, the size of the data index set is not greatly affected. Therefore, even if the sample data in the data set increases, the search speed of the target data will not be greatly reduced, thereby causing the problem of a significant increase in the search time of the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 An exemplary system architecture diagram of a data retrieval method provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of a data retrieval method provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of another data retrieval method provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a process for searching a target mapping relationship provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of a process for determining a mapping relationship provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of a process for confirming a target mapping relationship provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of a data retrieval architecture provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of a data retrieval device provided in an embodiment of the present application;

[0064] Fig. 9 A structural schematic diagram of a data retrieval terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the embodiments of the present application.

[0066] With the rapid development of information technology, people can find target data from data sets through retrieval methods. However, as sample data in data sets accumulate, it becomes difficult for people to retrieve target data from TB (Terabyte) or even PB (Petabyte) level data. In related technologies, when more and more sample data are in a data set, it is necessary to traverse all sample data in sequence to find the target data during retrieval. Therefore, the retrieval speed of the target data will be greatly reduced, resulting in a significant increase in the retrieval time of the target data.

[0067] In order to overcome the above technical problems, the embodiments of the present application provide a data retrieval method, device, terminal and computer-readable storage medium. The data retrieval method first establishes a mapping relationship between the sample data features of each sample data in the data set and the sample data location information, and then obtains the data index set of each sample data in the data set based on the above mapping relationship. When searching, the target search information and data index set of the input target data can be used for searching without traversing each sample data in the data set, so that the search speed is greatly increased, and when the sample data in the data set increases, the size of the data index set is not greatly affected. Therefore, even if the sample data in the data set increases, the search speed of the target data will not be greatly reduced, thereby causing the problem of a significant increase in the search time of the target data.

[0068] See also Figure 1 , Figure 1 An exemplary system architecture diagram of a data retrieval method provided in an embodiment of the present application.

[0069] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0070] The terminal 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the terminal 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various terminals, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the terminal 101 is software, it can be installed in the terminals listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0071] In an embodiment of the present application, terminal 101 can establish a mapping relationship between sample data features of each sample data in a data set and sample data location information, and establish a data index set for each sample data in the corresponding data set based on each mapping relationship; in response to the target retrieval information of the target data input by the retrieval operation, determine the target data features corresponding to the target retrieval information, and obtain the target mapping relationship corresponding to the target data features based on the data index set and the target data features; based on the target mapping relationship, determine the target data location information corresponding to the target data features, and search for the target data based on the target data location information.

[0072] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0073] Alternatively, the system architecture may not include the server 103. In other words, the server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification may be applied to a system structure that only includes the terminal 101, and the embodiments of this application are not limited to this.

[0074] It should be understood that Figure 1 The number of terminals, networks and servers in the figure is only for illustration and any number of terminals, networks and servers may be used according to implementation requirements.

[0075] See also Figure 2 , Figure 2 A flowchart of a data retrieval method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes the data retrieval method, or a processor in the terminal that executes the data retrieval method, or a data retrieval service in the terminal that executes the data retrieval method. For the convenience of description, the specific execution process of the device data retrieval method is introduced below by taking the execution subject being the processor in the terminal as an example.

[0076] S202: Establishing a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and establishing a data index set corresponding to each sample data in the data set based on each mapping relationship.

[0077] The sample data features of each sample data in the data set are features that are included in each sample data in the data set and have differentiation, so that the target data can be accurately found during retrieval.

[0078] By establishing a data index set including a mapping relationship between sample data features and sample data location information, the sample data features are extracted from the sample data, and a mapping relationship between the sample data features and the sample data location information is established, so as to avoid the need to traverse the sample data to match the sample data features during subsequent retrieval, and then obtain the sample data location information. The embodiment of the present application can directly perform retrieval through the data index set to make the retrieval more convenient. It should be noted that the number of mapping relationships in the data index set can be one or more.

[0079] In a feasible implementation, the sample data includes log information and information to be consulted, and each log information may correspond to a sample data. Generally, the log information is stored in the sample data corresponding thereto, and the log information may include sample data features of the sample data. The sample data features may be a string of fields that each sample data includes and has differentiation, for example, may be a string of id (code, Identity document), and the id may include but is not limited to a code consisting of numbers or letters, which is not limited here.

[0080] Figure 3 A flowchart of another data retrieval method provided in an embodiment of the present application is shown below. Figure 3As shown, in an embodiment provided by the present application, after establishing a data index set for each sample data in the corresponding data set based on each mapping relationship, it also includes:

[0081] S302: Store the data index set.

[0082] Storing the data index set includes storing the data index set internally or storing the data index set externally, and the data index set can be stored in a memory in the terminal or in an external memory outside the terminal. The sample data can also be stored in a memory in the terminal or in an external memory outside the terminal.

[0083] Since the existing retrieval index structure often needs to be extended to the sample data, the storage space required by the existing retrieval index structure is larger and more complex. The data index set of the present application can be saved independently of the sample data, and the data index set mainly includes the mapping relationship between the sample data features of each sample data and the sample data location information. Therefore, the introduction of the data index set saves the space occupied by the retrieval index structure and makes the retrieval index structure simpler.

[0084] In a feasible implementation, the storage method of the memory in the terminal is called internal storage, and the storage method of the memory outside the terminal is called external storage. The storage method of the data index set can be external storage, and the storage method of the sample data can be internal storage. When searching, the terminal can obtain the data index set in the external storage to find the target data from the sample data in the internal storage. For another example, the storage method of the data index set can be internal storage, and the storage method of the sample data can be external storage. When searching, the terminal can read the internal data index set to find the target data from the external sample data.

[0085] Of course, in some other implementations, the storage methods of the data index set and the sample data can be set to internal storage; or, the storage methods of the data index set and the sample data can be set to external storage, which will not be described in detail here. In addition, the data index set can also be stored in a distributed manner.

[0086] S304: Obtaining a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature includes:

[0087] The stored data index set is obtained, and the target data index corresponding to the target data feature is obtained according to the data index set and the target data feature.

[0088] Among them, the stored data index set is read. The data index set can be stored in the memory in the terminal or in the external memory outside the terminal. When the data index set can be stored in the external memory outside the terminal, the storage space of the memory in the terminal can be effectively saved.

[0089] In a feasible implementation, the data index set can be stored in an external storage device outside the terminal, such as a USB flash drive, a CD, etc., which can be connected to the terminal before retrieval, and the terminal reads the data index set in the external storage device to perform a retrieval operation.

[0090] In another feasible implementation, the data index set may be stored in a memory in the terminal, and the data index set occupies less space than the existing search index structure.

[0091] S204: In response to the target search information of the target data input by the search operation, determine the target data feature corresponding to the target search information, and obtain the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature.

[0092] Among them, based on the target retrieval information of the target data, the target data features in the target retrieval information are extracted, the target data features and the sample data features corresponding to each mapping relationship in the data index set are matched, and after the matching is successful, the mapping relationship corresponding to the successfully matched sample data features is used as the target mapping relationship.

[0093] In a feasible implementation, the target retrieval information may be a retrieval statement for retrieving target data, and the target data features in the retrieval statement of the target data may have an identification field for identification, such as "id=", wherein the identification field may be filled with the target data features, such as the encoding mentioned above, where the lengths of different encodings may be the same or different.

[0094] In other embodiments, when the target data features and the sample data features corresponding to each mapping relationship in the data index set do not match, it is determined that there is no target data in the data set, that is, the retrieval result is empty. At this time, the user can be prompted to re-enter the target retrieval information until the target data features and the sample data features corresponding to each mapping relationship in the data index set can be matched, and then the target mapping relationship corresponding to the target data features can be obtained according to the data index set and the target data features.

[0095] S206: Determine the target data location information corresponding to the target data feature based on the target mapping relationship, and search for the target data based on the target data location information.

[0096] After determining the target mapping relationship corresponding to the target data feature, the target data location information corresponding to the target data feature is obtained through the target mapping relationship, and then the target data is searched according to the target data location information.

[0097] In a feasible implementation, a mapping table can be established based on the mapping relationship between the sample data features of each sample data in the data set and the sample data location information, and the target mapping relationship can be obtained by querying the mapping table. Of course, in some other implementations, a mapping table can be established based on the mapping relationship between the representation value of the sample data features of each sample data in the data set and the sample data location information. Here, the representation value includes but is not limited to the function value obtained after the sample data features are processed by the function.

[0098] An embodiment of the present application provides a data retrieval method, which first establishes a mapping relationship between sample data features of each sample data in a data set and sample data location information, and then obtains a data index set for each sample data in the data set based on the above mapping relationship. When searching, the target search information and data index set of the input target data can be used for searching without traversing each sample data in the data set, thereby greatly increasing the retrieval speed. Moreover, when the number of sample data in the data set increases, the size of the data index set is not greatly affected.

[0099] Figure 4 A schematic diagram of a process for searching a target mapping relationship provided in an embodiment of the present application. Figure 4 As shown, in one embodiment provided in the present application, searching for target data based on target data location information includes:

[0100] S402: Obtain a storage path of the target data based on the target data location information of the target data.

[0101] The target data location information of the target data may include a storage directory of the target data, and a storage path of the target data is obtained based on a directory structure of the storage directory of the target data.

[0102] In a feasible implementation, the directory structure of the storage directory of the target data can be expressed as C: / A / A1 / 1, which can indicate that the target data is stored in a file named 1 under the A1 folder under the A folder in the C drive. Based on the directory structure of the storage directory of the target data, the storage path of the target data can be easily obtained.

[0103] S404: Searching for target data according to the storage path of the target data.

[0104] The target data is obtained by obtaining the location of the target data according to the storage path of the target data. In a feasible implementation, the target data can be directly jumped to the location of the target data based on the storage path of the target data to respond quickly and save retrieval time.

[0105] In an embodiment provided in the present application, establishing a mapping relationship between sample data features of each sample data in a data set and sample data location information includes:

[0106] A mapping relationship between the sample data feature and the data location information of each sample data in the data set is established based on at least one preset hash function.

[0107] The mapping relationship between the sample data feature and the data location information of each sample data in the data set established based on the preset hash function is unique and definite. Here, the hash function is used to map the key value of the sample data feature to the function of the sample data storage location. The key value is the value assigned to this sample data feature.

[0108] The preset hash function can map one or more sample data features to a point on an array of length m. When this point is 1, it means that the sample data exists and the mapping value can represent the storage location of the sample data. Otherwise, it does not exist. The disadvantage of this method is that when there are many sample data features to be detected, there may be duplication. The solution is to use k preset hash functions corresponding to k points. If all points are 1, then the sample data exists. Therefore, the number of preset hash functions can be one or more. When the number of preset hash functions is one, the amount of calculation can be simplified; when the number of hash functions is multiple, the retrieval accuracy can be improved.

[0109] In a feasible implementation, the number of preset hash functions can be 1, 2, 34, 5, 6, 7, 8, etc. When the number of preset hash functions is 8, the time advantage is obvious when the amount of data is relatively large. Take 1 million files to be queried as an example, the misjudgment rate is 0.01%, that is, the number of files that need to be queried in theory is: 1000000×0.01%=100, while the prior art needs to query 1 million files. The query efficiency of this application is 10,000 times that of the original query method.

[0110] Figure 5 A schematic diagram of a process for determining a mapping relationship provided in an embodiment of the present application. Figure 5 As shown, in one embodiment provided in the present application, establishing a mapping relationship between sample data features and data location information of each sample data in a data set based on at least one preset hash function includes:

[0111] S502: Inputting sample data features of each sample data in the data set into at least one preset hash function respectively, and obtaining at least one sample hash value corresponding to the sample data feature of each sample data in the data set.

[0112] Among them, the sample hash value can be a piece of data, which has a special meaning. It is the identity of the corresponding sample data feature or DNA. Here, the sample hash value is introduced based on the preset hash function to process the sample data feature to obtain a shorter piece of identity data. For example, the sample data feature can be a long sentence. When searching for the corresponding sample data feature based on the target data feature, it is not necessary to compare the long sentence, but to convert the long sentence into a hash value for comparison. First, the storage space occupied by the hash value is small. Secondly, the hash value corresponds to the long sentence one by one. Only the hash values ​​of two identical sentences will be the same, and the input length of the hash function is not limited. Therefore, compared with building a dictionary of sample data based on the preset hash function, taking a single file storing 100,000 logs as an example, when the id corresponding to each log is 20 digits, then it is necessary to store these 200,000 digits, which requires about 39MB. However, when the error rate of building a hash value based on the preset hash function is 0.01%, only 100KB can be done when the expected base number is 100,000 logs. The space occupied is 1 / 390 of the dictionary method.

[0113] In a feasible implementation, the sample data feature of the sample data may be one or more. When the sample data feature of a single sample data is one, the sample data feature is respectively input into one or more preset hash functions to obtain the corresponding one or more sample hash values; when the sample data feature of a single sample data is multiple, the multiple sample data features are input into one or more preset hash functions in sequence to obtain one or more sample hash values ​​corresponding to each sample data feature of the sample data.

[0114] S504: merge at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set.

[0115] Among them, when there is one preset hash function, one sample data feature corresponds to one sample hash value, and merging the sample hash values ​​still obtains the sample hash merged value that is the same as the sample hash value; when there are multiple preset hash functions, one sample data feature corresponds to multiple sample hash values, and the sample hash merged value obtained when merging multiple sample hash values ​​has more features. During retrieval, the more features the sample hash merged value has, the higher the retrieval accuracy and the lower the error rate.

[0116] In a feasible implementation, when a sample data feature corresponds to two sample hash values, the two sample hash values ​​may be 0010 and 1000, and the sample hash merge value obtained by merging the two sample hash values ​​may be 1010. When multiple sample hash values ​​have the same bit position as 1, the sample hash merge value is still 1 at that position when merged. In other words, the hash merge value when the same bit position 1 and 1 meet is still 1 at that position, the hash merge value when 1 and 0 meet is 1 at that position, and the hash merge value when the same bit position 0 and 0 meet is 0 at that position.

[0117] S506: Determine sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash combination value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0118] Among them, the hash merge value is uniquely corresponding to the corresponding sample data feature, and different sample data features correspond to different hash merge values. A mapping relationship between the hash merge value and the sample data location information of the corresponding sample data is established, that is, a mapping relationship between the sample data feature and the sample data location information of each sample data in the data set is established. In addition, since the space occupied by the hash merge value is smaller and simpler than that occupied by the sample data feature, the resulting data index set takes up less space when stored, and the retrieval speed is faster when the data index set is used for retrieval.

[0119] In a feasible implementation, the data index set mainly includes the mapping relationship between the sample hash value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set. In addition, the storage method of the data index set can be internal storage or external storage.

[0120] Figure 6 A schematic diagram of a process for confirming a target mapping relationship provided in an embodiment of the present application. Figure 6 As shown, in an embodiment provided in the present application, obtaining a target mapping relationship corresponding to a target data feature according to a data index set and a target data feature includes:

[0121] S602: Input the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature.

[0122] Among them, at least one preset hash function for processing the target data feature is the same as at least one preset hash function for processing the sample data feature. The length of the target data feature may be unlimited. When there is one preset hash function, one target data feature corresponds to one target hash value; when there are multiple preset hash functions, one target data feature corresponds to multiple target hash values.

[0123] In a feasible implementation, the length of the target data feature can be long or short, and different target hash values ​​can be obtained by inputting target data features of different lengths into a preset hash function. Moreover, even if the target data features are of the same length, different target hash values ​​can be obtained by inputting different target data features into a preset hash function.

[0124] S604: merge at least one target hash value to obtain a target hash merge value of a target data feature, determine a sample hash merge value matching the target hash merge value in a data index set, and determine a target mapping relationship based on a sample mapping relationship corresponding to the sample hash merge value.

[0125] The process of merging at least one target hash value to obtain the target hash merged value of the target data feature can refer to the process of merging at least one sample hash value to obtain the sample hash merged value of the sample data feature, which will not be described in detail here. By matching the sample hash merged value and the target hash merged value in the data index set, it is confirmed that there is sample data corresponding to the target data, and then the target mapping relationship is determined based on the sample mapping relationship corresponding to the sample hash merged value.

[0126] In a feasible implementation, determining a sample hash merge value that matches a target hash merge value in a data index set includes: comparing the target hash merge value and each sample hash merge value in the data index set in sequence, and when the same target hash merge value and sample hash merge value are obtained, obtaining a sample hash merge value that matches the target hash merge value in the data index set.

[0127] Figure 7 A schematic diagram of a data retrieval architecture provided in an embodiment of the present application. Figure 7 As shown, in the embodiment provided by the present application, the log in JSON format in the JSON module is first stored in Parquet format. When the data is written to the Parquet module, a BloomFilter index is constructed for a field with a large specific cardinality and many non-repeated values ​​(i.e., sample data features). Bloom filter uses a hash function method to map an element to a point on an array of length m. When this point is 1, the element is in the set, otherwise it is not in the set. The obtained Parquet file is then distributedly stored, that is, the Parquet file is saved in the DFS (distributed file system, Dedicated File Server) module.

[0128] After that, the Parquet file is read and the Bloom Filter index in the meta information of the Parquet file is written to the Clickhouse (column storage database) module. In the retrieval stage, the client issues a retrieval operation, and the terminal executes the query statement corresponding to the retrieval operation. The query statement then generates a Bloom query statement through the Bloom Query Generator module, assembles the Bloom array query conditions, and the Bloom query statement searches for the corresponding Bloom Filter index from the Clickhouse module through the Parser / Analyzer module. After that, the Parser / Analyzer module generates a query task based on the Bloom Filter index, and submits the task to the query engine Flink Cluster module through the Scheduler module, and finally searches for the corresponding Parquet file from the DFS module.

[0129] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0130] Figure 8 This is a schematic diagram of the structure of a data retrieval device provided in an embodiment of the present application. Figure 8 shown.

[0131] The data retrieval device 800 comprises:

[0132] The establishing module 810 is adapted to establish a mapping relationship between the sample data feature of each sample data in the data set and the sample data location information, and to establish a data index set corresponding to each sample data in the data set based on each mapping relationship.

[0133] The response module 820 is adapted to respond to the target search information of the target data input by the search operation, determine the target data features corresponding to the target search information, and obtain the target mapping relationship corresponding to the target data features according to the data index set and the target data features.

[0134] The search module 830 is adapted to determine the target data location information corresponding to the target data feature based on the target mapping relationship, and search for the target data based on the target data location information.

[0135] In an embodiment provided in the present application, the establishment module 810 includes:

[0136] The establishing submodule is adapted to establish a mapping relationship between the sample data feature and the data location information of each sample data in the data set based on at least one preset hash function.

[0137] In one embodiment provided in the present application, establishing a submodule includes:

[0138] The first submodule is adapted to input the sample data features of each sample data in the data set into at least one preset hash function respectively, to obtain at least one sample hash value corresponding to the sample data features of each sample data in the data set.

[0139] The second submodule is adapted to merge at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set.

[0140] The third submodule is adapted to determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash merge value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0141] In one embodiment provided in the present application, the response module 820 includes:

[0142] The target hash value acquisition submodule is adapted to input the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature.

[0143] The target mapping relationship determination submodule is suitable for merging at least one target hash value to obtain a target hash merge value of a target data feature, determining a sample hash merge value that matches the target hash merge value in a data index set, and determining a target mapping relationship based on a sample mapping relationship corresponding to the sample hash merge value.

[0144] In an embodiment provided in the present application, the sample data feature of each sample data in the data set is a feature that each sample data in the data set includes and has a differentiation feature.

[0145] In an embodiment provided in the present application, the data retrieval device 800 includes:

[0146] A storage module, adapted to store the data index set;

[0147] The response module 820 includes:

[0148] The acquisition submodule is adapted to acquire a stored data index set, and obtain a target data index corresponding to the target data feature according to the data index set and the target data feature.

[0149] In an embodiment provided in the present application, the search module 830 includes:

[0150] A storage path determination submodule, adapted to obtain a storage path of the target data based on the target data location information of the target data;

[0151] The target data search submodule is adapted to search for target data according to the storage path of the target data.

[0152] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of a data retrieval terminal provided in an embodiment of the present application. Fig. 9 As shown, the data retrieval terminal 900 may include: at least one processor 901 , at least one network interface 904 , a user interface 903 , a memory 905 , and at least one communication bus 902 .

[0153] The communication bus 902 is used to realize the connection and communication between these components.

[0154] The user interface 903 may include a display screen (Display) and a camera (Camera), and the optional user interface 903 may also include a standard wired interface and a wireless interface.

[0155] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0156] Among them, the processor 901 may include one or more processing cores. The processor 901 uses various interfaces and lines to connect the various parts of the entire experimental operation terminal 900, and executes various functions and processes data of the sample data management terminal 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 905, and calling data stored in the memory 905. Optionally, the processor 901 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 901 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 901, and it can be implemented by a single chip.

[0157] Among them, the memory 905 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 905 includes a non-transitory computer-readable storage medium. The memory 905 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 905 may optionally be at least one storage device located away from the aforementioned processor 901. As Fig. 9 As shown, the memory 905 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a sample data management program.

[0158] exist Fig. 9 In the data retrieval terminal 900 shown, the user interface 903 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 901 can be used to call the data retrieval program applied to the data retrieval terminal stored in the memory 905, and specifically perform the following operations:

[0159] Establishing a mapping relationship between sample data features and sample data location information of each sample data in the data set, and establishing a data index set corresponding to each sample data in the data set based on each mapping relationship;

[0160] In response to target search information of the target data input by the search operation, determine the target data feature corresponding to the target search information, and obtain the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature;

[0161] Based on the target mapping relationship, the target data location information corresponding to the target data feature is determined, and the target data is searched based on the target data location information.

[0162] In some embodiments, the establishment of a mapping relationship between the sample data feature of each sample data in the data set and the sample data location information performed by the processor 901 includes:

[0163] A mapping relationship between the sample data feature and the data location information of each sample data in the data set is established based on at least one preset hash function.

[0164] In some embodiments, the processor 901 performs the step of establishing a mapping relationship between the sample data feature and the data location information of each sample data in the data set based on at least one preset hash function, including:

[0165] Inputting the sample data features of each sample data in the data set into at least one preset hash function respectively, and obtaining at least one sample hash value corresponding to the sample data features of each sample data in the data set;

[0166] Merging at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set;

[0167] Determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0168] In some embodiments, the processor 901 obtains the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature, including:

[0169] Inputting the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature;

[0170] At least one target hash value is merged to obtain a target hash merge value of a target data feature, a sample hash merge value matching the target hash merge value is determined in a data index set, and a target mapping relationship is determined based on a sample mapping relationship corresponding to the sample hash merge value.

[0171] In some embodiments, the sample data features of each sample data in the data set executed by the processor 901 are features that are included in and have differentiation among each sample data in the data set.

[0172] In some embodiments, after the processor 901 executes the establishment of a data index set for each sample data in the corresponding data set based on each mapping relationship, the process further includes:

[0173] Storing data index sets;

[0174] The target mapping relationship corresponding to the target data feature obtained according to the data index set and the target data feature includes:

[0175] The stored data index set is obtained, and the target data index corresponding to the target data feature is obtained according to the data index set and the target data feature.

[0176] In some embodiments, searching for target data based on target data location information performed by processor 901 includes:

[0177] Obtaining a storage path of the target data based on target data location information of the target data;

[0178] Search for target data according to its storage path.

[0179] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0180] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0182] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0183] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0184] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] The above is a description of a data retrieval method, device, terminal and computer-readable storage medium provided in an embodiment of the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

[0186] The embodiment of the present invention discloses A1, a data retrieval method, wherein the method comprises:

[0187] Establishing a mapping relationship between sample data features and sample data location information of each sample data in the data set, and establishing a data index set corresponding to each sample data in the data set based on each mapping relationship;

[0188] In response to target search information of the target data input by the search operation, determine the target data feature corresponding to the target search information, and obtain the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature;

[0189] Based on the target mapping relationship, the target data location information corresponding to the target data feature is determined, and the target data is searched based on the target data location information.

[0190] A2. The data retrieval method as described in A1, wherein establishing a mapping relationship between the sample data feature of each sample data in the data set and the sample data location information comprises:

[0191] A mapping relationship between the sample data feature and the data location information of each sample data in the data set is established based on at least one preset hash function.

[0192] A3. The data retrieval method as described in A2, wherein establishing a mapping relationship between sample data features and data location information of each sample data in the data set based on at least one preset hash function comprises:

[0193] Inputting the sample data features of each sample data in the data set into at least one preset hash function respectively, and obtaining at least one sample hash value corresponding to the sample data features of each sample data in the data set;

[0194] Merging at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set;

[0195] Determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0196] A4. The data retrieval method as described in A3, wherein obtaining the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature comprises:

[0197] Inputting the target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature;

[0198] At least one target hash value is merged to obtain a target hash merge value of a target data feature, a sample hash merge value matching the target hash merge value is determined in a data index set, and a target mapping relationship is determined based on a sample mapping relationship corresponding to the sample hash merge value.

[0199] A5. The data retrieval method as described in A1, wherein the sample data features of each sample data in the data set are features that are included in and have differentiation among each sample data in the data set.

[0200] A6. The data retrieval method as described in any one of A1 to A5, wherein after establishing a data index set for each sample data in the corresponding data set based on each mapping relationship, the method further comprises:

[0201] Storing data index sets;

[0202] The target mapping relationship corresponding to the target data feature obtained according to the data index set and the target data feature includes:

[0203] The stored data index set is obtained, and the target data index corresponding to the target data feature is obtained according to the data index set and the target data feature.

[0204] A7. The data retrieval method as described in any one of A1 to A5, wherein searching for target data based on the target data location information comprises:

[0205] Obtaining a storage path of the target data based on target data location information of the target data;

[0206] Search for target data according to its storage path.

[0207] The embodiment of the present invention further discloses B8, a data retrieval device, wherein the device comprises:

[0208] An establishing module, adapted to establish a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and to establish a data index set corresponding to each sample data in the data set based on each mapping relationship;

[0209] A response module, adapted to respond to target search information of target data input by a search operation, determine target data features corresponding to the target search information, and obtain a target mapping relationship corresponding to the target data features according to a data index set and the target data features;

[0210] The search module is adapted to determine the target data location information corresponding to the target data feature based on the target mapping relationship, and search for the target data based on the target data location information.

[0211] B9. The data retrieval device as described in B8, wherein the establishment module 810 includes:

[0212] The establishing submodule is adapted to establish a mapping relationship between the sample data feature and the data location information of each sample data in the data set based on at least one preset hash function.

[0213] B10. The data retrieval device as described in B9, wherein the establishing submodule comprises:

[0214] The first submodule is adapted to input the sample data features of each sample data in the data set into at least one preset hash function respectively, to obtain at least one sample hash value corresponding to the sample data features of each sample data in the data set;

[0215] The second submodule is adapted to merge at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set;

[0216] The third submodule is adapted to determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash merge value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

[0217] B11. The data retrieval device as described in B10, wherein the response module comprises:

[0218] A target hash value acquisition submodule, adapted to input a target data feature into at least one preset hash function to obtain at least one target hash value of the target data feature;

[0219] The target mapping relationship determination submodule is suitable for merging at least one target hash value to obtain a target hash merge value of a target data feature, determining a sample hash merge value that matches the target hash merge value in a data index set, and determining a target mapping relationship based on a sample mapping relationship corresponding to the sample hash merge value.

[0220] B12. The data retrieval device as described in B8, wherein the sample data features of each sample data in the data set are features that are included in and have differentiation among each sample data in the data set.

[0221] B13. The data retrieval device as described in any one of B8 to B12, wherein the data retrieval device comprises:

[0222] A storage module, adapted to store the data index set;

[0223] The response modules include:

[0224] The acquisition submodule is adapted to acquire a stored data index set, and obtain a target data index corresponding to the target data feature according to the data index set and the target data feature.

[0225] B14. The data retrieval device as described in any one of B8 to B12, wherein the search module comprises:

[0226] A storage path determination submodule, adapted to obtain a storage path of the target data based on the target data location information of the target data;

[0227] The target data search submodule is adapted to search for target data according to the storage path of the target data.

[0228] C15. A terminal, wherein the terminal comprises:

[0229] Processor; and

[0230] A memory arranged to store computer executable instructions which, when executed, cause a processor to perform a method according to any one of A1 to A7.

[0231] D16. A computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, they implement any method of A1 to A7.

Claims

1. A data retrieval method, wherein: The method comprises: Establishing a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and establishing a data index set corresponding to each sample data in the data set based on each mapping relationship; In response to target search information of the target data input by the search operation, determining a target data feature corresponding to the target search information, and obtaining a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature; Based on the target mapping relationship, target data location information corresponding to the target data feature is determined, and the target data is searched based on the target data location information.

2. The data retrieval method according to claim 1, wherein: The step of establishing a mapping relationship between the sample data feature of each sample data in the data set and the sample data location information comprises: A mapping relationship between the sample data feature and the data location information of each sample data in the data set is established based on at least one preset hash function.

3. The data retrieval method according to claim 2, wherein: The step of establishing a mapping relationship between sample data features and data location information of each sample data in the data set based on at least one preset hash function includes: Inputting the sample data features of each sample data in the data set into at least one preset hash function respectively, to obtain at least one sample hash value corresponding to the sample data features of each sample data in the data set; Merging at least one sample hash value corresponding to the sample data feature of each sample data in the data set to obtain a sample hash merge value corresponding to the sample data feature of each sample data in the data set; Determine the sample data location information of each sample data in the data set, and establish a mapping relationship between the sample hash combination value corresponding to the sample data feature of each sample data in the data set and the sample data location information of each sample data in the data set.

4. The data retrieval method according to claim 3, wherein: The obtaining of the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature includes: Inputting the target data feature into the at least one preset hash function to obtain at least one target hash value of the target data feature; The at least one target hash value is merged to obtain a target hash merge value of the target data feature, a sample hash merge value matching the target hash merge value is determined in the data index set, and a target mapping relationship is determined based on a sample mapping relationship corresponding to the sample hash merge value.

5. The data retrieval method according to claim 1, wherein: The sample data features of each sample data in the data set are features that each sample data in the data set includes and has differentiated features.

6. The data retrieval method according to any one of claims 1 to 5, wherein: After the data index set corresponding to each sample data in the data set is established based on each mapping relationship, the method further includes: Storing the data index set; The obtaining of the target mapping relationship corresponding to the target data feature according to the data index set and the target data feature includes: The stored data index set is acquired, and the target data index corresponding to the target data feature is obtained according to the data index set and the target data feature.

7. The data retrieval method according to any one of claims 1 to 5, wherein: The searching for the target data based on the target data location information includes: Obtaining a storage path of the target data based on target data location information of the target data; The target data is searched for according to the storage path of the target data.

8. A data retrieval device, wherein: The device comprises: An establishing module, adapted to establish a mapping relationship between sample data features of each sample data in the data set and the sample data location information, and to establish a data index set corresponding to each sample data in the data set based on each mapping relationship; A response module, adapted to respond to target search information of target data input by a search operation, determine a target data feature corresponding to the target search information, and obtain a target mapping relationship corresponding to the target data feature according to the data index set and the target data feature; The search module is adapted to determine the target data location information corresponding to the target data feature based on the target mapping relationship, and search for the target data based on the target data location information.

9. A terminal, wherein: The terminal includes: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method of any one of claims 1 to 7.

Images