Index structure, methods for inserting and deleting data, and data query methods.

Abstract

This application provides an index structure, a method for inserting and deleting data, and a data query method. The index structure includes a first hash component and a second hash component. The first hash component includes a first hash function, a second hash function, and a two-dimensional fingerprint matrix. The first hash function determines the hash fingerprint of the data, and the second hash function determines a first storage location in the two-dimensional fingerprint matrix based on the hash fingerprint. The first storage location stores the hash fingerprint and records whether the hash fingerprint is duplicate data. The second hash component includes a third hash function and a one-dimensional fingerprint vector. The third hash function determines a second storage location in the one-dimensional fingerprint vector based on the hash fingerprint, and the second storage location records the number of times the hash fingerprint is repeated. The index structure provided by this application can support query, insertion, and deletion operations on large amounts of duplicate data while ensuring high performance.

Description

Technical Field

[0001] This application relates to the field of indexing technology, and in particular to an index structure, a method for inserting and deleting data, and a data query method. Background Technology

[0002] When querying data in a database, filters (such as Bloom filters, Cuckoo filters, and Quotient filters) are typically used to improve query efficiency. In some scenarios, databases contain duplicate data, especially when the data follows a power-law distribution, where a large number of data points may be repeated. In such cases, filters used for data querying must support fast querying of large amounts of duplicate data. Furthermore, to enable dynamic updates to this large amount of duplicate data, filters must also support fast insertion and deletion operations. However, currently used filters have certain limitations and cannot guarantee efficient querying, insertion, or deletion operations on large amounts of duplicate data. Summary of the Invention

[0003] This application provides an index structure, a method for inserting and deleting data, and a data query method to address the limitations of currently used filters, which cannot guarantee efficient querying, efficient insertion, or efficient deletion operations on large amounts of duplicate data.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] Firstly, an index structure is provided, including a first hash component and a second hash component, wherein:

[0006] The first hash component includes a first hash function, a second hash function, and a two-dimensional fingerprint matrix. The first hash function is used to determine the hash fingerprint of the data. The second hash function is used to determine a first storage location in the two-dimensional fingerprint matrix based on the hash fingerprint. The first storage location is used to store the hash fingerprint and record whether the hash fingerprint is duplicate data.

[0007] The second hash component includes a third hash function and a one-dimensional fingerprint vector. The third hash function is used to determine a second storage location in the one-dimensional fingerprint vector based on the hash fingerprint. The second storage location is used to record the number of times the hash fingerprint is repeated.

[0008] In a second aspect, a method for inserting data into an index structure is provided, the index structure including the index structure described in the first aspect above, the method comprising:

[0009] The hash fingerprint of the data to be inserted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0010] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0011] If the first storage location contains fingerprint data identical to the hash fingerprint, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions of the hash fingerprint is recorded in the second storage location.

[0012] If no fingerprint data identical to the hash fingerprint is stored in the first storage location, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0013] Thirdly, a method for deleting data in an index structure is provided, the index structure including the index structure described in the first aspect above, the method comprising:

[0014] The hash fingerprint of the data to be deleted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0015] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0016] If the first storage location contains fingerprint data identical to the hash fingerprint, determine whether the hash fingerprint is duplicate data based on the records in the first storage location.

[0017] If the hash fingerprint is not duplicate data, delete the hash fingerprint from the first storage location;

[0018] In the case that the hash fingerprint is duplicate data, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of duplicates recorded in the second storage location is decremented by one.

[0019] Fourthly, a data query method based on an index structure is provided, wherein the index structure includes the index structure described in the first aspect above, and the method includes:

[0020] Receive query requests;

[0021] Obtain multiple index files corresponding to multiple data files, with one index file corresponding to each data file, and each index file including the index structure;

[0022] Based on the query request and the multiple index files, a valid file is determined from the multiple data files;

[0023] Retrieve the target data corresponding to the query request from the valid file.

[0024] Fifthly, an electronic device is provided, comprising:

[0025] processor;

[0026] Memory used to store the processor's executable instructions;

[0027] The processor is configured to execute the instructions to implement the method as described in the second, third, or fourth aspect.

[0028] A sixth aspect provides a computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the second, third, or fourth aspect.

[0029] In a seventh aspect, a computer program product is provided, the computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the method as described in the second aspect, or to perform some or all of the steps in the method as described in the third aspect, or to perform some or all of the steps in the method as described in the fourth aspect.

[0030] Based on the index structure provided in this application embodiment, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint based on the hash fingerprint, when performing data queries, the first storage location can be determined based on the first hash component, and the data query can be performed in the first storage location. This can narrow the query range in the index structure, thereby achieving fast data query. Since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location based on the hash fingerprint of the data and record the number of times the data is repeated in the second storage location. Therefore, when deleting or inserting data, it can determine whether the data to be processed is duplicate data based on the record in the first storage location, and determine the number of times the data to be processed is repeated based on the record in the second storage location of the second hash component. Thus, by modifying the record in the first storage location indicating whether the data to be processed is duplicate data and / or the record in the second storage location indicating the number of times the data to be processed is repeated, fast deletion or insertion of the data to be processed can be achieved. The index structure provided in this application embodiment can support query, insertion, and deletion operations on a large amount of duplicate data while ensuring high performance. It not only solves the problem of index unavailability caused by data updates and duplicate data insertion and deletion in streaming scenarios, but also supports efficient data skipping, further improving query efficiency. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is an illustrative system architecture diagram provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of an embodiment of the index structure of this application;

[0034] Figure 3 This is a schematic diagram of a two-dimensional fingerprint matrix according to an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of a one-dimensional fingerprint vector according to an embodiment of this application;

[0036] Figure 5 This is a schematic flowchart illustrating a method for inserting data into an index structure according to an embodiment of this application;

[0037] Figure 6This is a schematic diagram illustrating the insertion of data into an index structure according to an embodiment of this application;

[0038] Figure 7 This is a schematic diagram illustrating the insertion of data into an index structure, representing yet another embodiment of this application.

[0039] Figure 8 This is a flowchart illustrating a method for deleting data in an index structure according to an embodiment of this application;

[0040] Figure 9 This is a schematic diagram illustrating the deletion of data in an index structure according to an embodiment of this application;

[0041] Figure 10 This is a flowchart illustrating a data query method based on an index structure, according to an embodiment of this application.

[0042] Figure 11 This is a flowchart illustrating another embodiment of the data query method based on an index structure in this application;

[0043] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0044] Figure 13 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0045] Figure 14 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0046] Figure 15 This is a schematic diagram of a device structure for inserting data into an index structure according to an embodiment of this application;

[0047] Figure 16 This is a schematic diagram of a device structure for deleting data in an index structure according to an embodiment of this application;

[0048] Figure 17 This is a schematic diagram of the structure of a data query device based on an index structure, according to an embodiment of this application. Detailed Implementation

[0049] In practical applications, duplicate data is a common occurrence in databases, especially when the data follows a power-law distribution, where a large number of data points may be repeated. When storing large amounts of duplicate data in a database, filters must support fast querying of this large amount of duplicate data to ensure query efficiency. Furthermore, to enable dynamic updates to duplicate data, filters must also support fast insertion and deletion of large amounts of duplicate data. However, currently used filters have limitations and cannot support efficient querying, insertion, or deletion of large amounts of duplicate data.

[0050] Take commonly used filters such as Bloom filters, Cuckoo filters, and Quotient filters as examples.

[0051] A Bloom filter is a probabilistic data structure that uses multiple hash functions to map elements to multiple bits and uses these bits to determine if data exists. However, once a bit is set to 1, it cannot be arbitrarily set to 0, causing Bloom filters to be unable to support data deletion operations.

[0052] The Cuckoo Filter is a data structure for fast element lookup. It uses multiple hash buckets to map and store data, calculating the data's location based on two hash functions and an XOR operation. The data's fingerprint is then stored in the free slots of the two hash buckets. However, while the Cuckoo Filter supports data deletion, its structural characteristics—that identical hash values ​​must be placed in the same two space-constrained hash buckets—make it ineffective at handling large amounts of duplicate data.

[0053] The core idea of ​​a quotient filter is to decompose the hash value of an element into a quotient and a remainder, determine the storage location based on the quotient, and store the remainder in that location. While quotient filters can support fast query, insertion, and deletion operations on data, their performance degrades significantly when dealing with a large amount of duplicate data.

[0054] This application provides an index structure that supports dynamic deletion and insertion of data. Especially when dealing with large amounts of duplicate data, it can effectively build and maintain the index structure while ensuring high performance in query operations, guaranteeing both time and space efficiency. This application also provides a method for inserting and deleting data in the index structure, and a data query method based on the index structure. These methods can support querying, inserting, and deleting large amounts of duplicate data while maintaining high performance. This solves the problem of index unavailability caused by data updates and duplicate data insertion / deletion in streaming scenarios, and also supports efficient data skipping, further improving query efficiency. The index structure includes a first hash component and a second hash component. The first hash component includes a first hash function, a second hash function, and a two-dimensional fingerprint matrix. The first hash function is used to determine the hash fingerprint of the data, and the second hash function is used to determine a first storage location in the two-dimensional fingerprint matrix based on the hash fingerprint. The first storage location stores the hash fingerprint and records whether the hash fingerprint is duplicate data. The second hash component includes a third hash function and a one-dimensional fingerprint vector. The third hash function is used to determine a second storage location in the one-dimensional fingerprint vector based on the hash fingerprint, and the second storage location records the number of times the hash fingerprint is repeated.

[0055] Based on the index structure provided in this application embodiment, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint based on the hash fingerprint, when performing data queries, the first storage location can be determined based on the first hash component, and the data query can be performed in the first storage location. This can narrow the query range in the index structure, thereby achieving fast data query. Since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location based on the hash fingerprint of the data and record the number of times the data is repeated in the second storage location. Therefore, when deleting or inserting data, it can determine whether the data to be processed is duplicate data based on the record in the first storage location, and determine the number of times the data to be processed is repeated based on the record in the second storage location of the second hash component. Thus, by modifying the record in the first storage location indicating whether the data to be processed is duplicate data and / or the record in the second storage location indicating the number of times the data to be processed is repeated, fast deletion or insertion of the data to be processed can be achieved. The index structure provided in this application embodiment can support query, insertion, and deletion operations on a large amount of duplicate data while ensuring high performance. It not only solves the problem of index unavailability caused by data updates and duplicate data insertion and deletion in streaming scenarios, but also supports efficient data skipping, further improving query efficiency.

[0056] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings of one or more embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0057] The terms "first," "second," etc., used in this application and the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this application and the claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0058] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0059] Figure 1 The system architecture shown can be logically divided into a compute layer 11 and a storage layer 12. The compute layer 11 is responsible for data management and access, and its data lake table format can be selected according to actual needs. The storage layer 12 is responsible for data storage and can use different storage formats, such as Parquet, Avro, and ORC, and can choose different storage methods, such as local storage, distributed storage, or cloud storage. The compute layer 21 and the storage layer 22 can communicate via a network connection.

[0060] The big data computing engine in Computation Layer 11 is a tool for processing large-scale data, enabling efficient data processing and analysis. Commonly used big data computing engines in the open-source community include Apache Spark, Apache Flink, and Presto. These engines play important roles in various scenarios and applications, providing users with rich data processing and analysis capabilities.

[0061] Data lake tables in Compute Layer 11 are a table format that enables data lakes to perform ACID (atomicity, consistency, isolation, and durability) transactions and enhances data management capabilities. In the open-source world, products such as Iceberg, Hudi, and Delta Lake are open-source solutions for data lake tables.

[0062] In actual deployment, depending on the application scenario, the deployment of the computing engine can include, but is not limited to, the following scenarios (1) to (5):

[0063] (1) A big data operating environment that does not use data lake tables for deployment;

[0064] (2) A big data operating environment deployed using data lake tables;

[0065] (3) Data warehouse operating environment that does not use data lake tables;

[0066] (4) The operating environment of the data warehouse deployed using data lake tables;

[0067] (5) Use a lake warehouse integrated operating environment that is jointly deployed with data lake tables, big data and data warehouse.

[0068] The operating environment described in any one of (1) to (5) above may include, but is not limited to, physical machine environment and cloud environment, without specific limitations.

[0069] The technical solutions provided in this application can be applied to data storage devices such as databases and data warehouses, as well as big data processing scenarios.

[0070] The index structure provided in this application embodiment can be applied to different data types. That is, the index structure provided in this application embodiment can support query, deletion and insertion operations on different types of data, including but not limited to integer, floating-point, character and date types.

[0071] The technical solutions provided in the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0072] Figure 2 This is a schematic diagram of an index structure according to an embodiment of this application.

[0073] Figure 2 The index structure shown includes a first hash component and a second hash component. The first hash component includes a first hash function, a second hash function, and a two-dimensional fingerprint matrix. The first hash function determines the hash fingerprint of the data, and the second hash function determines a first storage location in the two-dimensional fingerprint matrix based on the hash fingerprint. This first storage location stores the hash fingerprint and records whether the hash fingerprint is duplicate data. The second hash component includes a third hash function and a one-dimensional fingerprint vector. The third hash function determines a second storage location in the one-dimensional fingerprint vector based on the hash fingerprint, and this second storage location records the number of times the hash fingerprint is repeated.

[0074] The first hash function can be represented as a fingerprint hash function. The hash fingerprint determined by the first hash function is unique; that is, different data will yield different hash fingerprints when determining the data fingerprint using the first hash function. The length of the hash fingerprint can be a default value or specified by the user; no specific limitation is made here. For example, the user can specify the hash fingerprint length as 32 bits.

[0075] The second hash function can be represented as a mapping hash function. The first storage location determined by the second hash function can be one of multiple storage locations in the two-dimensional fingerprint matrix. In some implementations, the second hash function is used to determine the first storage location in the two-dimensional fingerprint matrix; specifically, the second hash function can be used to determine multiple row indices in the two-dimensional fingerprint matrix, and the first storage location can be a specific storage location in the row corresponding to those multiple row indices. In this way, when storing hash fingerprints in the index structure, a storage location can be flexibly selected from multiple rows of the two-dimensional fingerprint matrix for storage.

[0076] It should be noted that the multiple row indices determined by the second hash function can be row indices of a subset of rows in the two-dimensional fingerprint matrix. This facilitates data queries based on the index structure by determining a subset of rows in the two-dimensional fingerprint matrix using the second hash function and performing data queries within that subset, thereby narrowing the query scope and improving query efficiency. Furthermore, for different data, the second hash function can yield different row indices or the same row indices. When the same row indices are obtained, different data will correspond to different first storage locations, allowing the hash fingerprints of different data to be stored in different first storage locations.

[0077] When the second hash function is used to determine multiple row indices in the two-dimensional fingerprint matrix, the second hash function may include multiple mapping hash functions, each mapping hash function being used to determine one row index in the two-dimensional fingerprint matrix, and the multiple mapping hash functions being used to determine multiple row indices in the two-dimensional fingerprint matrix. Optionally, in some embodiments, the first hash function may specifically include two mapping hash functions, which can be represented as a first mapping hash function and a second mapping hash function, respectively. The first mapping hash function is used to determine a first row index based on the hash fingerprint and the number of rows in the two-dimensional fingerprint matrix, and the second mapping hash function is used to determine a second row index based on the hash fingerprint and the first row index. The first row index and the second row index can both be row indices in the two-dimensional fingerprint matrix. Specifically, the first mapping hash function can determine the first row index by taking the modulo operation of the hash fingerprint with the number of rows in the two-dimensional fingerprint matrix. Similarly, the second mapping hash function can determine the second row index by performing an XOR operation on the hash fingerprint with the first row index.

[0078] Optionally, in some implementations, the first mapping hash function can be expressed as:

[0079] B1(x)=F p modm;

[0080] The second mapping hash function can be expressed as:

[0081] B2(x)=B1(x)⊕Hash(F p ).

[0082] Among them, F p Let m be the hash fingerprint of the data, and m be the number of rows in the two-dimensional fingerprint matrix.

[0083] Two-dimensional fingerprint matrices can be implemented using different data structures, such as dictionaries, linked lists, and vectors. This embodiment will use an array structure, which is the most efficient data structure for the two-dimensional fingerprint matrix, as an example. Furthermore, for simplicity, in the following description, a "bucket" will represent a row of data in the two-dimensional fingerprint matrix, and a "slot" will represent an element (i.e., storage location) within the "bucket".

[0084] The two-dimensional fingerprint matrix includes multiple storage locations, each with the same structure. Taking the first storage location as an example, in some implementations, the first storage location may consist of data bits and flag bits. The data bits are used to store the hash fingerprint of the data, and the flag bits are used to record whether the hash fingerprint of the data is duplicate data. Specifically, a flag bit value of 0 indicates that the hash fingerprint of the data is not duplicate data (equivalent to the data not being duplicated), and a flag bit value of 1 indicates that the hash fingerprint of the data is duplicate data (equivalent to the data being duplicated). Alternatively, a flag bit value of 1 can indicate that the hash fingerprint of the data is not duplicate data (equivalent to the data not being duplicated), and a flag bit value of 0 can indicate that the hash fingerprint of the data is duplicate data (equivalent to the data being duplicated). This embodiment uses a flag bit value of 0 to represent non-duplicate data and a value of 1 to represent duplicate data as an example for explanation.

[0085] To facilitate understanding the structure of the two-dimensional fingerprint matrix, the following will use... Figure 3 The following is an example of a more specific implementation method.

[0086] Figure 3 This is a schematic diagram of a two-dimensional fingerprint matrix according to an embodiment of this application. Figure 3 The two-dimensional fingerprint matrix shown includes 4 rows and 4 columns (in practical applications, it can include more rows and more columns). The row indices of the 4 rows are represented as bucket[0], bucket[1], bucket[2] and bucket[3], respectively, and the column indices of the 4 columns (that is, the 4 elements in each row) are represented as slot[0], slot[1], slot[2] and slot[3], respectively. Figure 3 The two-dimensional fingerprint matrix shown includes 4×4 storage locations (i.e., 4×4 elements), and each storage location has the same structure. Taking the first storage location in the first row (corresponding to slot [0] in bucket [0]) as an example, this storage location includes a data bit and a flag bit. The data bit is used to store the hash fingerprint of the data, and the flag bit is used to record whether the hash fingerprint stored in the data bit is duplicate data.

[0087] The second storage location determined by the third hash function can be one of multiple storage locations for the one-dimensional fingerprint vector. Specifically, different third hash fingerprints can yield different second storage locations, allowing the number of repetitions of different hash fingerprints to be recorded at different second storage locations.

[0088] In some implementations, the third hash function may include a modulo function, which determines the remainder when the hash fingerprint of the data is divided by the length of the one-dimensional fingerprint vector. This remainder indicates a second storage location within the one-dimensional fingerprint vector. For example, the remainder may be a position index within the one-dimensional fingerprint vector, and the storage location corresponding to that position index is the second storage location.

[0089] Alternatively, in some implementations, the modulus function can be expressed as:

[0090] Modulo(F p ) = F p %n.

[0091] Among them, F p Let n be the hash fingerprint of the data, and n be the length of the one-dimensional fingerprint vector.

[0092] One-dimensional fingerprint vectors can be implemented using different data structures, such as dictionaries, linked lists, and vectors. This embodiment will use an array structure, which is the most efficient data structure for one-dimensional fingerprint vectors, as an example. Furthermore, for simplicity, in the following description, "slots" will be used to represent elements (i.e., storage locations) in a one-dimensional fingerprint vector.

[0093] Optionally, in some embodiments, the second hash component may further include a fourth hash function, which is used to determine a fingerprint identifier for the hash fingerprint, and the fingerprint identifier is used to identify the uniqueness of the fingerprint data. When the second hash component includes a fourth hash function, the second storage location is also used to store the fingerprint identifier so that the fingerprint identifier and the number of repetitions of the hash fingerprint are stored in a corresponding manner.

[0094] Optionally, in some embodiments, the fourth hash function may include a quotient function, which is used to determine the quotient of the hash fingerprint of the data and the length of the one-dimensional fingerprint vector. Correspondingly, the fingerprint identifier of the hash fingerprint may include this quotient. In some embodiments, the quotient function may be expressed as:

[0095] Quotient(F p ) = F p / n.

[0096] Among them, F p Let n be the hash fingerprint of the data, and n be the length of the one-dimensional fingerprint vector.

[0097] A one-dimensional fingerprint vector includes multiple storage locations, each with the same structure. Taking the second storage location as an example, in some implementations, when the second storage location is used to record the repetition count of the hash fingerprint, it includes a counter bit. When the second storage location is used to store the fingerprint identifier of the hash fingerprint and record the repetition count, it can consist of data bits and a counter bit. The data bits are used to store the fingerprint identifier of the hash fingerprint, and the counter bit is used to record the repetition count of the hash fingerprint. The number of bits in the counter bit can be specified by the user or be a default value; for example, the user can specify that the number of bits in the counter bit is 8.

[0098] To facilitate understanding the structure of a one-dimensional fingerprint vector, the following will use... Figure 4The following is an example of a more specific implementation method.

[0099] Figure 4 This is a schematic diagram of a one-dimensional fingerprint vector according to an embodiment of this application. Figure 4 The length of the one-dimensional fingerprint vector shown is 4 (it can be longer in practical applications), that is, the one-dimensional fingerprint vector includes 4 storage locations, which can be represented as slot [0], slot [1], slot [2] and slot [3] respectively. Figure 4 In the one-dimensional fingerprint vector shown, each storage location has the same structure. Taking the first storage location (corresponding to slot [0]) as an example, this storage location includes data bits and count bits. The data bits are used to store the fingerprint identifier of the hash fingerprint, and the count bits are used to record the number of times the hash fingerprint corresponding to the fingerprint identifier stored in the data bits is repeated, that is, to record the number of times the data corresponding to the hash fingerprint is repeated.

[0100] It should be noted that if the second hash component includes a third hash function but does not include a fourth hash function, that is, if the storage location in the one-dimensional fingerprint vector is used to record the number of repetitions of the hash fingerprint but not to store the fingerprint identifier of the hash fingerprint, then... Figure 4 Each storage location shown will include count bits but not data bits.

[0101] The index structure provided in this application embodiment, because the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint based on the hash fingerprint, can determine the first storage location based on the first hash component and perform data query in the first storage location when performing data query, thereby narrowing the query range in the index structure and achieving fast data query. Since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location based on the hash fingerprint of the data and record the number of times the data is repeated in the second storage location. Therefore, when deleting or inserting data, it can determine whether the data to be processed is duplicate data based on the record in the first storage location and determine the number of times the data to be processed is repeated based on the record in the second storage location of the second hash component. Thus, by modifying the record in the first storage location indicating whether the data to be processed is duplicate data and / or the record in the second storage location indicating the number of times the data to be processed is repeated, fast deletion or insertion of the data to be processed can be achieved. The index structure provided in this application embodiment can support query, insertion, and deletion operations on a large amount of duplicate data while ensuring high performance. It not only solves the problem of index unavailability caused by data updates and duplicate data insertion and deletion in streaming scenarios, but also supports efficient data skipping, further improving query efficiency.

[0102] The index structure provided in this application can store the hash fingerprint of data in the first hash component, ensuring fast query efficiency during data retrieval. Duplicate data can be handled by placing it in the second hash component. This index structure sets a counter bit for only a small portion of duplicate data and a flag bit for most single data, effectively reducing the space occupied by the index structure.

[0103] Based on the index structure provided in the embodiments of this application, the embodiments of this application also provide a method for inserting data into the index structure, a method for deleting data from the index structure, and a data query method based on the index structure. The following will use... Figures 5 to 12 The illustrated embodiment will be used as an example for explanation.

[0104] Figure 5 This is a flowchart illustrating a method for inserting data into an index structure according to an embodiment of this application. The index structure is the one provided in this application embodiment; for details, please refer to [link to relevant documentation]. Figures 2 to 4 The embodiments shown will not be described again here. Figure 5 The method shown includes the following steps.

[0105] Step S502: Determine the hash fingerprint of the data to be inserted according to the first hash function and determine the first storage location in the two-dimensional fingerprint matrix according to the second hash function.

[0106] When inserting data into an index structure, the hash fingerprint of the data to be inserted can first be determined based on the first hash function in the index structure. In some implementations, the first hash function can be represented as Hash(). The hash fingerprint of the data to be inserted can be determined using the following formula:

[0107] F p =Hash(x).

[0108] After obtaining the hash fingerprint of the data to be inserted, the first storage location in the two-dimensional fingerprint matrix can be determined according to the second hash function.

[0109] In some implementations, the second hash function may include multiple mapping hash functions, each mapping hash function being used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions being used to determine multiple row indices in the two-dimensional fingerprint matrix. Thus, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indices in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically the storage location in the row corresponding to the multiple row indices in the two-dimensional fingerprint matrix.

[0110] Taking the second hash function, which includes two hash mapping functions B1() and B2(), as an example, the first storage location of the two-dimensional fingerprint matrix can be determined using the following formula:

[0111] B1(x)=F p modm;

[0112] B2(x)=B1(x)⊕Hash(F p ).

[0113] Where m is the number of rows in the two-dimensional fingerprint matrix, F p This is the hash fingerprint of the data to be inserted.

[0114] Based on the two hash mapping functions B1() and B2() mentioned above, we can obtain two row indices of the two-dimensional fingerprint matrix. These two row indices point to two rows in the two-dimensional fingerprint matrix. The first storage location is the storage location in these two rows, indicating that the hash fingerprint of the data to be inserted can be stored in these two rows.

[0115] Step S504: Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint of the data to be inserted.

[0116] After determining the first storage location, before inserting the hash fingerprint of the data to be inserted into the index structure, it is necessary to determine whether the data to be inserted is duplicate data. This allows for different data insertion operations to be performed based on the determination result. To determine if the data to be inserted is duplicate data, we can check if the first storage location stores a fingerprint that matches the hash fingerprint of the data to be inserted. If the first storage location stores a fingerprint that matches the hash fingerprint of the data to be inserted, it means that the same data has already been inserted into the index structure, and the data to be inserted is duplicate data. If the first storage location does not store a fingerprint that matches the hash fingerprint of the data to be inserted, it means that the same data has not been inserted into the index structure before, and the data to be inserted is not duplicate data.

[0117] In some implementations, the first storage location may be a storage location in one of the two rows of a two-dimensional fingerprint matrix. In this case, when determining whether the first storage location stores fingerprint data that is the same as the hash fingerprint of the data to be inserted, the system can search for fingerprint data that is the same as the hash fingerprint of the data to be inserted in the two rows of the two-dimensional fingerprint matrix. If it exists, it can be determined that the data to be inserted is duplicate data; if it does not exist, it can be determined that the data to be inserted is not duplicate data.

[0118] If the first storage location contains fingerprint data that is identical to the hash fingerprint of the data to be inserted, i.e., if the data to be inserted is duplicate data, step S506 can be executed. If the first storage location does not contain fingerprint data that is identical to the hash fingerprint of the data to be inserted, i.e., if the data to be inserted is not duplicate data, step S508 can be executed.

[0119] Step S506: If the first storage location contains fingerprint data that is the same as the hash fingerprint of the data to be inserted, determine the second storage location in the one-dimensional fingerprint vector according to the third hash function, and record the number of repetitions of the hash fingerprint of the data to be inserted in the second storage location.

[0120] If the first storage location contains fingerprint data that is identical to the hash fingerprint of the data to be inserted, the insertion operation of the data to be inserted can be achieved by modifying the number of repetitions of the hash fingerprint of the data to be inserted recorded in the index structure (equivalent to the number of repetitions of the data to be inserted). This eliminates the need to repeatedly store the hash fingerprint of the data to be inserted in the two-dimensional fingerprint matrix, thereby avoiding duplicate storage and reducing the memory usage of the index structure.

[0121] When modifying the number of repetitions of the hash fingerprint of the data to be inserted recorded in the index structure, the second storage location in the one-dimensional fingerprint vector can be determined according to the third hash function in the index structure, and the number of repetitions of the hash fingerprint of the data to be inserted can be recorded at the second storage location.

[0122] In some implementations, recording the number of repetitions of the hash fingerprint of the data to be inserted in the second storage location may include the following steps:

[0123] Determine whether the hash fingerprint of the data to be inserted has already been recorded in the first storage location as duplicate data;

[0124] If the hash fingerprint of the data to be inserted is not recorded in the first storage location and is a duplicate, the number of times the hash fingerprint of the data to be inserted is recorded in the first storage location and is a duplicate in the second storage location is 2.

[0125] If the hash fingerprint of the data to be inserted is already recorded in the first storage location and is a duplicate, increment the number of duplicates recorded in the second storage location by one.

[0126] The first storage location is used to store the hash fingerprint of the data to be inserted and to record whether the hash fingerprint of the data to be inserted is duplicate data. If it is determined that the data to be inserted is duplicate data, it can be further determined whether the first storage location has already recorded the hash fingerprint of the data to be inserted as duplicate data. In some embodiments, the first storage location includes data bits and flag bits. The data bits are used to store the hash fingerprint of the data to be inserted, and the flag bits are used to record whether the hash fingerprint of the data to be inserted is duplicate data. When determining whether the first storage location has recorded the hash fingerprint of the data to be inserted as duplicate data, the determination can be made based on the flag bits in the first storage location. For example, if the flag bit value of 0 indicates duplicate data and a value of 1 indicates non-duplicate data, if the flag bit in the first storage location is 0, it means that the first storage location has not recorded the hash fingerprint of the data to be inserted as duplicate data; if the flag bit in the first storage location is 1, it means that the first storage location has recorded the hash fingerprint of the data to be inserted as duplicate data.

[0127] If the hash fingerprint of the data to be inserted is not recorded as duplicate data in the first storage location, it is necessary to record that the hash fingerprint of the data to be inserted is duplicate data in the first storage location. For example, the value of the flag bit in the first storage location can be set to 1. Furthermore, if the hash fingerprint of the data to be inserted is not recorded as duplicate data in the first storage location, it indicates that the hash fingerprint of the data to be inserted is a second repetition. Therefore, the repetition count needs to be recorded as 2 in the second storage location. For example, if the second storage location includes a counter bit used to record the repetition count of the hash fingerprint of the data to be inserted, the value of the counter bit can be set to 2.

[0128] If the hash fingerprint of the data to be inserted is already recorded as duplicate data in the first storage location, the count of duplicates recorded in the second storage location can be directly incremented by one without modifying the record in the first storage location. For example, if the count bit in the second storage location is 2 and the flag bit in the first storage location is 1, the count bit in the second storage location can be set to 3 without modifying the value of the flag bit in the first storage location.

[0129] Optionally, in some implementations, where the second hash component in the index structure includes a fourth hash function, the following steps may also be included:

[0130] The fingerprint identifier of the hash fingerprint of the data to be inserted is determined according to the fourth hash function;

[0131] The fingerprint identifier is stored in the second storage location.

[0132] When storing the fingerprint identifier in the second storage location, the fingerprint identifier can be stored in the data bits of the second storage location. The fourth hash function can be a quotient function, and the fingerprint identifier can be the quotient of the hash fingerprint of the data to be inserted and the length of the one-dimensional fingerprint vector.

[0133] Step S508: If the first storage location does not store fingerprint data that is the same as the hash fingerprint of the data to be inserted, store the hash fingerprint of the data to be inserted in the first storage location and record that the hash fingerprint of the data to be inserted is not duplicate data.

[0134] In some implementations, the second hash function in the index structure is used to determine multiple row indices of the two-dimensional fingerprint matrix. The first storage location is the storage location in the multiple rows of the two-dimensional fingerprint matrix that correspond to the multiple row indices. In this way, when the hash fingerprint of the data to be inserted is stored in the first storage location and it is recorded that the hash fingerprint of the data to be inserted is not duplicate data, a storage location can be selected from the multiple rows of the two-dimensional fingerprint matrix that correspond to the multiple row indices to store the hash fingerprint of the data to be inserted and to record that the hash fingerprint of the data to be inserted is not duplicate data.

[0135] When selecting a storage location from multiple rows corresponding to multiple row indices in a two-dimensional fingerprint matrix, optionally, the row with the lowest load can be determined first from the multiple rows corresponding to multiple row indices. Then, a storage location can be selected from the row with the lowest load to store the hash fingerprint of the data to be inserted, and it can be recorded that the hash fingerprint of the data to be inserted is not duplicate data. Here, the load can be the number of storage locations in the row that already contain data. The more storage locations there are, the higher the load of the row, and the fewer storage locations there are, the lower the load of the row. By selecting the storage location in the row with the lowest load to store the data, load balancing of each row of data in the two-dimensional fingerprint matrix can be achieved.

[0136] To facilitate understanding of the method for inserting data into an index structure provided in the embodiments of this application, the following will use... Figure 6 and Figure 7 The following is an example of a more specific implementation method.

[0137] Figure 6 and Figure 7 The index structure shown includes a first hash component and a second hash component. The first hash component includes a fingerprint hash function and two mapping hash functions. Figure 6 and Figure 7 (not shown) and a two-dimensional fingerprint matrix, the second hash component includes a quotient function and a remainder function (not shown). Figure 6 and Figure 7 (None of which are shown) and a one-dimensional fingerprint vector. The two-dimensional fingerprint matrix consists of 4 buckets, each bucket has 4 slots, and the one-dimensional fingerprint vector consists of 4 slots.

[0138] Suppose that the date column of data table T1 has data [20000506, 20060908, 20120504, 20151212, 20180909, 20200202, 20230801, 20240301] flowing in. The hash fingerprints of these data calculated by the fingerprint hash function are [11, 13, 15, 17, 102, 104, 106, 108]. All hash fingerprints are mapped to buckets [1] and [2] in the two-dimensional fingerprint matrix through two mapping hash functions. Then, when constructing the index structure provided in this application embodiment for the streaming data T1.date, the following can be obtained: Figure 6 The results are shown.

[0139] If two more data entries with a date of 20230801 are subsequently added to data table T1, then an update is required. Figure 6 The index structure shown. Specifically, as... Figure 7 As shown, the hash fingerprint 106 of 20230801 can be calculated first according to the fingerprint hash function. Then, according to the two mapping hash functions, the hash fingerprint 106 points to the buckets [1] and [2] in the two-dimensional fingerprint matrix respectively. Since the same fingerprint data 106 already exists in the bucket [1], the flag bit corresponding to 106 is set to 1, and then the data is inserted into the one-dimensional fingerprint vector. When inserting data into the one-dimensional fingerprint vector, since the one-dimensional fingerprint vector has a total of 4 slots, the quotient and remainder of the hash fingerprint 106 relative to 4 are 26 and 2 respectively. Therefore, the quotient 26 can be stored in the storage location corresponding to the slot [2] of the one-dimensional fingerprint vector. Since a total of 3 data with date 20230801 are inserted into the data table T1, the count bit in the slot [2] is set to 3.

[0140] Suppose that another data point with a date of 20230801 is subsequently added to data table T1, then it is necessary to... Figure 7 The index structure shown is updated. Figure 7 When updating the index structure shown, based on the method described above, the count bit in the slot [2] of the one-dimensional fingerprint vector can be directly incremented by 1.

[0141] As can be seen, when inserting data into the index structure, the fingerprint of the data is first calculated using the fingerprint hash function, and the positions of the two hash buckets are calculated using two mapping hash functions. Then, it is checked whether there is the same fingerprint data in the two-dimensional fingerprint matrix. If not, it is directly stored in an empty slot with the flag position set to 0. If there is the same fingerprint data, its flag bit is checked. If the flag bit is 0, it is set to 1, and the data is inserted into the two-dimensional fingerprint matrix with the corresponding count bit initially set to 2. If the flag bit is 1, the position of the data in the one-dimensional fingerprint vector is found and the count bit is incremented by one.

[0142] It should be noted that in practical applications, when using quotient and remainder functions to determine the quotient and remainder of a hash fingerprint relative to a one-dimensional fingerprint vector, there may be cases where different hash fingerprints have different quotients but the same remainder. In such cases, the fingerprint identifiers and repetition counts of different hash fingerprints will be stored in the same storage location within the one-dimensional fingerprint vector. To store the fingerprint identifiers and repetition counts of different hash fingerprints in the same storage location, a linked list or other storage method can be used, and the repetition counts of different hash fingerprints can be distinguished based on the fingerprint identifiers.

[0143] Based on the method for inserting data into an index structure provided in this application embodiment, since the first storage location in the index structure is used to record whether the hash fingerprint of the data is duplicate data, and the second hash component in the index structure can determine the second storage location based on the hash fingerprint of the data and record the number of times the data is repeated in the second storage location, when inserting data into the index structure, it is possible to determine whether the data to be processed is duplicate data based on the record in the first storage location, and to determine the number of times the data to be processed is repeated based on the record in the second storage location of the second hash component. Thus, by modifying the record in the first storage location indicating whether the data to be processed is duplicate data and / or the record in the second storage location indicating the number of times the data to be processed is repeated, fast insertion of the data to be processed can be achieved. This method can support the insertion operation of a large amount of duplicate data while ensuring high performance, and solves the problem of index unavailability caused by data updates and duplicate data insertion in streaming scenarios.

[0144] Figure 8 This is a flowchart illustrating a method for deleting data in an index structure according to an embodiment of this application. The index structure is the one provided in this application embodiment; for details, please refer to [link to relevant documentation]. Figures 2 to 4 The embodiments shown will not be described again here. Figure 8 The method shown includes the following steps.

[0145] Step S802: Determine the hash fingerprint of the data to be deleted according to the first hash function and determine the first storage location in the two-dimensional fingerprint matrix according to the second hash function.

[0146] When deleting data from an index structure, the hash fingerprint of the data to be deleted can first be determined based on the first hash function in the index structure. In some implementations, the first hash function can be represented as Hash(). The hash fingerprint of the data to be deleted can be determined using the following formula:

[0147] F p =Hash(x).

[0148] After obtaining the hash fingerprint of the data to be deleted, the first storage location in the two-dimensional fingerprint matrix can be determined according to the second hash function.

[0149] In some implementations, the second hash function may include multiple mapping hash functions, each mapping hash function being used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions being used to determine multiple row indices in the two-dimensional fingerprint matrix. Thus, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indices in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically the storage location in the row corresponding to the multiple row indices in the two-dimensional fingerprint matrix.

[0150] Taking the second hash function, which includes two hash mapping functions B1() and B2(), as an example, the first storage location of the two-dimensional fingerprint matrix can be determined using the following formula:

[0151] B1(x)=F p modm;

[0152] B2(x)=B1(x)⊕Hash(F p ).

[0153] Where m is the number of rows in the two-dimensional fingerprint matrix, F p The hash fingerprint of the data to be deleted.

[0154] Based on the two hash mapping functions B1() and B2() mentioned above, we can obtain two row indices of the two-dimensional fingerprint matrix. These two row indices point to two rows in the two-dimensional fingerprint matrix. The first storage location is the storage location in these two rows, indicating that if the index structure stores the hash fingerprint of the data to be deleted, then the hash fingerprint is stored in these two rows.

[0155] Step S804: Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint of the data to be deleted.

[0156] After determining the first storage location, before deleting the hash fingerprint of the data to be deleted from the index structure, it is necessary to determine whether the index structure already stores the hash fingerprint of the data to be deleted. To determine whether the index structure stores the hash fingerprint of the data to be deleted, we can check whether the first storage location stores fingerprint data that is identical to the hash fingerprint of the data to be deleted. If the first storage location stores fingerprint data that is identical to the hash fingerprint of the data to be deleted, then it can be determined that the index structure stores the hash fingerprint of the data to be deleted. If the first storage location does not store fingerprint data that is identical to the hash fingerprint of the data to be inserted, then it can be determined that the index structure does not store the hash fingerprint of the data to be deleted.

[0157] In some implementations, the first storage location can be a storage location in one of the two rows of a two-dimensional fingerprint matrix. In this case, when determining whether the first storage location stores fingerprint data that is the same as the hash fingerprint of the data to be inserted, the system can search for fingerprint data that is the same as the hash fingerprint of the data to be inserted in the hash fingerprints stored in the two rows of the two-dimensional fingerprint matrix. If it exists, it can be determined that the index structure stores the hash fingerprint of the data to be deleted. If it does not exist, it can be determined that the index structure does not store the hash fingerprint of the data to be deleted.

[0158] If the first storage location contains hash fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the index structure stores the hash fingerprint of the data to be deleted, step S806 can be executed. If the first storage location does not contain hash fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the index structure does not store the hash fingerprint of the data to be deleted, an exception can be returned.

[0159] Step S806: If the first storage location contains fingerprint data that is the same as the hash fingerprint of the data to be deleted, determine whether the hash fingerprint of the data to be deleted is duplicate data based on the records in the first storage location.

[0160] If the first storage location contains hash fingerprint data identical to the hash fingerprint of the data to be deleted, it is necessary to further determine whether the hash fingerprint of the data to be deleted is duplicate data (equivalent to determining whether the data to be deleted is duplicate data) in order to perform different data deletion operations based on the determination result. Since the first storage location is used to store the hash fingerprint of the data to be deleted and record whether the hash fingerprint of the data to be deleted is duplicate data, the determination of whether the hash fingerprint of the data to be deleted is duplicate data can be made based on the records in the first storage location.

[0161] In some implementations, the first storage location includes data bits and a flag bit. The data bits store the hash fingerprint of the data to be deleted, and the flag bit records whether the hash fingerprint of the data to be deleted is duplicate data. Thus, when determining whether the hash fingerprint of the data to be deleted is duplicate data, the flag bit can be used for judgment. For example, if the flag bit value of 0 indicates duplicate data and 1 indicates non-duplicate data, then if the flag bit in the first storage location is 0, it means that the hash fingerprint of the data to be deleted is not duplicate data; if the flag bit in the first storage location is 1, it means that the hash fingerprint of the data to be deleted is duplicate data.

[0162] If it is determined that the hash fingerprint of the data to be deleted is not duplicate data, step S808 can be executed; if it is determined that the hash fingerprint of the data to be deleted is duplicate data, step S810 can be executed.

[0163] Step S808: If the hash fingerprint of the data to be deleted is not duplicate data, delete the hash fingerprint of the data to be deleted in the first storage location.

[0164] Step S810: If the hash fingerprint of the data to be deleted is duplicate data, determine the second storage location in the one-dimensional fingerprint vector according to the third hash function, and decrement the number of duplicates of the hash fingerprint of the data to be deleted recorded in the second storage location by one.

[0165] If the hash fingerprint of the data to be deleted is not a duplicate, the hash fingerprint of the data to be deleted stored in the first storage location can be deleted directly. If the hash fingerprint of the data to be deleted is a duplicate, the deletion operation can be achieved by modifying the number of repetitions of the hash fingerprint of the data to be deleted recorded in the index structure, without having to delete the hash fingerprint of the data to be deleted stored in the index structure.

[0166] When performing a deletion operation by modifying the repetition count recorded in the index structure, specifically, the second storage location in the one-dimensional fingerprint vector can be determined based on the third hash function in the index structure, and the repetition count of the hash fingerprint of the data to be deleted recorded in the second storage location can be decremented by one. For example, if the second storage location includes a counter bit used to record the repetition count of the hash fingerprint of the data to be deleted, then when deleting the data to be deleted, the value of the counter bit in the second storage location can be decremented by one.

[0167] In some implementations, after decrementing the number of repetitions recorded in the second storage location by one, the following operations can be performed:

[0168] Determine if the number of repetitions is 1;

[0169] If the number of repetitions is 1, determine whether the deletion condition is met;

[0170] If the deletion conditions are met, delete the content stored in the second storage location and record in the first storage location that the hash fingerprint of the data to be deleted is not duplicate data.

[0171] After decrementing the count of duplicates recorded in the second storage location, the value of this count can be one of two things: greater than 1 or equal to 1. If the count is greater than 1, it means that the data to be deleted is still a duplicate after deletion, and there's no need to modify the relevant records in the index structure related to duplicate data. If the count is equal to 1, it means that the data to be deleted is no longer a duplicate after deletion, and the relevant records in the index structure related to duplicate data need to be modified to avoid errors. Therefore, after decrementing the count of duplicates recorded in the second storage location, it's necessary to check if the count becomes 1. If the count is not 1, there's no need to modify the relevant records in the index structure related to duplicate data. If the count is 1, then the relevant records in the index structure related to duplicate data need to be modified.

[0172] When modifying records in the index structure where the data to be deleted is duplicated, the content stored in the second storage location can be deleted, and the hash fingerprint of the data to be deleted can be recorded in the first storage location as not being duplicated. The content in the second storage location can be the number of repetitions (for cases where the second hash component includes the third hash function but excludes the fourth hash function), or the fingerprint identifier of the number of repetitions and the hash fingerprint of the data to be deleted (for cases where the second hash component includes both the third and fourth hash functions). When recording that the hash fingerprint of the data to be deleted is not duplicated in the first storage location, for example, the value of a flag bit in the first storage location can be changed from 1 to 0, where 0 indicates that the hash fingerprint of the data to be deleted is not duplicated.

[0173] In some implementations, considering that the data is dynamically updated, to avoid the impact of frequent deletion and insertion of data in the one-dimensional fingerprint vector on overall performance, when the repetition count is 1, the content stored in the second storage location may not be deleted immediately. Instead, it may be determined whether the deletion condition is met. If the deletion condition is not met, the content stored in the second storage location may not be deleted; only if the deletion condition is met will the content stored in the second storage location be deleted. This avoids the performance degradation caused by frequent deletion of data in the one-dimensional fingerprint vector. The deletion condition can be determined according to actual needs and is not specifically limited here. For example, the deletion condition could be a fixed time point, allowing for the periodic deletion of data corresponding to a repetition count of 1 in the one-dimensional fingerprint vector.

[0174] If the deletion conditions are met, after deleting the content stored in the second storage location, it can be further confirmed in the first storage location that the hash fingerprint of the data to be deleted is not duplicate data. In this way, since modifications to the records of duplicate data in the first storage location are also performed under the condition that the deletion conditions are met, frequent data modifications in the two-dimensional fingerprint matrix can be avoided, thus preventing the impact on overall performance.

[0175] To facilitate understanding of the method for deleting data in an index structure provided in the embodiments of this application, the following will use... Figure 7 and Figure 9 The following is an example of a more specific implementation method.

[0176] based on Figure 7 The index structure shown illustrates this. Suppose we need to delete a data item with a date of 20230801 and a data item with a date of 20240301 from data table T1. Then we need to... Figure 7 The index structure shown is updated. Specifically, as... Figure 9 As shown, when deleting a data item with data 20230801, the hash fingerprint 106 can be calculated based on the fingerprint hash function, and then the row index in the two-dimensional fingerprint matrix can be calculated by two mapping hash functions respectively, resulting in buckets [1] and [2]. It is found that there is a data fingerprint with the same data as 106 in bucket [1], and its flag bit is 1. Therefore, it needs to be processed in the second hash component. Since the quotient and remainder of hash fingerprint 106 relative to 4 are 26 and 2 respectively, it can be determined that the quotient and repetition count of hash fingerprint 106 are stored in slot [2] in the one-dimensional fingerprint vector, and the count bit in slot [2] is decremented by 1, that is, set to 2. When deleting a data item with data 20240301, the hash fingerprint 108 can be calculated based on the fingerprint hash function, and then the row index in the two-dimensional fingerprint matrix can be calculated by two mapping hash functions respectively, resulting in buckets [1] and [2]. It is found that there is a data fingerprint with the same data as 1 in bucket [1], and its flag bit is 0. Therefore, the data can be directly deleted in the first hash component.

[0177] Assuming that a data item with date 20230801 needs to be deleted later, based on the method described above, the count bit in the slot [2] of the one-dimensional fingerprint vector can be reduced by 1, that is, set to 1. And if the deletion condition is met, the quotient 26 and the count bit 1 in the slot [2] can be deleted, that is, set to Null and 0 respectively. At the same time, the flag bit corresponding to the hash fingerprint 106 in the two-dimensional fingerprint matrix can be set to 0.

[0178] As can be seen, when deleting data in the index structure, the positions of the two hash buckets are first calculated using two mapped hash functions. Then, it checks if there is any identical fingerprint data in the two-dimensional fingerprint matrix. If not, an exception is returned. If identical fingerprint data exists and the flag bit is 0, the corresponding fingerprint data is directly deleted. If identical fingerprint data exists and the flag bit is 1, the location of the data is found in the one-dimensional fingerprint vector, and the count bit is decremented by 1. To avoid frequent deletion and insertion of data in the one-dimensional fingerprint vector affecting overall performance, the corresponding data is not deleted immediately when the count bit in the one-dimensional fingerprint vector is 1. Instead, the corresponding data with a count bit of 0 is deleted only when the deletion condition is met, and the flag bit of the corresponding data in the two-dimensional fingerprint matrix is ​​set to 0.

[0179] Based on the method for deleting data in an index structure provided in this application embodiment, since the first storage location in the index structure is used to record whether the hash fingerprint of the data is duplicate data, and the second hash component in the index structure can determine the second storage location based on the hash fingerprint of the data and record the number of times the data is repeated in the second storage location, when deleting data in the index structure, it is possible to determine whether the data to be processed is duplicate data based on the record in the first storage location, and to determine the number of times the data to be processed is repeated based on the record in the second storage location of the second hash component. Thus, by modifying the record in the first storage location indicating whether the data to be processed is duplicate data and / or the record in the second storage location indicating the number of times the data to be processed is repeated, the data to be processed can be quickly deleted. This method can support the deletion of a large amount of duplicate data while ensuring high performance, and solves the problem of index unavailability caused by data updates and duplicate data deletion in streaming scenarios.

[0180] Figure 10 This is a flowchart illustrating a data query method based on an index structure according to an embodiment of this application. The index structure includes the index structure provided in the embodiments of this application, which can be found in the following examples. Figures 2 to 4 The embodiments shown will not be described again here. Figure 10 The data query method shown can be derived from Figure 1 The big data execution engine in the computing layer 11 shown is executed. Figure 10 The method shown includes the following steps.

[0181] Step S102: Receive query request.

[0182] When external clients or other systems have data query needs, they can send query requests to the big data execution engine. The big data query engine can then receive these requests. The query request can include SQL statements, which in turn include query filtering conditions. These filtering conditions can be equality filters or other types of filters; no specific limitations are specified here.

[0183] Step S104: Obtain multiple index files corresponding to multiple data files, with one index file corresponding to each data file, and each index file including an index structure.

[0184] Multiple data files are the objects to be queried; that is, when performing a data query, the desired target data is retrieved from multiple data files. In this embodiment, for each of the multiple data files, an index structure as provided in this application embodiment can be pre-constructed based on the data in the data file, and the index structure is stored in an index file. When performing a data query, the index file corresponding to the multiple data files can be retrieved. Each data file can correspond to one index file, and the index file includes the index structure.

[0185] When constructing the index structure provided in this application embodiment, for each data file, the corresponding index file can be determined in the following way:

[0186] Retrieve data from the data file and initialize the index structure;

[0187] The index structure is updated based on the data in the data file. Updating the index structure includes inserting or deleting data in the index structure.

[0188] The index structure is stored according to the specified format to obtain the index file corresponding to the data file.

[0189] Data in the data file can be stored in a data lake or data warehouse. When retrieving data from the data file, it can be obtained from the corresponding data lake or data warehouse. The data types in the data file include, but are not limited to, integer, floating-point, character, and date types. The index structure is the one provided in this application embodiment, which includes a first hash component and a second hash component; details can be found in [link to relevant documentation]. Figures 2 to 4 The illustrated embodiment will not be described again here. After initializing the index structure, an empty index structure can be obtained (equivalent to an empty filter, which does not store any data).

[0190] When updating the index structure based on data in the data file, considering that the data in the data file is usually dynamically changing, updating the index structure here can include inserting or deleting data in the index structure (corresponding to inserting or deleting data in the data file). For details on inserting data into the index structure, please refer to [link to relevant documentation]. Figure 5 The embodiment shown illustrates how data can be deleted from the index structure. For details, please refer to [link to relevant documentation]. Figure 8The embodiments shown will not be described in detail here. After updating the index structure, a usable index structure can be obtained.

[0191] After obtaining a usable index structure, the index structure can be formatted in the computing engine. This involves converting the first and second hash components of the index structure into a specified index format, and then writing the converted index data to the storage layer. Figure 1 In the storage layer 12 shown, the index data can be stored in Parquet, Avro, or ORC formats.

[0192] Step S106: Determine the valid file from multiple data files based on the query request and multiple index files.

[0193] After obtaining multiple index files corresponding to multiple data files, invalid files can be filtered out from the multiple data files based on the query request and the multiple index files, leaving only valid files. The valid files store the target data to be queried, and there can be one or more valid files.

[0194] In some implementations, determining a valid file from multiple data files based on a query request and multiple index files may include the following steps:

[0195] The query request is parsed and verified to determine the logical plan corresponding to the query request;

[0196] The logical plan is transformed into a physical execution plan, which includes query filtering conditions.

[0197] The query filters multiple index files, and the valid file is determined from multiple data files based on the query results.

[0198] The query request includes SQL query statements. During the parsing and analysis of the query request, an SQL parser can be used to perform lexical analysis on the SQL statements, obtaining a general symbolic stream. Then, syntax analysis is performed on the symbolic stream to construct a syntax tree. Afterwards, the SQL parser can be used to perform in-depth analysis on any unparsed relations in the syntax tree, generating a resolved logical plan. The specific implementation methods for using an SQL parser for lexical analysis and an SQL parser for syntax analysis can be found in related technical documentation and will not be detailed here.

[0199] After obtaining the logical plan, the index structure in the index file can be used as an index to optimize the logical plan and generate a physical execution plan optimized with the index. The physical execution plan includes query filtering conditions. Taking equality filtering conditions as an example, when generating the physical execution plan, the generated logical plan tree can be traversed, and several equality filtering conditions such as Filter in the logical plan can be replaced with methods that use the index for query judgment, generating and executing a physical execution plan with index optimization.

[0200] After obtaining the physical execution plan, multiple index files can be queried based on the query filters in the physical execution plan, and valid files can be determined from multiple data files based on the query results. In some implementations, when querying multiple index files based on query filters and determining valid files from multiple data files based on the query results, the following operations can be performed for each index file:

[0201] The hash fingerprint of the query filtering conditions is determined based on the first hash function in the index structure of the index file, and the first storage location in the two-dimensional fingerprint matrix is ​​determined based on the second hash function in the index structure.

[0202] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint of the query filter condition;

[0203] If the first storage location contains fingerprint data that is the same as the hash fingerprint of the query filter condition, the data file corresponding to the index file is determined to be a valid file;

[0204] If the first storage location does not store fingerprint data that is the same as the hash fingerprint of the query filter condition, the data file corresponding to the index file will be determined as an invalid file.

[0205] The index file includes an index structure. When querying data based on query filtering conditions, the hash fingerprint of the query filtering conditions (equivalent to the hash fingerprint of the target data) can be determined based on the first hash function in the index structure. In some implementations, the first hash function can be represented as Hash(). When determining the hash fingerprint of the query filtering conditions based on the first hash function, it can be determined using the following formula:

[0206] F p =Hash(x).

[0207] After obtaining the hash fingerprint of the query filtering conditions, the first storage location in the two-dimensional fingerprint matrix can be determined according to the second hash function in the index structure.

[0208] In some implementations, the second hash function may include multiple mapping hash functions, each mapping hash function being used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions being used to determine multiple row indices in the two-dimensional fingerprint matrix. Thus, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indices in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically the storage location in the row corresponding to the multiple row indices in the two-dimensional fingerprint matrix.

[0209] Taking the second hash function, which includes two hash mapping functions B1() and B2(), as an example, the first storage location of the two-dimensional fingerprint matrix can be determined using the following formula:

[0210] B1(x)=F p modm;

[0211] B2(x)=B1(x)⊕Hash(F p ).

[0212] Where m is the number of rows in the two-dimensional fingerprint matrix, F p This is the hash fingerprint of the data to be inserted.

[0213] Based on the two hash mapping functions B1() and B2() mentioned above, we can obtain two row indices of the two-dimensional fingerprint matrix. These two row indices point to two rows in the two-dimensional fingerprint matrix. The first storage location is the storage location in these two rows, indicating that if the index structure stores the hash fingerprint of the target data, then the hash fingerprint is stored in these two rows.

[0214] After determining the hash fingerprint of the query filter condition and the first storage location in the two-dimensional fingerprint matrix, it can be determined whether the hash fingerprint of the query filter condition is stored in the index structure. Specifically, it can be determined whether the first storage location stores fingerprint data identical to the hash fingerprint of the query filter condition. If the first storage location stores fingerprint data identical to the hash fingerprint of the query filter condition, it can be determined that the hash fingerprint of the query filter condition is stored in the index structure. Correspondingly, the target data is stored in the data file corresponding to the index file, and the data file corresponding to the index file can be identified as a valid file. If the first storage location does not store fingerprint data identical to the hash fingerprint of the query filter condition, it can be determined that the hash fingerprint of the query filter condition is not stored in the index structure. Correspondingly, the target data is not stored in the data file corresponding to the index file, and the data file corresponding to the index file can be identified as an invalid file. Invalid files can be skipped during data queries.

[0215] After performing the above operations on multiple index files, valid files can finally be filtered out from multiple data files.

[0216] Step S108: Query the target data corresponding to the query request from the valid file.

[0217] After identifying the valid files from multiple data files, the valid files can be scanned during data querying, and the target data can be retrieved from the valid files.

[0218] To facilitate understanding of how to perform data queries based on the index structure provided in the embodiments of this application, a more specific implementation method will be described below. Please refer to... Figure 11 .

[0219] exist Figure 11 In the illustrated embodiment, when performing a data query, it is necessary to filter multiple data files to be queried in order to obtain valid files. Specifically, this may include the following steps:

[0220] Step 1: Parse and validate the SQL query statement in the query request, and convert it into a logical plan.

[0221] Step 2: Obtain the filtering conditions in the logical plan, optimize the logical plan based on cost, and obtain the optimized logical plan.

[0222] Step 3: Generate and execute the physical plan based on the optimized logical plan.

[0223] Step 4: Filter multiple data files based on the index files corresponding to the data files.

[0224] When filtering multiple data files, the query filter conditions can be obtained first. The hash fingerprint of the query filter conditions is determined using the fingerprint hash function in the index structure. Then, the storage location in the two-dimensional fingerprint matrix is ​​determined using the mapping hash function in the index structure. It is then checked whether the storage location contains fingerprint data with the same hash fingerprint as the query filter conditions. If the storage location contains the same fingerprint data, the corresponding data file is determined as a valid file; otherwise, the corresponding data file is determined as an invalid file and skipped.

[0225] Step 5: Generate an executable physical plan.

[0226] Step 6: Execute the physical plan and query the target data in the valid files.

[0227] Step 7: Return the results.

[0228] The specific implementation methods of steps 1 to 7 above can be found in the specific implementation methods of the corresponding steps in steps S102 to S108 above, and will not be repeated here.

[0229] Next, a simple example will be used to illustrate the data query method provided in the embodiments of this application.

[0230] Assumption Figure 9 The index shown corresponds to a portion of the data in table T1. This data is stored in the storage layer as a Parquet file named T1-data-001.parquet, and T1 consists of multiple Parquet data files. To optimize queries using the index structure proposed in this embodiment, an index must first be generated based on the data in the data files and written to the storage layer. The index will construct corresponding index files for each data file. For the data file T1-data-001.parquet, an index will be generated and stored in the storage layer, assuming its corresponding index file name is T1-idx-001.parquet.

[0231] When performing data queries, assuming the user enters the SQL command "select * from T1 where T1.date = 20230801", which means querying all data items in table T1 whose date data is equal to 20230801, then the logical plan can be optimized by using the index written to the storage layer, and when the execution of the physical plan requires scanning the data file corresponding to table T1, the existing index can be used to achieve a fast judgment.

[0232] Specifically, before scanning the data file T1-data-001.parquet, the index file T1-idx-001.parquet in the storage layer is read first. Then, the data fingerprint corresponding to the data 20230801 is calculated using the fingerprint hash function, which is 106. Then, the buckets corresponding to the data fingerprint 106 are obtained through two mapping hash functions, namely bucket [1] and bucket [2]. The same fingerprint data is found in bucket [1], that is, the corresponding data file has related data items. Therefore, the data file T1-data-001.parquet needs to be scanned when the physical plan is executed.

[0233] Suppose that a user wants to query all data items in data table T1 whose data is equal to 20240309, and assume that the data fingerprint of 20240309 calculated using a hash function is 66, and that the data fingerprint is obtained by two mapping hash functions to the corresponding buckets [1] and [2]. However, there is no identical fingerprint data in buckets [1] and [2]. That is, the index structure corresponding to the index file T1-idx-001.parquet does not contain data items whose data is equal to 20240309. Therefore, the data file T1-data-001.parquet can be skipped directly.

[0234] Based on the data query method provided in this application embodiment, when performing data query, the existence of target data in a data file can be checked according to the index structure provided in this application embodiment. This allows invalid files to be filtered out, valid files to be obtained, and the target data to be queried from the valid files. This narrows the data scanning range and improves data query efficiency. Furthermore, in the index structure provided in this application embodiment, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint based on the hash fingerprint, when performing data query based on the index structure, the first storage location can be determined based on the first hash component, and data query can be performed at the first storage location. This narrows the query range in the index structure, enabling fast data query. When querying data from multiple data files, efficient data skipping is supported, allowing invalid files to be quickly filtered out and valid files to be obtained, further improving query efficiency.

[0235] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0236] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 12 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0237] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0238] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0239] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a logical mechanism for inserting data into an index structure. The processor executes the program stored in memory and specifically performs the following operations:

[0240] The hash fingerprint of the data to be inserted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0241] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0242] If the first storage location contains fingerprint data identical to the hash fingerprint, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions of the hash fingerprint is recorded in the second storage location.

[0243] If no fingerprint data identical to the hash fingerprint is stored in the first storage location, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0244] The above is as stated in this application. Figure 12The method for inserting data into an index structure as disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0245] The electronic device can also perform Figure 5 The method, and the apparatus for deleting data in the index structure Figure 5 The functions described in the illustrated embodiments will not be repeated here.

[0246] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0247] This application also discloses a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 5 The method of the illustrated embodiment is specifically used to perform the following operations:

[0248] The hash fingerprint of the data to be inserted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0249] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0250] If the first storage location contains fingerprint data identical to the hash fingerprint, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions of the hash fingerprint is recorded in the second storage location.

[0251] If no fingerprint data identical to the hash fingerprint is stored in the first storage location, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0252] Figure 13 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 13 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0253] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0254] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0255] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a mechanism at the logical level for deleting data from an index structure. The processor executes the program stored in memory and specifically performs the following operations:

[0256] The hash fingerprint of the data to be deleted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0257] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0258] If the first storage location contains fingerprint data identical to the hash fingerprint, determine whether the hash fingerprint is duplicate data based on the records in the first storage location.

[0259] If the hash fingerprint is not duplicate data, delete the hash fingerprint from the first storage location;

[0260] In the case that the hash fingerprint is duplicate data, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of duplicates recorded in the second storage location is decremented by one.

[0261] The above is as stated in this application. Figure 13The method for deleting data in an index structure disclosed in the illustrated embodiment can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0262] The electronic device can also perform Figure 8 The method, and the apparatus for deleting data in the index structure Figure 8 The functions described in the illustrated embodiments will not be repeated here.

[0263] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0264] This application also discloses a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 8 The method of the illustrated embodiment is specifically used to perform the following operations:

[0265] The hash fingerprint of the data to be deleted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function.

[0266] Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0267] If the first storage location contains fingerprint data identical to the hash fingerprint, determine whether the hash fingerprint is duplicate data based on the records in the first storage location.

[0268] If the hash fingerprint is not duplicate data, delete the hash fingerprint from the first storage location;

[0269] In the case that the hash fingerprint is duplicate data, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of duplicates recorded in the second storage location is decremented by one.

[0270] Figure 14 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 14 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0271] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0272] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0273] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a data lookup device based on an index structure at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0274] Receive query requests;

[0275] Obtain multiple index files corresponding to multiple data files, with one index file corresponding to each data file, and each index file including the index structure;

[0276] Based on the query request and the multiple index files, a valid file is determined from the multiple data files;

[0277] Retrieve the target data corresponding to the query request from the valid file.

[0278] The above is as stated in this application. Figure 14 The method executed by the index-based data query device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0279] The electronic device can also perform Figure 10 and Figure 11The method, and implements a data query device based on an index structure in Figure 10 and Figure 11 The functions described in the illustrated embodiments will not be repeated here.

[0280] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0281] This application also discloses a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 10 and Figure 11 The method of the illustrated embodiment is specifically used to perform the following operations:

[0282] Receive query requests;

[0283] Obtain multiple index files corresponding to multiple data files, with one index file corresponding to each data file, and each index file including the index structure;

[0284] Based on the query request and the multiple index files, a valid file is determined from the multiple data files;

[0285] Retrieve the target data corresponding to the query request from the valid file.

[0286] Figure 15 This is a schematic diagram of the structure of a device 150 for inserting data into an index structure, according to an embodiment of this application. Please refer to... Figure 15 In one software implementation, the device 150 for inserting data into the index structure may include: a first determining module 151, a second determining module 152, a first inserting module 153, and a second inserting module 154, wherein:

[0287] The first determining module 151 determines the hash fingerprint of the data to be inserted according to the first hash function and determines the first storage location in the two-dimensional fingerprint matrix according to the second hash function.

[0288] The second determining module 152 determines whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0289] The first insertion module 153, when the first storage location stores fingerprint data that is the same as the hash fingerprint, determines the second storage location in the one-dimensional fingerprint vector according to the third hash function, and records the number of repetitions of the hash fingerprint in the second storage location;

[0290] The second insertion module 154, when no fingerprint data identical to the hash fingerprint is stored in the first storage location, stores the hash fingerprint in the first storage location and records that the hash fingerprint is not duplicate data.

[0291] The apparatus 150 for inserting data into an index structure provided in this application can also perform... Figure 5 The method, and the apparatus 150 that implements the insertion of data into the index structure. Figure 5 The functions of the embodiments shown will not be described again in this application.

[0292] Figure 16 This is a schematic diagram of a device 160 for deleting data in an index structure, according to an embodiment of this application. Please refer to... Figure 16 In one software implementation, the device 160 for deleting data in the index structure may include: a first determining module 161, a second determining module 162, a third determining module 163, a first inserting module 164, and a second inserting module 165, wherein:

[0293] The first determining module 161 determines the hash fingerprint of the data to be deleted according to the first hash function and determines the first storage location in the two-dimensional fingerprint matrix according to the second hash function.

[0294] The second determining module 162 determines whether the first storage location stores fingerprint data that is the same as the hash fingerprint;

[0295] The third determining module 163 determines whether the hash fingerprint is duplicate data based on the records in the first storage location when the first storage location stores fingerprint data that is the same as the hash fingerprint.

[0296] The first insertion module 164 deletes the hash fingerprint from the first storage location if the hash fingerprint is not duplicate data.

[0297] The second insertion module 165, when the hash fingerprint is duplicate data, determines the second storage location in the one-dimensional fingerprint vector according to the third hash function, and decrements the number of duplicates recorded in the second storage location by one.

[0298] The apparatus 160 for deleting data in an index structure provided in this application can also perform... Figure 8The method, and the apparatus 160 for deleting data in the index structure. Figure 8 The functions of the embodiments shown will not be described again in this application.

[0299] Figure 17 This is a schematic diagram of the structure of a data query device 170 based on an index structure, according to an embodiment of this application. Please refer to... Figure 17 In one software implementation, the index-based data query device 170 may include: a receiving module 171, an acquiring module 172, a determining module 173, and a query module 174, wherein:

[0300] Receiver module 171 receives query requests;

[0301] The acquisition module 172 acquires multiple index files corresponding to multiple data files, with each data file corresponding to one index file, and each index file including the index structure;

[0302] The determination module 173 determines a valid file from the multiple data files based on the query request and the multiple index files;

[0303] The query module 174 queries the target data corresponding to the query request from the valid file.

[0304] The index-structure-based data query device 170 provided in this application can also perform... Figure 10 and Figure 11 The method, and implements the data query device 170 based on the index structure in Figure 10 and Figure 11 The functions of the embodiments shown will not be described again in this application.

[0305] This application also proposes a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the above-described method embodiment for inserting data in an index structure, or to perform some or all of the steps in the above-described method embodiment for deleting data in an index structure, or to perform some or all of the steps in the above-described method embodiment for querying data based on an index structure.

[0306] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0307] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0308] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0309] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0310] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A data query method based on an index structure, wherein the index structure is used to process data in a database, the data in the database including duplicate data and non-duplicate data, the index structure includes a first hash component and a second hash component, the first hash component includes a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used to determine the hash fingerprint of the data, the second hash function is used to determine a first storage location in the two-dimensional fingerprint matrix based on the hash fingerprint, the first storage location is used to store the hash fingerprint and record whether the hash fingerprint is duplicate data, the second hash component includes a third hash function and a one-dimensional fingerprint vector, the third hash function is used to determine a second storage location in the one-dimensional fingerprint vector based on the hash fingerprint, the second storage location is used to record the number of times the hash fingerprint is repeated; The method includes: Receive query requests; Obtain multiple index files corresponding to multiple data files, with one index file corresponding to each data file, and each index file including the index structure; Based on the query request and the multiple index files, a valid file is determined from the multiple data files; Retrieve the target data corresponding to the query request from the valid file.

2. The method as described in claim 1, wherein the second hash function is used to determine multiple row indices of the two-dimensional fingerprint matrix, and the first storage location is the storage location in the row corresponding to the multiple row indices.

3. The method of claim 2, wherein the second hash function comprises a first mapping hash function and a second mapping hash function, the first mapping hash function being used to determine a first row index based on the hash fingerprint and the number of rows in the two-dimensional fingerprint matrix, and the second mapping hash function being used to determine a second row index based on the hash fingerprint and the first row index.

4. The method as described in claim 1, wherein the first storage location comprises data bits and flag bits, the data bits being used to store the hash fingerprint, and the flag bits being used to record whether the hash fingerprint is duplicate data; in, A value of 0 for the flag bit indicates that the hash fingerprint is not duplicate data, while a value of 1 for the flag bit indicates that the hash fingerprint is duplicate data.

5. The method as described in claim 1, wherein the third hash function includes a modulo function; The modulo function is used to determine the remainder when the hash fingerprint is divided by the length of the one-dimensional fingerprint vector, and the remainder is used to indicate the second storage location.

6. The method of claim 1, wherein the second hash component further comprises a fourth hash function; The fourth hash function is used to determine the fingerprint identifier of the hash fingerprint, and the second storage location is also used to store the fingerprint identifier.

7. The method of claim 6, wherein the fourth hash function comprises a quotient function; The quotient function is used to determine the quotient of the length of the hash fingerprint and the length of the one-dimensional fingerprint vector, and the fingerprint identifier includes the quotient.

8. The method of claim 6, wherein the second storage location comprises data bits and counter bits, the data bits being used to store the fingerprint identifier, and the counter bits being used to record the number of repetitions of the hash fingerprint.

9. The method according to any one of claims 1 to 8, wherein the data type includes, but is not limited to, integer, floating-point, character, and date types.

10. The method of claim 1, wherein determining a valid file from the plurality of data files based on the query request and the plurality of index files comprises: The query request is parsed and verified to determine the logical plan corresponding to the query request; The logical plan is transformed into a physical execution plan, which includes query filtering conditions; The query filters the multiple index files, and the valid files are determined from the multiple data files based on the query results.

11. The method of claim 10, wherein querying the plurality of index files according to the query filtering conditions and determining a valid file from the plurality of data files based on the query results includes: For each index file, perform the following operations: The hash fingerprint of the query filtering condition is determined according to the first hash function in the index structure, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function in the index structure. Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint; If the first storage location contains fingerprint data identical to the hash fingerprint, the data file corresponding to the index file is determined to be a valid file; If no fingerprint data identical to the hash fingerprint is stored in the first storage location, the data file corresponding to the index file is determined to be an invalid file.

12. The method of any one of claims 1, 10, and 11, wherein for each data file, the index file corresponding to the data file is determined in the following manner: Obtain the data from the data file and initialize the index structure; The index structure is updated based on the data in the data file, and the update includes inserting or deleting data in the index structure; The index structure is stored according to a specified format to obtain an index file corresponding to the data file.

13. A method for inserting data into an index structure, said index structure comprising the index structure according to any one of claims 1 to 9, the method comprising: The hash fingerprint of the data to be inserted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function. Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint; If the first storage location contains fingerprint data identical to the hash fingerprint, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions of the hash fingerprint is recorded in the second storage location. If no fingerprint data identical to the hash fingerprint is stored in the first storage location, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

14. The method of claim 13, wherein recording the number of repetitions of the hash fingerprint at the second storage location comprises: Determine whether the hash fingerprint has been recorded as duplicate data in the first storage location; If the hash fingerprint is not recorded as duplicate data in the first storage location, the number of times the hash fingerprint is recorded as duplicate data in the first storage location and the number of times the hash fingerprint is recorded as duplicate data in the second storage location is 2. If the hash fingerprint has been recorded as duplicate data in the first storage location, the number of duplicates recorded in the second storage location is incremented by one.

15. The method of claim 14, wherein the second hash component further comprises a fourth hash function; and when the number of repetitions of the hash fingerprint recorded at the second storage location is 2, the method further comprises: The fingerprint identifier of the hash fingerprint is determined according to the fourth hash function; The fingerprint identifier is stored in the second storage location.

16. The method of claim 13, wherein the second hash function is used to determine multiple row indices of the two-dimensional fingerprint matrix, and the first storage location is the storage location in the row corresponding to the multiple row indices; Storing the hash fingerprint in the first storage location and recording that the hash fingerprint is not duplicate data includes: Determine the row with the lowest load from multiple rows corresponding to multiple row indexes; The hash fingerprint is stored in the row with the least load, and it is recorded that the hash fingerprint is not duplicate data.

17. A method for deleting data in an index structure, said index structure comprising the index structure according to any one of claims 1 to 9, the method comprising: The hash fingerprint of the data to be deleted is determined according to the first hash function, and the first storage location in the two-dimensional fingerprint matrix is ​​determined according to the second hash function. Determine whether the first storage location stores fingerprint data that is the same as the hash fingerprint; If the first storage location contains fingerprint data identical to the hash fingerprint, determine whether the hash fingerprint is duplicate data based on the records in the first storage location. If the hash fingerprint is not duplicate data, delete the hash fingerprint from the first storage location; In the case that the hash fingerprint is duplicate data, the second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of duplicates recorded in the second storage location is decremented by one.

18. The method of claim 17, further comprising, after decrementing the number of repetitions recorded in the second storage location by one: Determine whether the number of repetitions is 1; If the number of repetitions is 1, determine whether the deletion condition is met; If the deletion condition is met, delete the content stored in the second storage location and record in the first storage location that the hash fingerprint is not duplicate data.

19. An electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 18.

20. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 18.

21. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the method as claimed in any one of claims 1 to 18.

Images