Index structure, data insertion and deletion method and data query method

Abstract

The invention provides an index structure, a data insertion and deletion method and a data query method. The index structure comprises a first hash component and a second hash component, the first hash component comprises a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used for determining a hash fingerprint of data, and the second hash function is used for determining a first storage position in the two-dimensional fingerprint matrix according to the hash fingerprint, the first storage position is used for storing Hash fingerprints and recording whether the Hash fingerprints are duplicated data or not; the second hash component comprises a third hash function and a one-dimensional fingerprint vector, the third hash function is used for determining a second storage position in the one-dimensional fingerprint vector according to the hash fingerprint, and the second storage position is used for recording the number of repetition times of the hash fingerprint. The index structure provided by the embodiment of the invention can support query, insertion and deletion operations on a large amount of repeated data on the premise of ensuring high performance.

Description

Technical Field

[0001] The present application relates to the field of index technology, and in particular to an index structure, a method for inserting and deleting data, and a data query method. Background Art

[0002] When querying data in a database, filters (such as Bloom filters, cuckoo filters, quotient filters, etc.) are usually used to query data to improve query efficiency. In some scenarios, there will be duplicate data in the database, especially when the data is distributed according to a power law, some data will be repeated in large quantities. In this case, when using filters to query data, the filters are required to support fast query of large amounts of duplicate data. In addition, in order to achieve dynamic updates of large amounts of duplicate data, the filters are also required to support fast insertion and deletion of large amounts of duplicate data. However, the filters currently used have certain limitations and cannot guarantee efficient query, efficient insertion, or efficient deletion of large amounts of duplicate data. Summary of the invention

[0003] The present application provides an index structure, a method for inserting and deleting data, and a data query method, which are used to solve the problem that the currently used filters have certain limitations and cannot guarantee efficient query, efficient insertion or efficient deletion operations on a large amount of duplicate data.

[0004] To solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an index structure is provided, comprising a first hash component and a second hash component, wherein:

[0006] The first hash component includes a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used to determine a hash fingerprint of data, the second hash function is used to determine a first storage location in the two-dimensional fingerprint matrix according to the hash fingerprint, and the first storage location is used to store the hash fingerprint and record whether the hash fingerprint is duplicate data;

[0007] The second hash component includes a third hash function and a one-dimensional fingerprint vector, the third hash function is used to determine a second storage location in the one-dimensional fingerprint vector according to the hash fingerprint, and the second storage location is used to record the number of repetitions of the hash fingerprint.

[0008] In a second aspect, a method for inserting data into an index structure is provided, wherein the index structure includes the index structure as described in the first aspect above, and the method includes:

[0009] Determine a hash fingerprint of the data to be inserted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0010] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0011] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining a second storage location in the one-dimensional fingerprint vector according to the third hash function, and recording the number of repetitions of the hash fingerprint in the second storage location;

[0012] In a case where the first storage location does not store fingerprint data identical to the hash fingerprint, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0013] In a third aspect, a method for deleting data in an index structure is provided, wherein the index structure includes the index structure as described in the first aspect above, and the method includes:

[0014] Determine a hash fingerprint of the data to be deleted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0015] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0016] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining whether the hash fingerprint is duplicate data according to the record in the first storage location;

[0017] In a case where the hash fingerprint is not duplicate data, deleting the hash fingerprint in the first storage location;

[0018] In the case that the hash fingerprint is repeated data, a second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions recorded in the second storage location is reduced by one.

[0019] In a fourth aspect, a data query method based on an index structure is provided, wherein the index structure includes the index structure as described in the first aspect above, and the method includes:

[0020] receiving a query request;

[0021] Acquire multiple index files corresponding to multiple data files, each data file corresponds to an index file, and each index file includes the index structure;

[0022] Determining a valid file from the multiple data files according to the query request and the multiple index files;

[0023] The target data corresponding to the query request is searched from the valid file.

[0024] In a fifth aspect, an electronic device is provided, including:

[0025] processor;

[0026] a memory for storing instructions executable by the processor;

[0027] The processor is configured to execute the instructions to implement the method as described in the second aspect, the third aspect, or the fourth aspect.

[0028] In a sixth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the second aspect, the third aspect, or the fourth aspect.

[0029] In the seventh aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute some or all of the steps in the method described in the second aspect, or to execute some or all of the steps in the method described in the third aspect, or to execute some or all of the steps in the method described in the fourth aspect.

[0030] Based on the index structure provided by the embodiment of the present application, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint according to the hash fingerprint, therefore, when performing data query, the first storage location can be determined based on the first hash component, and the data query can be performed at the first storage location, thereby narrowing the query range in the index structure, thereby realizing fast data query; since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location according to the hash fingerprint of the data, and record the number of repetitions of the data at the second storage location, therefore, when deleting or inserting data, it can be determined whether the data to be processed is duplicate data according to the record in the first storage location, and the number of repetitions of the data to be processed can be determined according to the record in the second storage location of the second hash component, thereby realizing fast deletion or insertion of the data to be processed by modifying the record of whether the data to be processed is duplicate data in the first storage location and / or the record of the number of repetitions of the data to be processed in the second storage location. The index structure provided by the embodiment of the present application can support query, insertion and deletion operations on a large amount of duplicate data under the premise of ensuring high performance, which not only solves the problem of index unavailability caused by data update and insertion and deletion of duplicate data in streaming scenarios, but also supports efficient data skipping, further improving query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 It is a schematic diagram of a schematic system architecture provided in an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of an index structure of an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of a two-dimensional fingerprint matrix of an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of a one-dimensional fingerprint vector according to an embodiment of the present application;

[0036] Figure 5 It is a flowchart of a method for inserting data into an index structure according to an embodiment of the present application;

[0037] Figure 6is a schematic diagram of inserting data into an index structure according to an embodiment of the present application;

[0038] Figure 7 is a schematic diagram of inserting data into an index structure according to another embodiment of the present application;

[0039] Figure 8 is a flowchart of a method for deleting data in an index structure according to an embodiment of the present application;

[0040] Fig. 9 is a schematic diagram of deleting data in an index structure according to an embodiment of the present application;

[0041] Fig.10 It is a flowchart of a data query method based on an index structure according to an embodiment of the present application;

[0042] Fig.11 It is a flowchart of a data query method based on an index structure according to another embodiment of the present application;

[0043] Fig.12 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0044] Fig.13 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0045] Fig.14 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0046] Fig.15 is a schematic diagram of a device structure for inserting data into an index structure according to an embodiment of the present application;

[0047] Fig.16 is a schematic diagram of a device structure for deleting data in an index structure according to an embodiment of the present application;

[0048] Fig.17 It is a structural diagram of a data query device based on an index structure in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In practical applications, it is common to have duplicate data in the database, especially when the data is distributed according to the power law, some data will have a large number of duplicates. When a large amount of duplicate data is stored in the database, when using filters to query data, in order to ensure query efficiency, the filter is required to support fast query operations on a large amount of duplicate data. At the same time, in order to achieve dynamic update of duplicate data, the filter is also required to support fast insertion and deletion operations on a large amount of duplicate data. However, the filters currently used have certain limitations and cannot support efficient query, efficient insertion or efficient deletion operations on a large amount of duplicate data.

[0050] Take the commonly used Bloom filter, Cuckoo filter and Quotient filter as examples.

[0051] Bloom filter is a probabilistic data structure that can map elements to multiple bits using multiple hash functions and use bits to determine whether data exists. However, after a bit is set to 1, it cannot be set to 0 at will, which makes Bloom filter unable to support data deletion operations.

[0052] Cuckoo filter is a data structure for quickly finding elements. It can use multiple hash buckets to implement data mapping and storage, and calculate the data storage location based on two hash functions and XOR operations. The fingerprint of the data will be stored in the free slots of the two hash buckets. However, although the cuckoo filter supports data deletion operations, due to its structural characteristics, that is, the same hash value needs to be placed in the same two hash buckets with limited space, it cannot effectively handle a large amount of duplicate data.

[0053] The core idea of ​​the quotient filter is to decompose the hash value of an element into a quotient and a remainder, determine the storage location based on the quotient value, and store the remainder in the storage location. Although the quotient filter can support fast query, insertion, and deletion operations on data, the performance of the quotient filter will be significantly reduced when there is a lot of duplicate data.

[0054] The embodiment of the present application provides an index structure that can support dynamic deletion and insertion of data, especially when processing a large amount of duplicate data, the index structure can be effectively constructed and maintained, while ensuring the high performance of query operations and time and space efficiency. The embodiment of the present application also provides a method for inserting and deleting data in an index structure, and a data query method based on an index structure, which can support query, insertion and deletion operations on a large amount of duplicate data under the premise of ensuring high performance, which not only solves the problem of index unavailability caused by data update and insertion and deletion of duplicate data in a streaming scenario, but also supports efficient data skipping, further improving query efficiency. Among them, the index structure includes a first hash component and a second hash component, the first hash component includes a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used to determine the hash fingerprint of the data, the second hash function is used to determine the first storage position in the two-dimensional fingerprint matrix according to the hash fingerprint, the first storage position is used to store the hash fingerprint and record whether the hash fingerprint is duplicate data; the second hash component includes a third hash function and a one-dimensional fingerprint vector, the third hash function is used to determine the second storage position in the one-dimensional fingerprint vector according to the hash fingerprint, and the second storage position is used to record the number of repetitions of the hash fingerprint.

[0055] Based on the index structure provided by the embodiment of the present application, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint according to the hash fingerprint, therefore, when performing data query, the first storage location can be determined based on the first hash component, and the data query can be performed at the first storage location, thereby narrowing the query range in the index structure, thereby realizing fast data query; since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location according to the hash fingerprint of the data, and record the number of repetitions of the data at the second storage location, therefore, when deleting or inserting data, it can be determined whether the data to be processed is duplicate data according to the record in the first storage location, and the number of repetitions of the data to be processed can be determined according to the record in the second storage location of the second hash component, thereby realizing fast deletion or insertion of the data to be processed by modifying the record of whether the data to be processed is duplicate data in the first storage location and / or the record of the number of repetitions of the data to be processed in the second storage location. The index structure provided by the embodiment of the present application can support query, insertion and deletion operations on a large amount of duplicate data under the premise of ensuring high performance, which not only solves the problem of index unavailability caused by data update and insertion and deletion of duplicate data in streaming scenarios, but also supports efficient data skipping, further improving query efficiency.

[0056] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be described clearly and completely below in conjunction with the drawings in one or more embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0057] The terms "first", "second", etc. in the present application and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0058] Figure 1 It is a schematic diagram of a schematic system architecture provided in an embodiment of the present application.

[0059] Figure 1 The application environment of the system architecture shown can be logically divided into a computing layer 11 and a storage layer 12. The computing layer 11 is responsible for data management and access, and its data lake table format can be selected according to actual needs. The storage layer 12 is responsible for data storage, and can use different storage formats, such as Parquet, Avro, and ORC, and can choose different storage methods, such as local storage, distributed storage, or cloud storage. The computing layer 21 and the storage layer 22 can communicate through a network connection.

[0060] The big data computing engine in the computing layer 11 is a tool for processing large-scale data, which can achieve efficient data processing and analysis. In the open source community, commonly used big data computing engines include Apache Spark, Apache Flink, and Presto. These big data computing engines play an important role in different scenarios and applications, providing users with rich data processing and analysis functions.

[0061] The data lake table in the computing layer 11 is a table format that enables the data lake to have ACID (atomicity, consistency, isolation, and durability) transaction capabilities and enhances data management capabilities. In the open source field, there are some products such as Iceberg, Hudi, and Delta Lake, which are open source solutions for data lake tables.

[0062] In actual deployment, according to the application scenario, the deployment of the computing engine may include but is not limited to the following scenarios (1) to (5):

[0063] (1) A big data operating environment that does not use data lake tables for deployment;

[0064] (2) A big data operating environment deployed using data lake tables;

[0065] (3) Data warehouse operating environment that is not deployed using data lake tables;

[0066] (4) Data warehouse operating environment deployed using data lake tables;

[0067] (5) A lake-warehouse integrated operating environment that uses data lake tables, big data, and data warehouses.

[0068] The operating environment described in any one of (1) to (5) above may include, but is not limited to, a physical machine environment and a cloud environment, which is not specifically limited here.

[0069] The technical solution provided in the embodiments of the present application can be applied to data storage devices such as databases and data warehouses, as well as scenarios such as big data processing.

[0070] The index structure provided in the embodiment of the present application can be applicable to different data types, that is, the index structure provided in the embodiment of the present application can support query, deletion and insertion operations on different types of data, and the different data types include but are not limited to integer, floating point, character, and date types.

[0071] The technical solution provided by the embodiments of the present application is described in detail below in conjunction with the accompanying drawings.

[0072] Figure 2 It is a schematic diagram of an index structure of an embodiment of the present application.

[0073] Figure 2 The index structure shown includes a first hash component and a second hash component. The first hash component includes a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used to determine the hash fingerprint of the data, the second hash function is used to determine the first storage position in the two-dimensional fingerprint matrix according to the hash fingerprint, and the first storage position is used to store the hash fingerprint and record whether the hash fingerprint is repeated data. The second hash component includes a third hash function and a one-dimensional fingerprint vector, the third hash function is used to determine the second storage position in the one-dimensional fingerprint vector according to the hash fingerprint, and the second storage position is used to record the number of repetitions of the hash fingerprint.

[0074] The first hash function can be expressed as a fingerprint hash function. The hash fingerprint determined by the first hash function is unique, that is, when determining the data fingerprint according to the first hash function, different hash fingerprints can be obtained for different data. The length of the hash fingerprint can be a default value or can be specified by the user, and is not specifically limited here. For example, the user can specify that the length of the hash fingerprint is 32 bits.

[0075] The second hash function can be expressed as a mapping hash function. The first storage location determined by the second hash function can be a storage location among multiple storage locations of the two-dimensional fingerprint matrix. In some embodiments, the second hash function is used to determine the first storage location in the two-dimensional fingerprint matrix, and specifically, the second hash function can be used to determine multiple row indexes in the two-dimensional fingerprint matrix, and the first storage location can specifically be a storage location in the row corresponding to the multiple row indexes in the two-dimensional fingerprint matrix. In this way, when storing hash fingerprints in the index structure, a storage location can be flexibly selected from multiple rows of the two-dimensional fingerprint matrix for storage.

[0076] It should be noted that the multiple row indexes determined by the second hash function can be the row indexes of some rows in the two-dimensional fingerprint matrix, which can facilitate the determination of some rows in the two-dimensional fingerprint matrix according to the second hash function when performing data query based on the index structure, and perform data query in the partial rows, thereby narrowing the query scope and improving query efficiency. In addition, for different data, the second hash function can obtain different row indexes or the same row index. When the same row index is obtained, the first storage locations corresponding to different data are different, so that hash fingerprints of different data can be stored in different first storage locations.

[0077] In the case where the second hash function is used to determine multiple row indexes in the two-dimensional fingerprint matrix, the second hash function may include multiple mapping hash functions, each mapping hash function is used to determine a row index in the two-dimensional fingerprint matrix, and multiple mapping hash functions are used to determine multiple row indexes in the two-dimensional fingerprint matrix. Optionally, in some embodiments, the first hash function may specifically include two mapping hash functions, which may be represented as a first mapping hash function and a second mapping hash function, respectively, the first mapping hash function is used to determine the first row index according to the hash fingerprint and the number of rows of the two-dimensional fingerprint matrix, and the second mapping hash function is used to determine the second row index according to the hash fingerprint and the first row index. The first row index and the second row index may be a row index in the two-dimensional fingerprint matrix. The first mapping hash function is used to determine the first row index according to the hash fingerprint and the number of rows of the two-dimensional fingerprint matrix, and specifically, the first mapping hash function is used to perform a remainder operation on the number of rows of the two-dimensional fingerprint matrix according to the hash fingerprint to obtain the first row index. The second mapping hash function is used to determine the second row index according to the hash fingerprint and the first row index, and specifically, the second mapping hash function is used to perform an XOR operation on the first row index according to the hash fingerprint to obtain the second row index.

[0078] Optionally, in some implementations, the first mapping hash function may be expressed as:

[0079] B1(x)=F p modm;

[0080] The second mapping hash function can be expressed as:

[0081] B2(x)=B1(x)⊕Hash(F p ).

[0082] Among them, F p is the hash fingerprint of the data, and m is the number of rows of the two-dimensional fingerprint matrix.

[0083] The two-dimensional fingerprint matrix can be implemented using different data structures, such as dictionaries, linked lists, vectors, etc. In this embodiment, the data structure of the two-dimensional fingerprint matrix can be used as an example to illustrate the most efficient array structure. In addition, to simplify the description, in the subsequent description, a "bucket" will be used to represent a row of data in the two-dimensional fingerprint matrix, and a "slot" will be used to represent an element in the "bucket" (i.e., a storage location).

[0084] The two-dimensional fingerprint matrix includes multiple storage locations, and the structure of each storage location is the same. Taking the first storage location as an example, in some embodiments, the first storage location can be composed of a data bit and a flag bit, and the data bit is used to store the hash fingerprint of the data, and the flag bit is used to record whether the hash fingerprint of the data is repeated data. Wherein, the flag bit value is 0 to indicate that the hash fingerprint of the data is not repeated data (equivalent to the data is not repeated data), and the flag bit value is 1 to indicate that the hash fingerprint of the data is repeated data (equivalent to the data is repeated data), or, it can also be that the flag bit value is 1 to indicate that the hash fingerprint of the data is not repeated data (equivalent to the data is not repeated data), and the flag bit value is 0 to indicate that the hash fingerprint of the data is repeated data (equivalent to the data is repeated data). This embodiment is illustrated by taking the flag bit value as 0 to indicate non-repeated data and the value 1 as an example to indicate repeated data.

[0085] In order to understand the structure of the two-dimensional fingerprint matrix, Figure 3 A more specific implementation method shown is used as an example for explanation.

[0086] Figure 3 It is a schematic diagram of a two-dimensional fingerprint matrix according to an embodiment of the present application. Figure 3 The two-dimensional fingerprint matrix shown includes 4 rows and 4 columns (in actual applications, it may include more rows and more columns), the row indexes of the 4 rows are respectively represented as bucket [0], bucket [1], bucket [2] and bucket [3], and the column indexes of the 4 columns (that is, the 4 elements in each row) are respectively represented as slot [0], slot [1], slot [2] and slot [3]. Figure 3 The two-dimensional fingerprint matrix shown includes 4×4 storage locations (ie, 4×4 elements), and the structure of each storage location is the same. Taking the first storage location of the first row (corresponding to slot [0] in bucket [0]) as an example, the storage location includes a data bit and a flag bit, the data bit is used to store the hash fingerprint of the data, and the flag bit is used to record whether the hash fingerprint stored in the data bit is duplicate data.

[0087] The second storage location determined by the third hash function may be a storage location among multiple storage locations of the one-dimensional fingerprint vector. For different data, the third hash fingerprint may obtain different second storage locations, so that the number of repetitions of different hash fingerprints may be recorded in different second storage locations.

[0088] In some embodiments, the third hash function may include a modulus function, which is used to determine the remainder of the hash fingerprint of the data divided by the length of the one-dimensional fingerprint vector, and the remainder is used to indicate the second storage position in the one-dimensional fingerprint vector. For example, the remainder may be a position index in the one-dimensional fingerprint vector, and the storage position corresponding to the position index in the one-dimensional fingerprint vector is the second storage position.

[0089] Alternatively, in some embodiments, the modulo function may be expressed as:

[0090] Modulo(F p )=F p %n.

[0091] Among them, F p is the hash fingerprint of the data, and n is the length of the one-dimensional fingerprint vector.

[0092] The one-dimensional fingerprint vector can be implemented using different data structures, such as a dictionary, a linked list, a vector, etc. In this embodiment, the data structure of the one-dimensional fingerprint vector can be used as an example to illustrate the most efficient array structure. In addition, to simplify the description, in the subsequent description, "slots" will be used to represent the elements (i.e., storage locations) in the one-dimensional fingerprint vector.

[0093] Optionally, in some embodiments, the second hash component may further include a fourth hash function, the fourth hash function is used to determine a fingerprint identifier of the hash fingerprint, and the fingerprint identifier is used to identify the uniqueness of the fingerprint data. In the case where the second hash component includes the fourth hash function, the second storage location is also used to store the fingerprint identifier, so that the fingerprint identifier and the number of repetitions of the hash fingerprint are stored correspondingly.

[0094] Optionally, in some embodiments, the fourth hash function may include a quotient function, which is used to determine the quotient of the hash fingerprint of the data and the length of the one-dimensional fingerprint vector. Accordingly, the fingerprint identifier of the hash fingerprint may include the quotient. In some embodiments, the quotient function may be expressed as:

[0095] Quotient(F p )=F p / n.

[0096] Among them, F p is the hash fingerprint of the data, and n is the length of the one-dimensional fingerprint vector.

[0097] The one-dimensional fingerprint vector includes multiple storage locations, and the structure of each storage location is the same. Taking the second storage location as an example, in some embodiments, when the second storage location is used to record the number of repetitions of the hash fingerprint, the second storage location includes a counting bit, and when the second storage location is used to store the fingerprint identifier of the hash fingerprint and record the number of repetitions of the hash fingerprint, the second storage location can be composed of a data bit and a counting bit. Among them, the data bit is used to store the fingerprint identifier of the hash fingerprint, and the counting bit is used to record the number of repetitions of the hash fingerprint. The number of counting bits can be specified by the user or be a default value, for example, the user can specify the number of counting bits to be 8.

[0098] In order to understand the structure of the one-dimensional fingerprint vector, Figure 4A more specific implementation method shown is used as an example for explanation.

[0099] Figure 4 It is a schematic diagram of a one-dimensional fingerprint vector according to an embodiment of the present application. Figure 4 The length of the one-dimensional fingerprint vector shown is 4 (it can be longer in actual applications), that is, the one-dimensional fingerprint vector includes 4 storage locations, which can be represented as slot [0], slot [1], slot [2] and slot [3] respectively. Figure 4 In the one-dimensional fingerprint vector shown, the structure of each storage location is the same. Taking the first storage location (corresponding to slot [0]) as an example, the storage location includes a data bit and a count bit. The data bit is used to store the fingerprint identifier of the hash fingerprint, and the count bit is used to record the number of repetitions of the hash fingerprint corresponding to the fingerprint identifier stored in the data bit, that is, the number of repetitions of the data corresponding to the hash fingerprint.

[0100] It should be noted that if the second hash component includes the third hash function but does not include the fourth hash function, that is, the storage location in the one-dimensional fingerprint vector is used to record the number of repetitions of the hash fingerprint but not to store the fingerprint identifier of the hash fingerprint, then Figure 4 Each storage location shown will include a count bit but not a data bit.

[0101] The index structure provided by the embodiment of the present application, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint according to the hash fingerprint, therefore, when performing data query, the first storage location can be determined based on the first hash component, and the data query can be performed at the first storage location, thereby narrowing the query range in the index structure, thereby realizing fast data query; since the first storage location is also used to record whether the hash fingerprint of the data is duplicate data, the second hash component can determine the second storage location according to the hash fingerprint of the data, and record the number of repetitions of the data at the second storage location, therefore, when deleting or inserting data, it can be determined whether the data to be processed is duplicate data according to the record in the first storage location, and the number of repetitions of the data to be processed can be determined according to the record in the second storage location of the second hash component, thereby realizing fast deletion or insertion of the data to be processed by modifying the record of whether the data to be processed is duplicate data in the first storage location and / or the record of the number of repetitions of the data to be processed in the second storage location. The index structure provided by the embodiment of the present application can support query, insertion and deletion operations on a large amount of duplicate data under the premise of ensuring high performance, which not only solves the problem of index unavailability caused by data update and insertion and deletion of duplicate data in streaming scenarios, but also supports efficient data skipping, further improving query efficiency.

[0102] The index structure provided by the embodiment of the present application can store the hash fingerprint of the data in the first hash component, which can ensure fast query efficiency when performing data query. For duplicate data, it can be placed in the second hash component to solve the problem of processing duplicate data. The index structure only sets a count bit for a small part of the duplicate data and sets a flag bit for most of the single data, which can effectively reduce the space occupied by the index structure.

[0103] Based on the index structure provided in the embodiment of the present application, the embodiment of the present application also provides a method for inserting data in the index structure, a method for deleting data in the index structure, and a method for querying data based on the index structure. Figures 5 to 12 The illustrated embodiment is described by way of example.

[0104] Figure 5 is a flow chart of a method for inserting data into an index structure according to an embodiment of the present application. The index structure is an index structure provided by an embodiment of the present application. For details, see Figures 2 to 4 The illustrated embodiment will not be described again here. Figure 5 The method shown includes the following steps.

[0105] Step S502: determining a hash fingerprint of the data to be inserted according to the first hash function and determining a first storage position in the two-dimensional fingerprint matrix according to the second hash function.

[0106] When inserting data into the index structure, the hash fingerprint of the data to be inserted can be first determined according to the first hash function in the index structure. In some embodiments, the first hash function can be represented as Hash(), and when the hash fingerprint of the data to be inserted is determined according to the first hash function, it can be determined by the following formula:

[0107] F p =Hash(x).

[0108] After the hash fingerprint of the data to be inserted is obtained, the first storage position in the two-dimensional fingerprint matrix can be determined according to the second hash function.

[0109] In some embodiments, the second hash function may include multiple mapping hash functions, each mapping hash function is used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions are used to determine multiple row indexes in the two-dimensional fingerprint matrix. In this way, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indexes in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically a storage location in the row corresponding to the multiple row indexes in the two-dimensional fingerprint matrix.

[0110] Taking the second hash function including two hash mapping functions B1() and B2() as an example, when determining the first storage position of the two-dimensional fingerprint matrix according to the second hash function, it can be determined by the following formula:

[0111] B1(x)=F p modm;

[0112] B2(x)=B1(x)⊕Hash(F p ).

[0113] Where m is the number of rows in the two-dimensional fingerprint matrix, F p is the hash fingerprint of the data to be inserted.

[0114] According to the above two hash mapping functions B1() and B2(), two row indexes of the two-dimensional fingerprint matrix can be obtained. The two row indexes point to two rows in the two-dimensional fingerprint matrix. The first storage position is the storage position in these two rows, indicating that the hash fingerprint of the data to be inserted can be stored in these two rows.

[0115] Step S504: Determine whether the first storage location stores fingerprint data that is identical to the hash fingerprint of the data to be inserted.

[0116] After determining the first storage location, before inserting the hash fingerprint of the data to be inserted into the index structure, it is necessary to determine whether the data to be inserted is duplicate data, so as to perform different data insertion operations according to the determination result. When determining whether the data to be inserted is duplicate data, it can be determined whether the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted. If the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted, it can be explained that the same data has been inserted into the index structure before, and the data to be inserted is duplicate data. If the first storage location does not store fingerprint data identical to the hash fingerprint of the data to be inserted, it can be explained that the same data has not been inserted into the index structure before, and the data to be inserted is not duplicate data.

[0117] In some embodiments, the first storage location may be a storage location in two rows in a two-dimensional fingerprint matrix. In this case, when determining whether the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted, it is possible to search the hash fingerprints stored in these two rows of the two-dimensional fingerprint matrix to see whether there is fingerprint data identical to the hash fingerprint of the data to be inserted. If so, it can be determined that the data to be inserted is duplicate data. If not, it can be determined that the data to be inserted is not duplicate data.

[0118] If the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the data to be inserted is duplicate data, step S506 may be performed. If the first storage location does not store fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the data to be inserted is not duplicate data, step S508 may be performed.

[0119] Step S506: When fingerprint data identical to the hash fingerprint of the data to be inserted is stored in the first storage location, determine the second storage location in the one-dimensional fingerprint vector according to the third hash function, and record the number of repetitions of the hash fingerprint of the data to be inserted in the second storage location.

[0120] In the case where fingerprint data identical to the hash fingerprint of the data to be inserted is stored at the first storage location, the insertion operation of the data to be inserted can be implemented by modifying the number of repetitions of the hash fingerprint of the data to be inserted recorded in the index structure (equivalent to the number of repetitions of the data to be inserted), without repeatedly storing the hash fingerprint of the data to be inserted in the two-dimensional fingerprint matrix, so as to avoid repeated storage and reduce the memory usage of the index structure.

[0121] When modifying the number of repetitions of the hash fingerprint of the data to be inserted recorded in the index structure, the second storage position in the one-dimensional fingerprint vector can be determined according to the third hash function in the index structure, and the number of repetitions of the hash fingerprint of the data to be inserted can be recorded in the second storage position.

[0122] When recording the number of repetitions of the hash fingerprint of the data to be inserted in the second storage location, in some embodiments, the following steps may be included:

[0123] Determine whether the hash fingerprint of the data to be inserted has been recorded in the first storage location as duplicate data;

[0124] In the case where the hash fingerprint of the data to be inserted is not recorded as duplicate data in the first storage location, the number of repetitions of recording the hash fingerprint of the data to be inserted as duplicate data in the first storage location and recording the hash fingerprint of the data to be inserted in the second storage location is 2;

[0125] In the case that the hash fingerprint of the data to be inserted recorded in the first storage location is duplicate data, the number of repetitions recorded in the second storage location is increased by one.

[0126] The first storage location is used to store the hash fingerprint of the data to be inserted and record whether the hash fingerprint of the data to be inserted is duplicate data. In the case of determining that the data to be inserted is duplicate data, it can be further determined whether the first storage location has recorded the hash fingerprint of the data to be inserted as duplicate data. In some embodiments, the first storage location includes a data bit and a flag bit, the data bit is used to store the hash fingerprint of the data to be inserted, and the flag bit is used to record whether the hash fingerprint of the data to be inserted is duplicate data. When determining whether the first storage location has recorded the hash fingerprint of the data to be inserted as duplicate data, it can be determined according to the flag bit in the first storage location. For example, when the flag bit value is 0 for duplicate data and 1 for non-duplicate data, if the flag bit in the first storage location is 0, it means that the first storage location has not recorded the hash fingerprint of the data to be inserted as duplicate data, and if the flag bit in the first storage location is 1, it means that the first storage location has recorded the hash fingerprint of the data to be inserted as duplicate data.

[0127] In the case where the hash fingerprint of the data to be inserted is not recorded as duplicate data in the first storage location, it is necessary to record in the first storage location that the hash fingerprint of the data to be inserted is duplicate data. For example, the value of the flag bit in the first storage location can be set to 1. In addition, in the case where the hash fingerprint of the data to be inserted is not recorded as duplicate data in the first storage location, it can be explained that the hash fingerprint of the data to be inserted is the second repetition, and therefore, it is necessary to record the number of repetitions as 2 in the second storage location. For example, the second storage location includes a count bit, and the count bit is used to record the number of repetitions of the hash fingerprint of the data to be inserted, and the value of the count bit can be set to 2.

[0128] In the case where the hash fingerprint of the data to be inserted recorded in the first storage location is duplicate data, the number of repetitions recorded in the second storage location can be directly increased by one without modifying the record in the first storage location. For example, if the count bit in the second storage location is 2 and the flag bit in the first storage location is 1, the count bit in the second storage location can be set to 3 without modifying the value of the flag bit in the first storage location.

[0129] Optionally, in some implementations, when the second hash component in the index structure includes a fourth hash function, the following steps may also be included:

[0130] Determine a fingerprint identifier of a hash fingerprint of the data to be inserted according to a fourth hash function;

[0131] The fingerprint identification is stored in the second storage location.

[0132] When the fingerprint identifier is stored in the second storage location, the fingerprint identifier can be stored in the data bit of the second storage location. The fourth hash function can be a quotient function, and the fingerprint identifier can be the quotient of the fingerprint identifier of the hash fingerprint of the data to be inserted and the length of the one-dimensional fingerprint vector.

[0133] Step S508: If fingerprint data identical to the hash fingerprint of the data to be inserted is not stored in the first storage location, the hash fingerprint of the data to be inserted is stored in the first storage location and it is recorded that the hash fingerprint of the data to be inserted is not duplicate data.

[0134] In some embodiments, the second hash function in the index structure is used to determine multiple row indexes of the two-dimensional fingerprint matrix, and the first storage location is a storage location in multiple rows corresponding to the multiple row indexes in the two-dimensional fingerprint matrix. In this way, when the hash fingerprint of the data to be inserted is stored in the first storage location and the hash fingerprint of the data to be inserted is recorded as not duplicate data, a storage location can be selected from multiple rows corresponding to the multiple row indexes in the two-dimensional fingerprint matrix to store the hash fingerprint of the data to be inserted and record that the hash fingerprint of the data to be inserted is not duplicate data.

[0135] When selecting storage locations from multiple rows corresponding to multiple row indexes in a two-dimensional fingerprint matrix, optionally, the row with the smallest load can be first determined from the multiple rows corresponding to the multiple row indexes, and then a storage location can be selected in the row with the smallest load to store the hash fingerprint of the data to be inserted and record that the hash fingerprint of the data to be inserted is not duplicate data. The load can be the number of storage locations in the row that have stored data. The larger the number, the greater the load of the row, and the smaller the number, the smaller the load of the row. By selecting the storage location in the row with the smallest load to store data, load balancing of each row of data in the two-dimensional fingerprint matrix can be achieved.

[0136] In order to facilitate understanding of the method for inserting data into an index structure provided in the embodiment of the present application, the following will be Figure 6 and Figure 7 A more specific implementation method shown is used as an example for explanation.

[0137] Figure 6 and Figure 7 The index structure shown includes a first hash component and a second hash component. The first hash component includes a fingerprint hash function, two mapping hash functions ( Figure 6 and Figure 7 (not shown) and a two-dimensional fingerprint matrix, the second hash component includes a quotient function, a remainder function ( Figure 6 and Figure 7 The two-dimensional fingerprint matrix consists of 4 buckets, each bucket has 4 slots, and the one-dimensional fingerprint vector consists of 4 slots.

[0138] Assume that the date column in the data table T1 has data [20000506, 20060908, 20120504, 20151212, 20180909, 20200202, 20230801, 20240301] flowing in, and the hash fingerprints of these data calculated by the fingerprint hash function are [11, 13, 15, 17, 102, 104, 106, 108] respectively. All hash fingerprints point to buckets [1] and [2] in the two-dimensional fingerprint matrix respectively through two mapping hash functions. Then, when constructing the index structure provided by the embodiment of the present application for the streaming data T1.date, the following can be obtained: Figure 6 Results shown.

[0139] Assume that two more data with the date of 20230801 flow into data table T1, then it is necessary to update Figure 6 Specifically, Figure 7 As shown, the hash fingerprint 106 of 20230801 can be calculated according to the fingerprint hash function, and then the hash fingerprint 106 is determined to point to bucket [1] and bucket [2] in the two-dimensional fingerprint matrix according to the two mapping hash functions. Since the same fingerprint data 106 already exists in bucket [1], the flag bit corresponding to 106 is set to 1, and then the data is inserted into the one-dimensional fingerprint vector. When inserting data into the one-dimensional fingerprint vector, since the one-dimensional fingerprint vector has a total of 4 slots, the quotient and remainder of the hash fingerprint 106 relative to 4 are 26 and 2 respectively. Therefore, the quotient 26 can be stored in the storage position corresponding to the slot [2] of the one-dimensional fingerprint vector. Since a total of 3 data with the date of 20230801 are inserted into the data table T1, the count bit in slot [2] is set to 3.

[0140] Assume that there is another data with the date of 20230801 flowing into the data table T1, then, Figure 7 The index structure shown in the following table is updated. Figure 7 When the index structure shown is updated, based on the method described above, the count bit in slot [2] of the one-dimensional fingerprint vector can be directly increased by 1.

[0141] It can be seen that when inserting data into the index structure, the fingerprint of the data is first calculated according to the fingerprint hash function, and the positions of the two hash buckets are calculated using two mapping hash functions. Then, it is checked whether the same fingerprint data exists in the two-dimensional fingerprint matrix. If not, it is directly stored in the free slot with the flag position set to 0. If the same fingerprint data exists, its flag bit is checked. If the flag bit is 0, it is set to 1, and the data is inserted into the two-dimensional fingerprint matrix and the initial value of the corresponding count bit is set to 2. If the flag bit is 1, the position of the data in the one-dimensional fingerprint vector is found and the count bit is increased by one.

[0142] It should be noted that, in practical applications, when using the quotient function and remainder function to determine the quotient and remainder of the hash fingerprint relative to the one-dimensional fingerprint vector, there may be a situation where different hash fingerprints correspond to different quotients but the same remainder. In this case, the fingerprint identifiers and repetition times of different hash fingerprints will be stored in the same storage location in the one-dimensional fingerprint vector. In order to be able to store the fingerprint identifiers and repetition times of different hash fingerprints in the same storage location, the storage method can use a linked list or other forms, and distinguish the repetition times of different hash fingerprints according to the fingerprint identifier of the hash fingerprint.

[0143] Based on the method for inserting data in an index structure provided in an embodiment of the present application, since the first storage position in the index structure is used to record whether the hash fingerprint of the data is duplicate data, the second hash component in the index structure can determine the second storage position according to the hash fingerprint of the data, and record the number of repetitions of the data in the second storage position. Therefore, when inserting data in the index structure, it is possible to determine whether the data to be processed is duplicate data according to the record in the first storage position, and determine the number of repetitions of the data to be processed according to the record in the second storage position of the second hash component. Therefore, by modifying the record of whether the data to be processed is duplicate data in the first storage position and / or the record of the number of repetitions of the data to be processed in the second storage position, the data to be processed can be quickly inserted, and the insertion operation of a large amount of duplicate data can be supported under the premise of ensuring high performance, which solves the problem of index unavailability caused by data update and duplicate data insertion in streaming scenarios.

[0144] Figure 8 is a flowchart of a method for deleting data in an index structure according to an embodiment of the present application. The index structure is an index structure provided by an embodiment of the present application. For details, see Figures 2 to 4 The illustrated embodiment will not be described again here. Figure 8 The method shown includes the following steps.

[0145] Step S802: determining a hash fingerprint of the data to be deleted according to a first hash function and determining a first storage position in a two-dimensional fingerprint matrix according to a second hash function.

[0146] When deleting data in the index structure, the hash fingerprint of the data to be deleted can be determined first according to the first hash function in the index structure. In some embodiments, the first hash function can be represented as Hash(), and when the hash fingerprint of the data to be deleted is determined according to the first hash function, it can be determined by the following formula:

[0147] F p =Hash(x).

[0148] After the hash fingerprint of the data to be deleted is obtained, the first storage position in the two-dimensional fingerprint matrix can be determined according to the second hash function.

[0149] In some embodiments, the second hash function may include multiple mapping hash functions, each mapping hash function is used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions are used to determine multiple row indexes in the two-dimensional fingerprint matrix. In this way, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indexes in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically a storage location in the row corresponding to the multiple row indexes in the two-dimensional fingerprint matrix.

[0150] Taking the second hash function including two hash mapping functions B1() and B2() as an example, when determining the first storage position of the two-dimensional fingerprint matrix according to the second hash function, it can be determined by the following formula:

[0151] B1(x)=F p modm;

[0152] B2(x)=B1(x)⊕Hash(F p ).

[0153] Where m is the number of rows in the two-dimensional fingerprint matrix, F p is the hash fingerprint of the data to be deleted.

[0154] According to the above two hash mapping functions B1() and B2(), two row indexes of the two-dimensional fingerprint matrix can be obtained. The two row indexes point to two rows in the two-dimensional fingerprint matrix. The first storage position is the storage position in these two rows, indicating that if the hash fingerprint of the data to be deleted is stored in the index structure, the hash fingerprint is stored in these two rows.

[0155] Step S804: Determine whether the first storage location stores fingerprint data that is identical to the hash fingerprint of the data to be deleted.

[0156] After determining the first storage location, before deleting the hash fingerprint of the data to be deleted in the index structure, it is necessary to determine whether the hash fingerprint of the data to be deleted is stored in the index structure. When determining whether the hash fingerprint of the data to be deleted is stored in the index structure, it can be determined whether the first storage location stores fingerprint data identical to the hash fingerprint of the data to be deleted. If the first storage location stores fingerprint data identical to the hash fingerprint of the data to be deleted, it can be determined that the hash fingerprint of the data to be deleted is stored in the index structure; if the first storage location does not store fingerprint data identical to the hash fingerprint of the data to be inserted, it can be determined that the hash fingerprint of the data to be deleted is not stored in the index structure.

[0157] In some embodiments, the first storage location may be a storage location in two rows in a two-dimensional fingerprint matrix. In this case, when determining whether the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted, it is possible to search the hash fingerprints stored in the two rows of the two-dimensional fingerprint matrix to see whether there is fingerprint data identical to the hash fingerprint of the data to be inserted. If so, it can be determined that the hash fingerprint of the data to be deleted is stored in the index structure. If not, it can be determined that the hash fingerprint of the data to be deleted is not stored in the index structure.

[0158] If the first storage location stores fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the hash fingerprint of the data to be deleted is stored in the index structure, step S806 may be executed. If the first storage location does not store fingerprint data identical to the hash fingerprint of the data to be inserted, that is, if the hash fingerprint of the data to be deleted is not stored in the index structure, an exception may be returned.

[0159] Step S806: When fingerprint data identical to the hash fingerprint of the data to be deleted is stored in the first storage location, determine whether the hash fingerprint of the data to be deleted is duplicate data according to the record in the first storage location.

[0160] In the case where the first storage location stores fingerprint data identical to the hash fingerprint of the data to be deleted, it is necessary to further determine whether the hash fingerprint of the data to be deleted is duplicate data (equivalent to whether the data to be deleted is duplicate data) so as to perform different data deletion operations according to the judgment result. Since the first storage location is used to store the hash fingerprint of the data to be deleted and record whether the hash fingerprint of the data to be deleted is duplicate data, when determining whether the hash fingerprint of the data to be deleted is duplicate data, it can be determined based on the record of the first storage location.

[0161] In some embodiments, the first storage location includes a data bit and a flag bit, the data bit is used to store the hash fingerprint of the data to be deleted, and the flag bit is used to record whether the hash fingerprint of the data to be deleted is duplicate data. In this way, when determining whether the hash fingerprint of the data to be deleted is duplicate data, a judgment can be made based on the flag bit. For example, when the flag bit value is 0 to indicate duplicate data and the flag bit value is 1 to indicate non-duplicate data, if the flag bit in the first storage location is 0, it means that the hash fingerprint of the data to be deleted is not duplicate data, and if the flag bit in the first storage location is 1, it means that the hash fingerprint of the data to be deleted is duplicate data.

[0162] If it is determined that the hash fingerprint of the data to be deleted is not duplicate data, step S808 may be executed; if it is determined that the hash fingerprint of the data to be deleted is duplicate data, step S810 may be executed.

[0163] Step S808: If the hash fingerprint of the data to be deleted is not duplicate data, the hash fingerprint of the data to be deleted in the first storage location is deleted.

[0164] Step S810: when the hash fingerprint of the data to be deleted is duplicate data, determine the second storage position in the one-dimensional fingerprint vector according to the third hash function, and reduce the number of repetitions of the hash fingerprint of the data to be deleted recorded in the second storage position by one.

[0165] If the hash fingerprint of the data to be deleted is not duplicate data, the hash fingerprint of the data to be deleted stored in the first storage location can be directly deleted. If the hash fingerprint of the data to be deleted is duplicate data, the deletion operation can be achieved by modifying the number of repetitions of the hash fingerprint of the data to be deleted recorded in the index structure, without deleting the hash fingerprint of the data to be deleted stored in the index structure.

[0166] When the deletion operation is implemented by modifying the number of repetitions recorded in the index structure, specifically, the second storage position in the one-dimensional fingerprint vector can be determined according to the third hash function in the index structure, and the number of repetitions of the hash fingerprint of the data to be deleted recorded in the second storage position can be reduced by 1. For example, the second storage position includes a counting bit, and the counting bit is used to record the number of repetitions of the hash fingerprint of the data to be deleted. When deleting the data to be deleted, the value of the counting bit in the second storage position can be reduced by 1.

[0167] In some implementations, after the number of repetitions recorded in the second storage location is reduced by one, the following operations may be performed:

[0168] Determine if the number of repetitions is 1;

[0169] When the number of repetitions is 1, determine whether the deletion condition is met;

[0170] When the deletion condition is met, the content stored in the second storage location is deleted and the hash fingerprint of the to-be-deleted data is recorded in the first storage location as non-duplicate data.

[0171] After the number of repetitions recorded in the second storage location is reduced by one, there are two cases for the value of the number of repetitions, one is that the number of repetitions is greater than 1, and the other is that the number of repetitions is equal to 1. If the number of repetitions is greater than 1, it means that after deleting the data to be deleted, the data to be deleted is still duplicate data, and there is no need to modify the relevant records in the index structure that the data to be deleted is duplicate data. If the number of repetitions is equal to 1, it means that after deleting the data to be deleted, the data to be deleted is not duplicate data, and it is necessary to modify the relevant records in the index structure that the data to be deleted is duplicate data to avoid recording errors. In this way, after the number of repetitions recorded in the second storage location is reduced by one, it is necessary to determine whether the number of repetitions becomes 1. If the number of repetitions is not 1, there is no need to modify the relevant records in the index structure that the data to be deleted is duplicate data. If the number of repetitions is 1, it is necessary to modify the relevant records in the index structure that the data to be deleted is duplicate data.

[0172] When modifying the relevant records of the data to be deleted being duplicate data in the index structure, the content stored in the second storage location can be deleted and the hash fingerprint of the data to be deleted can be recorded in the first storage location as not being duplicate data. Among them, the content of the second storage location can be the number of repetitions (corresponding to the case where the second hash component includes the third hash function but does not include the fourth hash function), or the fingerprint identifier of the number of repetitions and the hash fingerprint of the data to be deleted (corresponding to the case where the second hash component includes the third hash function and the fourth hash function). When recording in the first storage location that the hash fingerprint of the data to be deleted is not duplicate data, for example, the value of the flag bit in the first storage location can be set from 1 to 0, where 0 indicates that the hash fingerprint of the data to be deleted is not duplicate data.

[0173] In some embodiments, considering that the data is dynamically updated, in order to avoid frequent deletion and insertion of data in a one-dimensional fingerprint vector affecting the overall performance, when the number of repetitions is 1, the content stored in the second storage location may not be deleted immediately, but it is determined whether the deletion condition is met. If the deletion condition is not met, the content stored in the second storage location may not be deleted. If the deletion condition is met, the content stored in the second storage location will be deleted. In this way, the performance degradation problem caused by frequent deletion of data in the one-dimensional fingerprint vector can be avoided. Among them, the deletion condition can be determined according to actual needs and is not specifically limited here. For example, the deletion condition can be a fixed time point, so that the corresponding data with a repetition number of 1 in the one-dimensional fingerprint vector can be regularly deleted.

[0174] When the deletion condition is met, after deleting the content stored in the second storage location, it is possible to further record in the first storage location that the hash fingerprint of the data to be deleted is not duplicate data. In this way, since the modification of the record of duplicate data in the first storage location is also performed when the deletion condition is met, it is possible to avoid frequent modification of data in the two-dimensional fingerprint matrix to affect the overall performance.

[0175] In order to facilitate understanding of the method for deleting data in an index structure provided in an embodiment of the present application, the following will be described in detail. Figure 7 and Fig. 9 A more specific implementation method shown is used as an example for explanation.

[0176] based on Figure 7 As shown in the index structure, suppose you need to delete a data item with a date of 20230801 and a data item with a date of 20240301 in the data table T1, then you need to Figure 7 Specifically, Fig. 9 As shown, when deleting a data item with data 20230801, the hash fingerprint 106 calculated by the fingerprint hash function can be used, and then the row index in the two-dimensional fingerprint matrix can be calculated by two mapping hash functions respectively. The results are buckets [1] and buckets [2]. It is found that the same data fingerprint as 106 exists in bucket [1], and its flag bit is 1. Therefore, it needs to be processed in the second hash component. Since the quotient and remainder of the hash fingerprint 106 relative to 4 are 26 and 2 respectively, it can be determined that the quotient and the number of repetitions of the hash fingerprint 106 are stored in slot [2] in the one-dimensional fingerprint vector, and the count bit in slot [2] is reduced by 1, that is, set to 2. When deleting a data item with data 20240301, the hash fingerprint 108 calculated by the fingerprint hash function can be used, and then the row index of the two-dimensional fingerprint matrix can be calculated by two mapping hash functions respectively. The results are buckets [1] and buckets [2]. It is found that the same data fingerprint exists in bucket [1], and its flag bit is 0. Therefore, the data can be directly deleted in the first hash component.

[0177] Assuming that a data item with a date of 20230801 needs to be deleted later, based on the method described above, the count bit in slot [2] of the one-dimensional fingerprint vector can be reduced by 1, that is, set to 1, and when the deletion condition is met, the quotient 26 and the count bit 1 in slot [2] are deleted, that is, set to Null and 0 respectively, and the flag bit corresponding to the hash fingerprint 106 in the two-dimensional fingerprint matrix is ​​set to 0.

[0178] It can be seen that when deleting data in the index structure, two mapping hash functions are first used to calculate the positions of the two hash buckets, and then check whether the same fingerprint data exists in the two-dimensional fingerprint matrix. If not, an exception is returned. If the same fingerprint data exists and the flag bit is 0, the corresponding fingerprint data is directly deleted; if the same fingerprint data exists and the flag bit is 1, the location of the data is found in the one-dimensional fingerprint vector and the count bit is reduced by 1. In order to avoid frequent deletion and insertion of data in the one-dimensional fingerprint vector affecting the overall performance, the corresponding data will not be deleted immediately when the count bit of the data in the one-dimensional fingerprint vector is 1, but the corresponding data with the count bit 0 will be deleted only when the deletion conditions are met, and the flag bit of the corresponding data in the two-dimensional fingerprint matrix will be set to 0.

[0179] Based on the method for deleting data in an index structure provided in an embodiment of the present application, since the first storage position in the index structure is used to record whether the hash fingerprint of the data is duplicate data, the second hash component in the index structure can determine the second storage position according to the hash fingerprint of the data, and record the number of repetitions of the data in the second storage position. Therefore, when deleting data in the index structure, it is possible to determine whether the data to be processed is duplicate data according to the record in the first storage position, and determine the number of repetitions of the data to be processed according to the record in the second storage position of the second hash component. Therefore, by modifying the record of whether the data to be processed is duplicate data in the first storage position and / or the record of the number of repetitions of the data to be processed in the second storage position, rapid deletion of the data to be processed can be achieved, and deletion operations of a large amount of duplicate data can be supported while ensuring high performance, thereby solving the problem of index unavailability caused by data updates and duplicate data deletion in streaming scenarios.

[0180] Fig.10 is a flow chart of a data query method based on an index structure in an embodiment of the present application. The index structure includes the index structure provided in the embodiment of the present application. For details, see Figures 2 to 4 The illustrated embodiment will not be described again here. Fig.10 The data query method shown can be Figure 1 The big data execution engine in the computing layer 11 shown executes. Fig.10 The method shown includes the following steps.

[0181] Step S102: receiving a query request.

[0182] When an external client or other system has a data query requirement, it can send a query request to the big data execution engine, and the big data query engine can receive the query request sent by the external client or system. The query request may include an SQL statement, and the SQL query statement includes a query filter condition, which may be an equal value filter condition or other filter condition, which is not specifically limited here.

[0183] Step S104: Acquire multiple index files corresponding to multiple data files, each data file corresponds to an index file, and each index file includes an index structure.

[0184] Multiple data files are objects that need to be queried, that is, when performing data query, the required target data is queried from multiple data files. In this embodiment, for each data file in multiple data files, an index structure provided by the embodiment of the present application can be constructed in advance according to the data in the data file, and the index structure is stored in the index file. When performing data query, the index file corresponding to the multiple data files can be obtained. Each data file can correspond to an index file, and the index file includes an index structure.

[0185] When constructing the index structure provided in the embodiment of the present application, for each data file, the index file corresponding to the data file can be determined in the following manner:

[0186] Get the data in the data file and initialize the index structure;

[0187] The index structure is updated according to the data in the data file, and the updating of the index structure includes inserting data or deleting data in the index structure;

[0188] The index structure is stored in a specified format to obtain an index file corresponding to the data file.

[0189] The data in the data file can be stored in a data lake or a data warehouse. When obtaining the data in the data file, it can be obtained from the corresponding data lake or data warehouse. The types of data in the data file include but are not limited to integer, floating point, character, and date types. The index structure is the index structure provided in the embodiment of the present application. The index structure includes a first hash component and a second hash component. For details, see Figures 2 to 4 The illustrated embodiment will not be described again here. After the index structure is initialized, an empty index structure (equivalent to an empty filter, in which no data is stored) can be obtained.

[0190] When updating the index structure according to the data in the data file, considering that the data in the data file usually changes dynamically, the update of the index structure here may include inserting or deleting data in the index structure (corresponding to inserting or deleting data in the data file). Figure 5 In the embodiment shown, when deleting data in the index structure, the specific implementation method can be found in Figure 8The embodiments shown will not be described in detail here. After the index structure is updated, a usable index structure can be obtained.

[0191] After obtaining the available index structure, the index structure can be formatted in the computing engine, the first hash component and the second hash component in the index structure are converted into the specified index format, and then the converted index data is written to the storage layer ( Figure 1 In the storage layer 12) shown, the storage format of the index data in the storage layer can be Parquet, Avro and ORC formats.

[0192] Step S106: determining a valid file from a plurality of data files according to the query request and a plurality of index files.

[0193] After obtaining multiple index files corresponding to multiple data files, invalid files can be filtered out from the multiple data files according to the query request and the multiple index files to obtain valid files. The valid files store the target data to be queried, and the number of valid files can be one or more.

[0194] In some implementations, when determining a valid file from a plurality of data files according to a query request and a plurality of index files, the following steps may be included:

[0195] Parse and verify the query request and determine the logical plan corresponding to the query request;

[0196] Convert the logical plan into a physical execution plan, which includes query filter conditions;

[0197] Multiple index files are queried according to the query filter condition, and valid files are determined from multiple data files according to the query result.

[0198] The query request includes an SQL query statement. When parsing and analyzing the query request, the SQL parser can be used to perform lexical analysis on the SQL statement in the query request to obtain a lexical analysis general symbol stream, and then perform syntax analysis on the symbol stream to construct a syntax tree. After that, the SQL parser can be used to perform in-depth analysis on the unparsed relations in the syntax tree and generate a resolved logical plan. The specific implementation of using the SQL parser for lexical analysis and using the SQL parser for syntax analysis can be found in the specific implementation in the related art, which will not be described in detail here.

[0199] After obtaining the logical plan, the index structure in the index file can be used as an index to optimize the logical plan and generate a physical execution plan using index optimization, which includes query filter conditions. When generating a physical execution plan, taking the query filter condition as an equal value filter condition as an example, the generated logical plan tree can be traversed, and several equal value filter conditions such as Filter in the logical plan can be replaced with a method of using indexes for query judgment, and a physical execution plan with index optimization can be generated and executed.

[0200] After obtaining the physical execution plan, multiple index files may be queried according to the query filter condition in the physical execution plan, and valid files may be determined from multiple data files according to the query result. In some implementations, when multiple index files are queried according to the query filter condition and valid files are determined from multiple data files according to the query result, the following operations may be performed for each index file:

[0201] Determine a hash fingerprint of the query filter condition according to a first hash function in an index structure of the index file and determine a first storage location in a two-dimensional fingerprint matrix according to a second hash function in the index structure;

[0202] determining whether the first storage location stores fingerprint data identical to the hash fingerprint of the query filter condition;

[0203] In the case where fingerprint data identical to the hash fingerprint of the query filter condition is stored in the first storage location, determining the data file corresponding to the index file as a valid file;

[0204] When the first storage location does not store fingerprint data identical to the hash fingerprint of the query filter condition, the data file corresponding to the index file is determined to be an invalid file.

[0205] The index file includes an index structure. When data is queried according to the query filter condition, the hash fingerprint of the query filter condition (equivalent to the hash fingerprint of the target data) can be determined according to the first hash function in the index structure. In some embodiments, the first hash function can be expressed as Hash(). When the hash fingerprint of the query filter condition is determined according to the first hash function, it can be determined by the following formula:

[0206] F p =Hash(x).

[0207] After the hash fingerprint of the query filtering condition is obtained, the first storage position in the two-dimensional fingerprint matrix can be determined according to the second hash function in the index structure.

[0208] In some embodiments, the second hash function may include multiple mapping hash functions, each mapping hash function is used to determine a row index in the two-dimensional fingerprint matrix, and multiple hash mapping functions are used to determine multiple row indexes in the two-dimensional fingerprint matrix. In this way, when determining the first storage location in the two-dimensional fingerprint matrix according to the second hash function, multiple row indexes in the two-dimensional fingerprint matrix may be determined according to multiple mapping hash functions, and the first storage location is specifically a storage location in the row corresponding to the multiple row indexes in the two-dimensional fingerprint matrix.

[0209] Taking the second hash function including two hash mapping functions B1() and B2() as an example, when determining the first storage position of the two-dimensional fingerprint matrix according to the second hash function, it can be determined by the following formula:

[0210] B1(x)=F p modm;

[0211] B2(x)=B1(x)⊕Hash(F p ).

[0212] Where m is the number of rows in the two-dimensional fingerprint matrix, F p is the hash fingerprint of the data to be inserted.

[0213] According to the above two hash mapping functions B1() and B2(), two row indexes of the two-dimensional fingerprint matrix can be obtained. The two row indexes point to two rows in the two-dimensional fingerprint matrix. The first storage position is the storage position in these two rows, indicating that if the hash fingerprint of the target data is stored in the index structure, the hash fingerprint is stored in these two rows.

[0214] After determining the hash fingerprint of the query filter condition and the first storage position in the two-dimensional fingerprint matrix, it can be determined whether the hash fingerprint of the query filter condition is stored in the index structure, that is, it can be determined whether the first storage position stores fingerprint data identical to the hash fingerprint of the query filter condition. If the first storage position stores fingerprint data identical to the hash fingerprint of the query filter condition, it can be determined that the hash fingerprint of the query filter condition is stored in the index structure, and correspondingly, the target data is stored in the data file corresponding to the index file. At this time, the data file corresponding to the index file can be determined as a valid file. If the first storage position does not store fingerprint data identical to the hash fingerprint of the query filter condition, it can be determined that the hash fingerprint of the query filter condition is not stored in the index structure, and correspondingly, the target data is not stored in the data file corresponding to the index file. At this time, the data file corresponding to the index file can be determined as an invalid file, and the invalid file can be skipped when performing data query.

[0215] After performing the above operations on multiple index files, valid files can be finally filtered out from multiple data files.

[0216] Step S108: Query the target data corresponding to the query request from the valid files.

[0217] After valid files are determined from multiple data files, when performing data query, the valid files can be scanned and target data can be searched in the valid files.

[0218] In order to facilitate understanding of how to perform data query according to the index structure provided in the embodiment of the present application, a more specific implementation method may be used as follows. Fig.11 .

[0219] exist Fig.11 In the embodiment shown, when performing a data query, it is necessary to filter multiple data files to be queried to obtain valid files. Specifically, the following steps may be included:

[0220] Step 1: Parse and verify the SQL query statement in the query request and convert it into a logical plan.

[0221] Step 2: Obtain the filter conditions in the logical plan, optimize the logical plan based on the cost, and obtain the optimized logical plan.

[0222] Step 3: Generate and execute a physical plan based on the optimized logical plan.

[0223] Step 4: Filter multiple data files according to the index files corresponding to the data files.

[0224] When filtering multiple data files, the query filtering condition can be obtained first, the hash fingerprint of the query filtering condition can be determined according to the fingerprint hash function in the index structure, and then the storage location in the two-dimensional fingerprint matrix can be determined according to the mapping hash function in the index structure to determine whether the storage location stores fingerprint data identical to the hash fingerprint of the query filtering condition. If the same fingerprint data is stored in the storage location, the corresponding data file is determined as a valid file, otherwise the corresponding data file is determined as an invalid file and the invalid file is skipped.

[0225] Step 5: Generate an executable physical plan.

[0226] Step 6: Execute the physical plan and query the target data in the valid file.

[0227] Step 7: Return the result.

[0228] The specific implementation of the above steps 1 to 7 can refer to the specific implementation of the corresponding steps in the above steps S102 to S108, and will not be repeated here.

[0229] Next, a simple example is used to illustrate the data query method provided in the embodiment of the present application.

[0230] Assumptions Fig. 9 The index shown corresponds to part of the data in the data table T1, which is stored in the storage layer in the form of a parquet file. The data file is named T1-data-001.parquet, and T1 consists of multiple parquet data files. In order to optimize queries using the index structure proposed in the embodiment of the present application, it is first necessary to generate an index based on the data in the data file and write it to the storage layer. The index will construct corresponding index files for each data file. For the data file T1-data-001.parquet, an index will be generated and stored in the storage layer, assuming that its corresponding index file is named T1-idx-001.parquet.

[0231] When querying data, suppose the user enters the SQL command select * from T1 where T1.date = 20230801, that is, to query all data items in the data table T1 whose date data is equal to 20230801. Then, the logical plan can be optimized using the index written to the storage layer, and when the physical plan needs to scan the data file corresponding to table T1, the existing index is used first to achieve quick judgment.

[0232] Specifically, before scanning the data file T1-data-001.parquet, the index file T1-idx-001.parquet in the storage layer is first read. Then, the data fingerprint corresponding to the data 20230801 calculated using the fingerprint hash function is 106. Then, through two mapping hash functions, the buckets corresponding to the data fingerprint 106 are obtained as bucket [1] and bucket [2]. It is found that the same fingerprint data exists in bucket [1], that is, the corresponding data file has related data items. Therefore, the data file T1-data-001.parquet needs to be scanned when the physical plan is executed.

[0233] Assume that the user wants to query all data items in data table T1 whose data is equal to 20240309, and assume that the data fingerprint calculated by the hash function for 20240309 is 66, and the data fingerprints corresponding to buckets [1] and [2] are obtained through two mapping hash functions. However, buckets [1] and [2] do not have the same fingerprint data, that is, the index structure corresponding to the index file T1-idx-001.parquet does not contain data items whose date data is equal to 20240309. Therefore, the data file T1-data-001.parquet can be skipped directly.

[0234] Based on the data query method provided by the embodiment of the present application, when performing data query, the index structure provided by the embodiment of the present application can be used to query whether the target data exists in the data file, and then invalid files can be filtered out to obtain valid files, and the target data can be queried from the valid files, thereby narrowing the data scanning range and improving the data query efficiency. In addition, in the index structure provided by the embodiment of the present application, since the first hash component can determine the hash fingerprint of the data and determine the first storage location for storing the hash fingerprint based on the hash fingerprint, when performing data query based on the index structure, the first storage location can be determined based on the first hash component, and data query can be performed at the first storage location, thereby narrowing the query range in the index structure, realizing fast data query, and when performing data query from multiple data files, efficient data skipping can be supported, so that invalid files can be quickly filtered out to obtain valid files, further improving the query efficiency.

[0235] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0236] Fig.12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig.12 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0237] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0238] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0239] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for inserting data into the index structure at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0240] Determine a hash fingerprint of the data to be inserted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0241] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0242] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining a second storage location in the one-dimensional fingerprint vector according to the third hash function, and recording the number of repetitions of the hash fingerprint in the second storage location;

[0243] In a case where the first storage location does not store fingerprint data identical to the hash fingerprint, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0244] The above application Fig.12The method performed by the device for inserting data in the index structure disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0245] The electronic device may also perform Figure 5 A method for deleting data in the index structure is implemented in Figure 5 The functions in the illustrated embodiments will not be described in detail in this application.

[0246] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0247] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 5 The method of the embodiment shown is specifically used to perform the following operations:

[0248] Determine a hash fingerprint of the data to be inserted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0249] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0250] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining a second storage location in the one-dimensional fingerprint vector according to the third hash function, and recording the number of repetitions of the hash fingerprint in the second storage location;

[0251] In a case where the first storage location does not store fingerprint data identical to the hash fingerprint, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

[0252] Fig.13 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig.13 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0253] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0254] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0255] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for deleting data in the index structure at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0256] Determine a hash fingerprint of the data to be deleted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0257] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0258] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining whether the hash fingerprint is duplicate data according to the record in the first storage location;

[0259] In a case where the hash fingerprint is not duplicate data, deleting the hash fingerprint in the first storage location;

[0260] In the case that the hash fingerprint is repeated data, a second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions recorded in the second storage location is reduced by one.

[0261] The above application Fig.13The method performed by the device for deleting data in the index structure disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0262] The electronic device may also perform Figure 8 A method for deleting data in the index structure is implemented in Figure 8 The functions in the illustrated embodiments will not be described in detail in this application.

[0263] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0264] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 8 The method of the embodiment shown is specifically used to perform the following operations:

[0265] Determine a hash fingerprint of the data to be deleted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0266] determining whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0267] In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining whether the hash fingerprint is duplicate data according to the record in the first storage location;

[0268] In a case where the hash fingerprint is not duplicate data, deleting the hash fingerprint in the first storage location;

[0269] In the case that the hash fingerprint is repeated data, a second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions recorded in the second storage location is reduced by one.

[0270] Fig.14 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig.14 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0271] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0272] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0273] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data query device based on the index structure at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0274] receiving query requests;

[0275] Acquire multiple index files corresponding to multiple data files, each data file corresponds to an index file, and each index file includes the index structure;

[0276] Determining a valid file from the multiple data files according to the query request and the multiple index files;

[0277] The target data corresponding to the query request is searched from the valid file.

[0278] The above application Fig.14 The method performed by the data query device based on the index structure disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0279] The electronic device may also perform Fig.10 and Fig.11The method is to realize a data query device based on an index structure. Fig.10 and Fig.11 The functions in the illustrated embodiments will not be described in detail in this application.

[0280] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0281] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Fig.10 and Fig.11 The method of the embodiment shown is specifically used to perform the following operations:

[0282] receiving a query request;

[0283] Acquire multiple index files corresponding to multiple data files, each data file corresponds to an index file, and each index file includes the index structure;

[0284] Determining a valid file from the multiple data files according to the query request and the multiple index files;

[0285] The target data corresponding to the query request is searched from the valid file.

[0286] Fig.15 This is a schematic diagram of the structure of a device 150 for inserting data into an index structure according to an embodiment of the present application. Fig.15 In a software implementation, the device 150 for inserting data into the index structure may include: a first determination module 151, a second determination module 152, a first insertion module 153, and a second insertion module 154, wherein:

[0287] A first determining module 151 determines a hash fingerprint of the data to be inserted according to the first hash function and determines a first storage position in the two-dimensional fingerprint matrix according to the second hash function;

[0288] A second determination module 152 determines whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0289] A first inserting module 153, when the first storage location stores fingerprint data identical to the hash fingerprint, determines a second storage location in the one-dimensional fingerprint vector according to the third hash function, and records the number of repetitions of the hash fingerprint in the second storage location;

[0290] The second insertion module 154 stores the hash fingerprint in the first storage location and records that the hash fingerprint is not duplicate data when the first storage location does not store fingerprint data identical to the hash fingerprint.

[0291] The device 150 for inserting data into the index structure provided by the present application can also execute Figure 5 The method and the device 150 for inserting data into the index structure are implemented in Figure 5 The functions of the illustrated embodiments will not be described in detail in this application.

[0292] Fig.16 This is a schematic diagram of the structure of a device 160 for deleting data in an index structure according to an embodiment of the present application. Fig.16 In a software implementation, the device 160 for deleting data in the index structure may include: a first determination module 161, a second determination module 162, a third determination module 163, a first insertion module 164, and a second insertion module 165, wherein:

[0293] A first determining module 161 determines a hash fingerprint of the data to be deleted according to the first hash function and determines a first storage location in the two-dimensional fingerprint matrix according to the second hash function;

[0294] A second determination module 162 determines whether the first storage location stores fingerprint data identical to the hash fingerprint;

[0295] A third determination module 163, when the first storage location stores fingerprint data identical to the hash fingerprint, determines whether the hash fingerprint is duplicate data according to the record in the first storage location;

[0296] A first inserting module 164, deleting the hash fingerprint in the first storage location if the hash fingerprint is not duplicate data;

[0297] The second inserting module 165 determines a second storage location in the one-dimensional fingerprint vector according to the third hash function when the hash fingerprint is repeated data, and reduces the number of repetitions recorded in the second storage location by one.

[0298] The device 160 for deleting data in the index structure provided in the present application can also execute Figure 8The method and the device 160 for deleting data in the index structure are implemented in Figure 8 The functions of the illustrated embodiments will not be described in detail in this application.

[0299] Fig.17 This is a schematic diagram of the structure of a data query device 170 based on an index structure according to an embodiment of the present application. Fig.17 In a software implementation, the index structure-based data query device 170 may include: a receiving module 171, an acquisition module 172, a determination module 173 and a query module 174, wherein:

[0300] A receiving module 171 receives a query request;

[0301] An acquisition module 172 is configured to acquire a plurality of index files corresponding to a plurality of data files, wherein each data file corresponds to an index file, and each index file includes the index structure;

[0302] A determination module 173, determining a valid file from the multiple data files according to the query request and the multiple index files;

[0303] The query module 174 queries the target data corresponding to the query request from the valid file.

[0304] The data query device 170 based on the index structure provided in the present application can also execute Fig.10 and Fig.11 The method is implemented by the data query device 170 based on the index structure. Fig.10 and Fig.11 The functions of the illustrated embodiments will not be described in detail in this application.

[0305] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be operated to cause a computer to execute some or all of the steps in the above-mentioned method embodiment for inserting data in an index structure, or execute some or all of the steps in the above-mentioned method embodiment for deleting data in an index structure, or execute some or all of the steps in the above-mentioned data query method embodiment based on an index structure.

[0306] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0307] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0308] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0309] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0310] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. An index structure, comprising a first hash component and a second hash component, wherein: The first hash component includes a first hash function, a second hash function and a two-dimensional fingerprint matrix, the first hash function is used to determine a hash fingerprint of data, the second hash function is used to determine a first storage location in the two-dimensional fingerprint matrix according to the hash fingerprint, and the first storage location is used to store the hash fingerprint and record whether the hash fingerprint is duplicate data; The second hash component includes a third hash function and a one-dimensional fingerprint vector, the third hash function is used to determine a second storage location in the one-dimensional fingerprint vector according to the hash fingerprint, and the second storage location is used to record the number of repetitions of the hash fingerprint.

2. The index structure as described in claim 1, wherein the second hash function is used to determine multiple row indexes of the two-dimensional fingerprint matrix, and the first storage location is a storage location in the row corresponding to the multiple row indexes.

3. In the index structure as described in claim 2, the second hash function includes a first mapping hash function and a second mapping hash function, the first mapping hash function is used to determine a first row index according to the hash fingerprint and the number of rows of the two-dimensional fingerprint matrix, and the second mapping hash function is used to determine a second row index according to the hash fingerprint and the first row index.

4. The index structure according to claim 1, wherein the first storage location is composed of a data bit and a flag bit, the data bit is used to store the hash fingerprint, and the flag bit is used to record whether the hash fingerprint is duplicate data; in, The flag bit value of 0 indicates that the hash fingerprint is not duplicate data, and the flag bit value of 1 indicates that the hash fingerprint is duplicate data.

5. The index structure of claim 1, wherein the third hash function comprises a modulo function; The modulo function is used to determine a remainder of dividing the hash fingerprint by the length of the one-dimensional fingerprint vector, and the remainder is used to indicate the second storage location.

6. The index structure as claimed in claim 1, wherein the second hash component further comprises a fourth hash function; The fourth Hash function is used to determine a fingerprint identifier of the hash fingerprint, and the second storage location is also used to store the fingerprint identifier.

7. The index structure of claim 6, wherein the fourth Hash function comprises a quotient function; The quotient function is used to determine the quotient of the hash fingerprint and the length of the one-dimensional fingerprint vector, and the fingerprint identifier includes the quotient.

8. The index structure as described in claim 6, wherein the second storage location is composed of a data bit and a counting bit, the data bit is used to store the fingerprint identifier, and the counting bit is used to record the number of repetitions of the hash fingerprint.

9. The index structure according to any one of claims 1 to 8, wherein the data type includes but is not limited to integer, floating point, character, and date types.

10. A method for inserting data into an index structure, the index structure comprising the index structure according to any one of claims 1 to 9, the method comprising: Determine a hash fingerprint of the data to be inserted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function; determining whether the first storage location stores fingerprint data identical to the hash fingerprint; In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining a second storage location in the one-dimensional fingerprint vector according to the third hash function, and recording the number of repetitions of the hash fingerprint in the second storage location; In a case where the first storage location does not store fingerprint data identical to the hash fingerprint, the hash fingerprint is stored in the first storage location and it is recorded that the hash fingerprint is not duplicate data.

11. The method according to claim 10, wherein recording the number of repetitions of the hash fingerprint in the second storage location comprises: determining whether the first storage location has recorded the hash fingerprint as duplicate data; In the case where the hash fingerprint is not recorded as duplicate data in the first storage location, the hash fingerprint is recorded as duplicate data in the first storage location and the number of repetitions of the hash fingerprint recorded in the second storage location is 2; In the case that the hash fingerprint has been recorded as repeated data in the first storage location, the number of repetitions recorded in the second storage location is increased by one.

12. The method according to claim 11, wherein the second hash component further comprises a fourth hash function; and when the number of repetitions of the hash fingerprint recorded in the second storage location is 2, the method further comprises: Determine a fingerprint identifier of the hash fingerprint according to the fourth Hash function; The fingerprint identifier is stored in the second storage location.

13. The method of claim 10, wherein the second hash function is used to determine a plurality of row indexes of the two-dimensional fingerprint matrix, and the first storage location is a storage location in a row corresponding to the plurality of row indexes; Storing the hash fingerprint in the first storage location and recording that the hash fingerprint is not duplicate data includes: Determine a row with the smallest load from a plurality of rows corresponding to a plurality of row indexes; The hash fingerprint is stored in the row with the smallest load, and it is recorded that the hash fingerprint is not duplicate data.

14. A method for deleting data in an index structure, the index structure comprising the index structure according to any one of claims 1 to 9, the method comprising: Determine a hash fingerprint of the data to be deleted according to the first hash function and determine a first storage location in the two-dimensional fingerprint matrix according to the second hash function; Determining whether the first storage location stores fingerprint data identical to the hash fingerprint; In the case where the first storage location stores fingerprint data identical to the hash fingerprint, determining whether the hash fingerprint is duplicate data according to the record in the first storage location; In a case where the hash fingerprint is not duplicate data, deleting the hash fingerprint in the first storage location; In the case that the hash fingerprint is repeated data, a second storage location in the one-dimensional fingerprint vector is determined according to the third hash function, and the number of repetitions recorded in the second storage location is reduced by one.

15. The method of claim 14, after reducing the number of repetitions recorded in the second storage location by one, the method further comprises: Determine whether the number of repetitions is 1; When the number of repetitions is 1, determining whether a deletion condition is met; When the deletion condition is met, the content stored in the second storage location is deleted and the hash fingerprint is recorded in the first storage location as not being duplicate data.

16. A data query method based on an index structure, the index structure comprising the index structure according to any one of claims 1 to 9, the method comprising: receiving query requests; Acquire multiple index files corresponding to multiple data files, each data file corresponds to an index file, and each index file includes the index structure; Determining a valid file from the multiple data files according to the query request and the multiple index files; The target data corresponding to the query request is searched from the valid file.

17. The method of claim 16, wherein determining a valid file from the plurality of data files according to the query request and the plurality of index files comprises: Parsing and verifying the query request to determine a logical plan corresponding to the query request; Converting the logical plan into a physical execution plan, wherein the physical execution plan includes a query filtering condition; The multiple index files are queried according to the query filter condition, and valid files are determined from the multiple data files according to the query result.

18. The method according to claim 17, wherein querying the plurality of index files according to the query filter condition and determining a valid file from the plurality of data files according to the query result comprises: For each index file, do the following: Determine a hash fingerprint of the query filter condition according to a first hash function in the index structure and determine a first storage location in a two-dimensional fingerprint matrix according to a second hash function in the index structure; Determining whether the first storage location stores fingerprint data identical to the hash fingerprint; In a case where fingerprint data identical to the hash fingerprint is stored in the first storage location, determining the data file corresponding to the index file as a valid file; In a case where the first storage location does not store fingerprint data identical to the hash fingerprint, the data file corresponding to the index file is determined to be an invalid file.

19. The method according to any one of claims 16 to 18, wherein for each data file, the index file corresponding to the data file is determined by: Obtaining data from the data file and initializing the index structure; Updating the index structure according to the data in the data file, wherein the updating includes inserting data or deleting data in the index structure; The index structure is stored in a specified format to obtain an index file corresponding to the data file.

20. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 10 to 19.

21. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 10 to 19.

22. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method according to any one of claims 10 to 19.

Images