Dynamic k-mer data set-oriented approximate member query method and system

Abstract

The invention relates to the technical field of data storage, and particularly provides a dynamic k-mer data set-oriented approximate member query method and system. The method comprises the following steps: calculating an element cardinal number of a target data set needing to be stored; selecting a target first unit needing to be stored according to the element cardinal number; generating a second unit index of the target element according to the target element of the target data set through the grouping function of the target first unit, and dividing the target element into a target second unit corresponding to the second unit index; and storing the target element into a target third unit in the target second unit, and distributing the target element to a Bloom filter of the target data set. Therefore, the technical problem that the use requirement of the dynamically stored gene sequence is difficult to meet due to the fact that the storage architecture and the query mode of the gene sequence in the related technology cannot give consideration to the space efficiency and the query speed or do not support incremental updating and deletion is solved.

Description

Technical Field

[0001] The invention relates to the technical field of gene data storage, and in particular to an approximate member query method and system for a dynamic k-mer data set. Background Art

[0002] The datasets in genomic research change frequently due to the release of new sequences, the generation of new experimental data, and the revision of existing data. Such frequently changing datasets are called dynamic datasets. Faced with the increasingly large scale of data sets, the paradigm of storing and indexing datasets as k-length gene sequence substrings (k-mer) datasets has become increasingly prominent.

[0003] This type of k-mer data search problem can be abstracted as a multi-set multi-member query problem: the database storing various DNA sequences is abstracted into multiple data sets. For a DNA sequence of interest, the goal is to detect in which sets the sequence exists.

[0004] Although some methods in related technologies have made significant progress in dealing with sequence search problems in large-scale genomic data, such as setting up a Bloom filter for each data set to form a Bloom filter matrix, this method wastes a lot of space. Other methods have limitations when dealing with dynamic data sets, and cannot take into account both space efficiency and query speed, or do not support incremental updates and deletions, making it difficult to meet the needs of dynamically stored gene sequences.

[0005] Regarding the storage architecture and query method of gene sequences in related technologies, in the case of dynamically stored data sets, it is impossible to take into account both space efficiency and query speed, or it does not support incremental updates and deletions, resulting in difficulty in meeting the usage requirements of dynamically stored gene sequences. No effective solution has been proposed yet. Summary of the invention

[0006] The present invention provides an approximate member query method and system for a dynamic k-mer data set, which at least solves the problem that the storage architecture and query method of gene sequences in the related art cannot take into account both space efficiency and query speed in the case of dynamically stored data sets, or does not support incremental updates and deletions, making it difficult to meet the usage requirements of dynamically stored gene sequences.

[0007] According to one aspect of the invention, a method for generating an approximate member query index for a dynamic k-mer data set is provided, comprising: determining the element cardinality of a target data set to be stored according to a cardinality estimation algorithm; selecting a first-level target first unit to be stored according to the element cardinality, wherein the number of Bloom filters of the multiple first units of the first level is different, the capacity of the Bloom filters of the first units is the same, the Bloom filters are used to store the corresponding elements in the target data set, and the element cardinality that can be stored in the first unit is not less than the element cardinality of the corresponding data set; generating the target element according to the target element of the target data set through a grouping function of the target first unit. The target element is divided into the target second unit corresponding to the second unit index according to the second unit index of the second level of the element, wherein the first unit includes multiple second units, and the number of the second units of each data set stored in the first unit is the same as the number of Bloom filters required by the corresponding data set; the target element is stored in the target third unit of the third level in the target second unit, and is assigned to the Bloom filter of the target data set, wherein the second unit includes multiple third units of fixed size, and the organization form of the third unit is an interleaved Bloom filter, and the third unit reads the same bit of different Bloom filters at one time according to the interleaved Bloom filter when querying the element.

[0008] As an optional embodiment, a target first unit of the first level to be stored is selected according to the cardinality, including: comparing the element cardinality of the target data set with the element cardinality that can be stored by the first unit; when the element cardinality of the target data set is less than or equal to the element cardinality that can be stored by the comparison first unit, and greater than the element cardinality that can be stored by the previous first unit, using the comparison first unit as the target first unit; wherein the comparison first unit is one of multiple first units, and the previous first unit is a first unit whose number of Bloom filters is less than that of the comparison first unit by a preset difference.

[0009] As an optional embodiment, the target element is stored in a target third unit of a third level in the target second unit and assigned to a Bloom filter of the target data set, including: storing the target element in a target third unit of a third level in the target second unit; using multiple independent hash functions of the target third unit to hash the target element and map it to a corresponding position in the corresponding Bloom filter, and setting the corresponding position to 1.

[0010] As an optional embodiment, the method also includes: receiving a first data set to be inserted; determining a corresponding first unit to be inserted according to the element cardinality of the first data set; converting the organizational form of the third unit inserted into the first unit from an interleaved Bloom filter to a Bloom filter matrix; searching for a corresponding insertion position in the inserted first unit, wherein the insertion position is found through a Bloom filter; inserting the first data set into the insertion position, and assigning a corresponding Bloom filter to the first data set; converting the organizational form of the third unit inserted into the first unit from a Bloom filter matrix to an interleaved Bloom filter.

[0011] As an optional embodiment, the method also includes: receiving a second data set to be deleted; querying the first unit corresponding to the second data set; converting the organization form of the third unit of the first unit from an interleaved Bloom filter to a Bloom filter matrix; when the element set of the second data set is a partial element set of the third unit, deleting the Bloom filter corresponding to the second data set and reserving empty spaces; when the element set of the second data set is a complete element set of the third unit, deleting the Bloom filter corresponding to the second data set and releasing the space resources of the third unit; converting the organization form of the third unit of the first unit from a Bloom filter matrix to an interleaved Bloom filter.

[0012] According to another aspect of the invention, there is provided an approximate member query method for a dynamic k-mer data set, characterized in that it includes: generating a second unit index of the second level of the target element in the first unit through a grouping function of the first unit of the first level and a target element to be queried, so as to identify the target second unit where the target element is located, wherein the target element is indexed level by level in a multi-level manner, and when stored in the second level, the corresponding second unit index is generated using the grouping function of the first unit to which it belongs, and stored in the target second unit corresponding to the second unit index; when the target second unit exists in the first unit, Determine that the target element is stored in the target second unit of the first unit; obtain the interleaved Bloom filter of the third unit of the third level in the target second unit, and obtain multiple hash indexes based on the interleaved Bloom filter, wherein the third unit is a component of the second unit to which it belongs, and when the target element is stored in the third unit, it will be assigned to the Bloom filter of the target data set to which it belongs, and the organization form of the third unit is an interleaved Bloom filter, the number of bits of the hash index is the same as the number of Bloom filters of the target second unit, and the Bloom filters correspond to different data sets; query the target data set to which the target element belongs based on the hash index.

[0013] As an optional embodiment, querying the target data set to which the target element belongs according to the hash index includes: performing a bitwise AND operation on multiple hash indexes to obtain a target index, wherein the hash index is data of the same bit of different Bloom filters, the hash index is a mapping of the same hash function to elements of different data sets, different bits of the hash index represent numerical values ​​of different Bloom filters, and the numerical values ​​represent whether the target element exists in the Bloom filter; determining the target data set to which the target element belongs according to the target index.

[0014] As an optional embodiment, after querying the target data set to which the target element belongs according to the hash index, the method further includes: storing the target data set in an output list; continuing to query whether the index of the target element exists in other first units; if the index of the target element exists in the first unit, obtaining the target data set to which it belongs and storing it in the output list; and outputting the output list when all first units have been traversed.

[0015] As an optional embodiment, the first unit of the first level is a segment unit, the segment unit includes multiple Bloom filters, the capacity of the Bloom filters of each segment unit is the same, one Bloom filter corresponds to recording the corresponding element in a data set, and the element cardinality of the data set that can be stored in the segment unit is greater than or equal to the element cardinality of the data set that needs to be stored; the second unit of the second level is a group unit, and the number of group units for each data set stored in the segment unit is the same as the number of Bloom filters required for the corresponding data set; the third unit of the third level is a block unit, the block unit is a data block with a fixed data size, is a component unit of the group unit, and stores different elements in the organizational form of interleaved Bloom filters. When querying an element, the block unit reads all data at the position corresponding to the query element in different Bloom filters at one time according to the interleaved Bloom filters.

[0016] According to another aspect of the invention, there is provided an approximate membership query system for a dynamic k-mer data set, comprising: a processor, and a memory for storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the above method.

[0017] The index generation method provided by the embodiment of the present invention selects the target first unit of the first level that needs to be stored according to the element cardinality, and uses the first unit containing a fixed number of Bloom filters to store the data set with the element cardinality corresponding to the fixed number, so that sufficient and as small a space as possible can be allocated to each data set to avoid space waste.

[0018] Through the grouping function of the target first unit, the second unit index of the second level of the target element is generated according to the target element of the target data set, and the target element is divided into the target second unit corresponding to the second unit index. The number of second units stored in each data set is the same as the number of Bloom filters required by the corresponding data set. Each second unit stores a different set of elements of the data set, so when querying, only one Bloom filter needs to be queried for an element to determine whether the element is in the second unit and the first unit to which it belongs.

[0019] The target element is stored in the target third unit of the third level in the target second unit, and is assigned to the Bloom filter of the target data set. By using the interleaved Bloom filter of the third unit, the same bit of different Bloom filters can be read at one time in the third unit according to the interleaved Bloom filter when querying the element, so as to determine the target data cluster where the query element is located at one time, avoiding the use of the Bloom filter matrix which can only read the data of one Bloom filter at a time, and needs to traverse the Bloom filter to determine the data set to which the query element belongs.

[0020] This solves the problem of the storage architecture and query method of gene sequences in related technologies. In the case of dynamically stored data sets, it is impossible to take into account both space efficiency and query speed, or it does not support incremental updates and deletions, which makes it difficult to meet the usage needs of dynamically stored gene sequences. This achieves the technical effect of supporting incremental updates and deletions to meet data usage needs while ensuring that the space occupied is as small as possible and the number of queries is as small as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0022] Figure 1 The present invention is a flowchart of a method for generating an approximate member query index for a dynamic k-mer data set according to an embodiment of the present invention.

[0023] Figure 2 The present invention is a flowchart of an approximate member query method for a dynamic k-mer data set according to an embodiment of the present invention.

[0024] Figure 3 It is a schematic diagram of the index hierarchical architecture of an implementation method created by the present invention.

[0025] Figure 4It is a schematic diagram of the index construction process of data in the embodiment mode created by the present invention.

[0026] Figure 5 It is a schematic diagram of the element query process of data indexing in the embodiment mode created by the present invention.

[0027] Figure 6 It is a schematic diagram of a data set adding process using a data index according to an embodiment of the present invention.

[0028] Figure 7 It is a schematic diagram of a data set deletion process using a data index according to an embodiment of the present invention.

[0029] Figure 8 It is a schematic diagram of an approximate member query index generation device for a dynamic k-mer data set according to an embodiment of the present invention.

[0030] Fig. 9 It is a schematic diagram of an approximate member query device for a dynamic k-mer data set according to an embodiment of the present invention.

[0031] Fig.10 It is a structural schematic diagram of an approximate member query system for dynamic k-mer data sets created by the present invention. DETAILED DESCRIPTION

[0032] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0033] Approximate Membership Queries (AMQ) uses a compact space to represent a set and answer whether the query element belongs to the set. It has high space efficiency and query efficiency and allows for certain false positives. Bloom Filter is the most classic approximate membership query data structure. It maintains a bit array of length m, maps elements to several positions in the bit array through multiple hash functions, and sets the bits at these positions to "1" to mark the existence of the element. Bloom filter supports fast insertion and search, but cannot delete elements.

[0034] Approximate member queries have a wide range of applications in multiple fields, including genome sequence searches. With the development of genome sequencing technology, especially the application of next-generation sequencing (NGS) technology, sequencing throughput has been greatly improved, driving the rapid growth of data generated by biotechnology protocols such as WGS and RNA-Seq. These data are stored in public databases such as the European Nucleotide Archive (ENA) and the Sequence Read Archive (SRA). However, the exponential growth of data size makes it extremely challenging to efficiently search for DNA sequences in them. In order to make full use of existing resources, it is crucial to achieve fast and efficient genome sequence searches.

[0035] The following three strategies are mainly used in related technologies for sequence search:

[0036] 1) The classic approach is to assign a Bloom filter to each set and organize it into a Bloom filter matrix. This structure is more flexible, and the addition and deletion of data sets can be achieved by creating or destroying Bloom filters. However, in order to maintain the expected false alarm rate, the Bloom filter size needs to be set according to the maximum set, which causes serious space waste for small sets.

[0037] 2) By using partitioned hash functions to divide the set into multiple regions, each region is assigned an approximate member query structure. Compared with the matrix structure, this method reduces the number of Bloom filters accessed during the query, which can speed up the query when the data scale is small. However, since Bloom filters do not support deletion, the method of storing multiple set elements in a mixed manner does not support the deletion of data sets, otherwise it will cause false negatives.

[0038] 3) Organize the Bloom filter into a hierarchical tree structure. At each level, large sets are split and small sets are merged to deal with the unbalanced set size distribution, thereby alleviating space waste to a certain extent. However, the tree structure complicates the query process and there is a problem of index reconstruction when inserting and deleting data sets.

[0039] In order to solve the problem that the above-mentioned existing approximate member query structures for genome sequence search cannot handle dynamic data sets well, this embodiment aims to design an indexing and query method for dynamically stored data sets, which supports incremental update and deletion of data sets while maintaining good space efficiency and query speed.

[0040] Figure 1 : is a flowchart of a method for generating an approximate member query index for a dynamic k-mer data set according to an embodiment of the present invention. Figure 1 As shown, the embodiment of the invention provides a method for generating an approximate member query index for a dynamic k-mer data set. The steps of the method are as follows:

[0041] Step S101, determining the element cardinality of the target data set to be stored according to a cardinality estimation algorithm;

[0042] Step S102, selecting a target first unit of the first level to be stored according to the cardinality of elements, wherein the number of Bloom filters of the multiple first units of the first level is different, the capacity of the Bloom filters of the first units are the same, the Bloom filters are used to store the corresponding elements in the target data set, and the cardinality of elements that can be stored in the first unit is not less than the cardinality of elements of the corresponding data set;

[0043] Step S103, generating a second unit index of a second level of the target element according to the target element of the target data set through the grouping function of the target first unit, and dividing the target element into the target second unit corresponding to the second unit index, wherein the first unit includes a plurality of second units, and the number of second units of each data set stored in the first unit is the same as the number of Bloom filters required by the corresponding data set;

[0044] Step S104, stores the target element in the target third unit of the third level in the target second unit, and assigns it to the Bloom filter of the target data set, wherein the second unit includes multiple third units of fixed size, and the third unit is organized in the form of an interleaved Bloom filter. When querying an element, the third unit reads the same bit of different Bloom filters at one time according to the interleaved Bloom filter.

[0045] The index generation method for the above-mentioned dynamically stored data set provided by the embodiment created by the present invention selects the target first unit of the first level that needs to be stored according to the cardinality of elements, and uses the first unit containing a fixed number of Bloom filters to store the data set with the cardinality of elements corresponding to the fixed number, so that sufficient and as small as possible space can be allocated to each data set to avoid space waste.

[0046] Through the grouping function of the target first unit, the second unit index of the second level of the target element is generated according to the target element of the target data set, and the target element is divided into the target second unit corresponding to the second unit index. The number of second units stored in each data set is the same as the number of Bloom filters required by the corresponding data set. Each second unit stores a different set of elements of the data set, so when querying, only one Bloom filter needs to be queried for an element to determine whether the element is in the second unit and the first unit to which it belongs.

[0047] The target element is stored in the target third unit of the third level in the target second unit, and is assigned to the Bloom filter of the target data set. By using the interleaved Bloom filter of the third unit, the same bit of different Bloom filters can be read at one time in the third unit according to the interleaved Bloom filter when querying the element, so as to determine the target data cluster where the query element is located at one time, avoiding the use of the Bloom filter matrix which can only read the data of one Bloom filter at a time, and needs to traverse the Bloom filter to determine the data set to which the query element belongs.

[0048] This achieves the technical effect of supporting incremental updates and deletions to meet data usage requirements while ensuring that the space occupied is as small as possible and the number of queries is as small as possible.

[0049] The data set may be a k-mer data set, and the target data set may be a target k-mer data set.

[0050] The execution subject of the above steps may be a storage device for k-mer data, and the storage device may include a processor or a processing chip for performing data processing to realize different storage, reading, query and other functions.

[0051] In the above step S101, the k-mer data set may be a set containing multiple k-mer data, and the k-mer data may be understood as a gene sequence composed of k bases. For example, CGAT may be simply 4-mer data. The above cardinality may be understood as the number of possible classifications after the elements in the k-mer data set are classified.

[0052] The cardinality estimation algorithm may be a HyperLogLog algorithm, and the cardinality of elements of a target k-mer data set to be stored may be determined according to the cardinality estimation algorithm.

[0053] Since in the index architecture, the first level includes a plurality of first units, the difference between different first units is that the number of Bloom filters of each first unit is different, and the capacity of the Bloom filters of the same first unit is the same.

[0054] The first unit can store multiple k-mer data sets, and each stored k-mer data set is allocated with a number of Bloom filters corresponding to the cardinality of the elements, and is allocated with a number of second units corresponding to the cardinality of the elements.

[0055] The main purpose of adopting such an architecture is to assign the element cardinality of different k-mer datasets to the first unit of the corresponding element cardinality, while ensuring the index record of the k-mer dataset and occupying as little space as possible.

[0056] In the above step S102, the target first unit of the first level to be stored is selected according to the element cardinality, that is, the first unit whose element cardinality of the storable data set is not less than the element cardinality of the corresponding k-mer data set needs to be selected as the first unit to be stored.

[0057] That is, a Bloom filter of the first unit of the first level corresponds to the element index of a k-mer data set, and the element cardinality of the data set that the first unit can store is not less than the element cardinality of the corresponding k-mer data set to at least meet the element storage requirements of the k-mer data set.

[0058] However, the cardinality of the elements of the storable dataset of the first unit cannot exceed the cardinality of the elements of the k-mer dataset by too much, otherwise it will easily cause a waste of space.

[0059] Preferably, in this embodiment, the element cardinality of the storable data set of the first unit is equal to the element cardinality of the corresponding k-mer data set, which can ensure that the element index of the k-mer data set is effectively stored while reducing the memory space occupied as much as possible, thereby improving space utilization.

[0060] In the above step S103, the target first unit is the first unit that meets the element cardinality requirement of the k-mer dataset. Each first unit includes a second unit of the second level; the number of the second units of each k-mer dataset stored in the first unit is the same as the number of Bloom filters required by the corresponding k-mer dataset, so that different element sets of different k-mer datasets can be stored respectively by the same number of second units.

[0061] With such a storage structure, when querying, only one Bloom filter needs to be queried for a query element to determine whether the element is in the second unit and the first unit to which it belongs.

[0062] Specifically, through the grouping function of the target first unit, according to the target element of the target k-mer data set, a second unit index of the second level of the target element is generated; this second unit index can establish a one-to-one mapping between its target element and the second unit.

[0063] When querying an element, it is only necessary to directly generate the corresponding second unit index for the query tuple through the grouping function corresponding to the first unit of the query. By querying whether the second unit index exists in the first unit of the query, it is possible to determine whether the second unit to which the query element belongs exists in the first unit. When it is determined that the second unit to which the query element belongs exists in the first unit, it is possible to determine which second unit the second unit is.

[0064] When the index is generated, the target element is divided into the target second unit corresponding to the second unit index. Specifically, as in step S104 above, the target element is stored in the target third unit of the third level in the target second unit and assigned to the Bloom filter of the target k-mer data set.

[0065] The second unit includes multiple third units of fixed size, such as 64 bits. In addition, the third unit can be created and deleted according to demand.

[0066] In the related art, the data block is generally organized in the form of a Bloom filter matrix, that is, different Bloom filters are arranged in a matrix form, and a read operation can only read all the data of the same Bloom filter. However, different positions of the same Bloom filter correspond to whether different elements exist. In this way, if you want to query whether a certain element exists in the data block, you need to traverse multiple Bloom filters corresponding to the data block.

[0067] And only after the final traversal is completed can it be determined that the data block does not store the element. As the name suggests, the interleaved Bloom filter is to stagger the different positions of different Bloom filters. Arrange different Bloom filters according to their positions, so that one read operation can read the same position of the corresponding query element in different Bloom filters, and then determine whether the query element exists in the third unit through one read operation, and which specific Bloom filter the query element belongs to, and determine the k-mer data set to which it belongs according to the Bloom filter to which it belongs.

[0068] Therefore, in this embodiment, the third unit is organized in the form of an interleaved Bloom filter. When querying an element, the third unit reads the same bit of different Bloom filters at one time according to the interleaved Bloom filter.

[0069] In summary, through the multi-layer architecture and characteristics of the first unit, the second unit, and the third unit, it is possible to ensure a smaller footprint during storage, higher efficiency, and better accuracy during query. Moreover, the entire architecture can be flexibly adjusted, independent of the initial data set, to create indexes, and to add and delete corresponding data sets as needed. The specific methods of adding and deleting will be described later.

[0070] As an optional embodiment, a target first unit of the first level to be stored is selected according to the cardinality, including: comparing the element cardinality of the target k-mer data set with the element cardinality that can be stored in the first unit; when the element cardinality of the target k-mer data set is less than or equal to the element cardinality that can be stored in the comparison first unit, and greater than the element cardinality that can be stored in the previous first unit, the comparison first unit is used as the target first unit; wherein the comparison first unit is one of multiple first units, and the previous first unit is a first unit whose number of Bloom filters is less than that of the comparison first unit by a preset difference.

[0071] In the first unit of the first level, ideally, each k-mer data set that needs to be indexed can find the first unit with the same cardinality of elements, but this requires a large number of first units, and the difference in the number of Bloom filters between the first units with the closest Bloom filters is only 1. This may require a very large span in the number of first units, which is not conducive to space allocation.

[0072] Considering that the number of elements in different k-mer datasets varies greatly, the number of Bloom filters between adjacent first units can be increased. In this way, when selecting the first target unit of the first level to be stored based on the cardinality, the cardinality of the elements of the k-mer dataset that each first unit satisfies is often a numerical range.

[0073] In this way, when selecting the target first unit of the first level to be stored according to the cardinality, it is often difficult to find a first unit with the same number of Bloom filters. When the cardinality of the elements in the target k-mer dataset is less than or equal to the cardinality of the elements that can be stored in the comparison first unit and greater than the cardinality of the elements that can be stored in the previous first unit, the comparison first unit can be used as the target first unit.

[0074] Therefore, the index storage space can be reduced as much as possible while the number of first units is kept as small as possible.

[0075] As an optional embodiment, the target element is stored in the target third unit of the third level in the target second unit and assigned to the Bloom filter of the target k-mer data set, including: storing the target element in the target third unit of the third level in the target second unit; using multiple independent hash functions of the target third unit to hash the target element and map it to the corresponding position in the corresponding Bloom filter, and setting the corresponding position to 1.

[0076] The target element is stored in the Bloom filter assigned to the k-mer dataset in the second unit of the target. The specific storage method is to use k independent hash functions h0,…h k-1, each function randomly and uniformly maps the target element in the k-mer dataset to an integer in {0, ..., m-1}, that is, any integer in the size m of the Bloom filter.

[0077] And set the corresponding k positions mapped in this Bloom filter to 1, that is, B[h0(x)],…, B[h k-1 (x)] is set to 1.

[0078] As an optional embodiment, the method also includes: receiving a first k-mer data set to be inserted; determining a corresponding first unit to be inserted according to the element cardinality of the first k-mer data set; converting the organizational form of the third unit inserted into the first unit from an interleaved Bloom filter to a Bloom filter matrix; searching for a corresponding insertion position in the inserted first unit, wherein the insertion position is found through a Bloom filter; inserting the first k-mer data set into the insertion position, and assigning a corresponding Bloom filter to the first k-mer data set; converting the organizational form of the third unit inserted into the first unit from the Bloom filter matrix to an interleaved Bloom filter.

[0079] When inserting the first k-mer data set, the corresponding first unit to be inserted can be determined according to the element cardinality of the first k-mer data set. After the first unit is determined, the organization form of the third unit in the first unit is converted from an interleaved Bloom filter to a Bloom filter matrix.

[0080] In this way, during the insertion operation, a single insertion operation can be performed on the entire Bloom filter, without having to insert the Bloom filters that need to be inserted multiple times based on the position.

[0081] After the insertion, the organization form of the third unit inserted into the first unit is converted from a Bloom filter matrix to an interleaved Bloom filter, which can facilitate subsequent element queries.

[0082] It should be noted that when inserting the first k-mer data set, you can first check whether there is a vacancy in the third unit of the first unit due to the deleted Bloom filter. If so, insert the first k-mer data set into the vacancy. Since the size m and element cardinality of the Bloom filter of the first unit are consistent, the vacancy of the deleted Bloom filter can be directly inserted into the newly added Bloom filter.

[0083] If there is no empty space in the third unit of the first unit due to the deleted Bloom filter, the first k-mer data set is inserted into the third unit where there is no space for subsequent insertion. It should be noted that the insertion is adjacent to the inserted Bloom filter to improve space utilization.

[0084] As an optional embodiment, the method also includes: receiving a second k-mer data set that needs to be deleted; querying the first unit corresponding to the second k-mer data set; converting the organization form of the third unit belonging to the first unit from an interleaved Bloom filter to a Bloom filter matrix; when the element set of the second k-mer data set is a partial element set of the third unit, deleting the Bloom filter corresponding to the second k-mer data set and retaining empty spaces; when the element set of the second k-mer data set is the entire element set of the third unit, deleting the Bloom filter corresponding to the second k-mer data set and releasing the space resources of the third unit; converting the organization form of the third unit belonging to the first unit from the Bloom filter matrix to an interleaved Bloom filter.

[0085] Similarly, when deleting the second k-mer data set, the first unit to which the second k-mer data set belongs can be queried first. After determining the first unit, the organization form of the third unit in the first unit is converted from an interleaved Bloom filter to a Bloom filter matrix.

[0086] In this way, during the deletion operation, a single deletion operation can be performed on a single Bloom filter as a whole, without having to delete the Bloom filters that need to be deleted multiple times based on the position.

[0087] After the deletion, the organizational form of the third unit inserted into the first unit is converted from a Bloom filter matrix to an interleaved Bloom filter, which can facilitate subsequent element queries.

[0088] It should be noted that, when the element set of the second k-mer data set is a partial element set of the third unit, after the Bloom filter corresponding to the second k-mer data set is deleted, if the third unit still stores element information of other Bloom filters, only empty spaces are retained. When a Bloom filter is subsequently inserted, the inserted Bloom filter can be inserted into the empty space to improve space utilization efficiency.

[0089] In the case where the element set of the second k-mer data set is the entire element set of the third unit, it means that after deleting the Bloom filter corresponding to the second k-mer data set, the third unit no longer stores data with any Bloom filter, and the space resources of the third unit can be released. For example, the third unit can be deleted to reduce the memory usage of the index.

[0090] As an optional embodiment, the first unit of the first level is a segment unit, which includes multiple Bloom filters. The capacity of the Bloom filters of each segment unit is the same, and one Bloom filter records the corresponding element in a k-mer data set. The element cardinality of the data set that can be stored in the segment unit is greater than or equal to the element cardinality of the k-mer data set that needs to be stored; the second unit of the second level is a group unit, and the number of group units of each data set stored in the segment unit is the same as the number of Bloom filters required for the corresponding data set; the third unit of the third level is a block unit, which is a data block with a fixed data size, and is a component unit of the group unit. It stores different elements in the organizational form of interleaved Bloom filters. When querying an element, the block unit reads all data at positions corresponding to the query element in different Bloom filters at one time according to the interleaved Bloom filters.

[0091] By adopting the above segmentation method, sufficient and as small a space as possible can be allocated to each k-mer dataset, thereby optimizing space utilization. The Bloom filter group of each segment unit is divided into multiple group units, and the group index is determined using a group hash function, which ensures that only one Bloom filter is queried for each k-mer dataset during a k-mer query. This grouping strategy reduces the number of Bloom filters that need to be accessed during the query process, thereby improving query efficiency.

[0092] The block unit is organized in the form of an interleaved Bloom filter instead of a classic Bloom filter matrix. By interleaving the bits of multiple Bloom filters, the target bits obtained by a hash operation are made as continuous as possible in the storage space, thereby further improving the query performance.

[0093] Figure 2 is a flowchart of an approximate member query method for a dynamic k-mer data set according to an embodiment of the present invention, such as Figure 2 As shown, the embodiment of the present invention also provides an approximate member query method for a dynamic k-mer data set, the method comprising the following steps:

[0094] Step S201, generating a second unit index of the second level of the target element in the first unit through the grouping function of the first unit of the first level and the target element to be queried, so as to identify the target second unit where the target element is located, wherein the target element is indexed level by level in a multi-level manner, and when being stored in the second level, the corresponding second unit index is generated by using the grouping function of the first unit to which it belongs, and is stored in the target second unit corresponding to the second unit index;

[0095] Step S202, when the target second unit exists in the first unit, determining that the target element is stored in the target second unit of the first unit;

[0096] Step S203, obtaining an interleaved Bloom filter of a third unit of a third level in the target second unit, and obtaining a plurality of hash indexes according to the interleaved Bloom filter, wherein the third unit is a component of the second unit to which it belongs, and when the target element is stored in the third unit, it will be assigned to the Bloom filter of the target data set to which it belongs, and the organization form of the third unit is an interleaved Bloom filter, and the number of bits of the hash index is the same as the number of Bloom filters of the target second unit, and the Bloom filters correspond to different data sets;

[0097] Step S204: query the target data set to which the target element belongs according to the hash index.

[0098] The index query method for the above-mentioned dynamically stored data set provided by the embodiment created by the present invention generates the second unit index of the second level of the target element in the first unit through the grouping function of the first unit of the first level and the target element to be queried, and directly generates the second unit index by using the grouping function of the first unit containing the Bloom filter with the same element cardinality. By querying whether the second unit index is the same, only one Bloom filter needs to be queried for a target element to determine whether the element is in the second unit and the first unit to which it belongs.

[0099] Obtain the interleaved Bloom filter of the third unit of the third level in the target second unit, obtain multiple hash indexes according to the interleaved Bloom filter, query the target k-mer data set to which the target element belongs, and use the interleaved Bloom filter of the third unit to read the same bit of different Bloom filters at one time according to the interleaved Bloom filter when querying the element in the third unit, so as to determine the target k-mer number cluster where the query element is located at one time, avoiding the use of the Bloom filter matrix that can only read the data of one Bloom filter at a time, and needing to traverse the Bloom filter to determine the k-mer data set to which the query element belongs.

[0100] This achieves the technical effect of supporting incremental updates and deletions to meet the usage requirements of k-mer data while ensuring that the space occupied is as small as possible and the number of queries is as small as possible.

[0101] The execution subject of the above steps may be a storage device for k-mer data, and the storage device may include a processor or a processing chip for performing data processing to realize different storage, reading, query and other functions.

[0102] In the above step S201, the second unit index of the second level of the target element in the first unit is generated through the grouping function of the first unit of the first level and the target element to be queried, so as to identify the target second unit where the target element is located.

[0103] This is when the index is generated, the target element is indexed step by step in a multi-level manner, and when stored in the second level, the corresponding second unit index is generated using the grouping function of the first unit to which it belongs, and stored in the target second unit corresponding to the second unit index.

[0104] In this way, when querying, only one Bloom filter needs to be queried for a query element to determine whether the element is in the second unit and the first unit to which it belongs.

[0105] Specifically, through the grouping function of the target first unit, according to the target element of the target k-mer data set, a second unit index of the second level of the target element is generated; this second unit index can establish a one-to-one mapping between its target element and the second unit.

[0106] When querying an element, it is only necessary to directly generate the corresponding second unit index for the query tuple through the grouping function corresponding to the first unit of the query, and by querying whether the second unit index exists in the first unit of the query, it is possible to determine whether there is a second unit to which the query element belongs in the first unit, and when it is determined that the second unit to which the query element belongs exists in the first unit, determine which second unit the second unit is. As in the above step S202, when the target second unit exists in the first unit, it is determined that the target element is stored in the target second unit of the first unit.

[0107] In the above step S203, the interleaved Bloom filter of the third unit of the third level in the target second unit is obtained, and a plurality of hash indexes are obtained according to the interleaved Bloom filter.

[0108] This is because when the index is generated, the third unit is a component of the second unit to which it belongs. When the target element is stored in the third unit, it will be assigned to the Bloom filter of the target k-mer dataset to which it belongs. The third unit is organized as an interleaved Bloom filter. The number of bits of the hash index is the same as the number of Bloom filters of the target second unit. Bloom filters correspond to different k-mer datasets.

[0109] The second unit includes multiple third units of fixed size, such as 64 bits. In addition, the third unit can be created and deleted according to demand.

[0110] Different Bloom filters are arranged according to their positions, so that one read operation can read the same position of the corresponding query element in different Bloom filters, and then through one read operation, it is determined whether the query element exists in the third unit, and which specific Bloom filter the query element belongs to, and the k-mer data set to which it belongs is determined according to the Bloom filter to which it belongs.

[0111] Therefore, in this embodiment, the third unit is organized as an interleaved Bloom filter, and the object of the original single read / delete operation is changed from the data of a single Bloom filter to all the data of multiple Bloom filters at multiple locations. When querying an element, the third unit reads all the data of different Bloom filters at multiple locations at one time according to the interleaved Bloom filter to form a hash index.

[0112] On this basis, according to the above step S204, the target k-mer data set to which the target element belongs can be queried according to the hash index to complete the query of the target element.

[0113] Subsequently, the storage location of the target k-mer data can be found according to the index of the target k-mer data set determined by the query result, and the target k-mer data set can be extracted. After the target k-mer data set is obtained, the query target element in the target k-mer data set can also be obtained.

[0114] As an optional embodiment, querying the target k-mer dataset to which the target element belongs according to the hash index includes: performing a bitwise AND operation on multiple hash indexes to obtain a target index, wherein the hash index is the data of the same bit of different Bloom filters, the hash index is the mapping of the same hash function to the elements of different k-mer datasets, different bits of the hash index represent the values ​​of different Bloom filters, and the values ​​represent whether the target element exists in the Bloom filter; determining the target k-mer dataset to which the target element belongs according to the target index.

[0115] For each third unit in the target second unit, when the Bloom filter obtains multiple hash indexes, obtain k hash indexes h0(x),…h in the third unit. k-1 The k-row bit vector corresponding to (x) is the above hash index.

[0116] Then perform a bitwise AND operation on these hash indexes to generate a bit vector of equal length, which is equivalent to the above target index. Concatenate the target indexes obtained by each third unit. Assuming that the number of sets stored in the first unit is n, the final query result of the first unit is a bit vector of length n, in which each bit indicates whether the corresponding set contains the target element q. Specifically, if a bit is 1, it means that the corresponding set contains q; if it is 0, it means that q is not contained.

[0117] As an optional embodiment, after querying the target k-mer dataset to which the target element belongs according to the hash index, the method also includes: storing the target k-mer dataset in an output list; continuing to query whether the index of the target element exists in other first units; when the index of the target element exists in the first unit, obtaining the target k-mer dataset to which it belongs and storing it in the output list; when all first units are traversed, outputting the output list.

[0118] The above output list can be created synchronously when the query task is established, and initialized to an empty list after creation to store the queried target k-mer data set.

[0119] As an optional embodiment, the first unit of the first level is a segment unit, which includes multiple Bloom filters. The capacity of the Bloom filters of each segment unit is the same, and one Bloom filter records the corresponding element in a k-mer data set. The element cardinality of the data set that can be stored in the segment unit is greater than or equal to the element cardinality of the k-mer data set that needs to be stored; the second unit of the second level is a group unit, and the number of group units for each data set stored in the segment unit is the same as the number of Bloom filters required for the corresponding data set; the third unit of the third level is a block unit, which is a data block with a fixed data size and a component unit of the group unit. It stores different elements in the organizational form of interleaved Bloom filters. When querying an element, the block unit reads all data at positions corresponding to the query element in different Bloom filters at one time according to the interleaved Bloom filters.

[0120] like Figure 3 As shown in the figure, the above segmentation method can allocate enough and as small a space as possible for each k-mer data set, thereby optimizing space utilization. The Bloom filter group of each segment unit is divided into multiple group units, and the group index is determined using a group hash function, which ensures that each k-mer data set is queried by only one Bloom filter in a k-mer query. This grouping strategy reduces the number of Bloom filters that need to be accessed during the query process, thereby improving query efficiency.

[0121] The block unit is organized in the form of an interleaved Bloom filter instead of a classic Bloom filter matrix. By interleaving the bits of multiple Bloom filters, the target bits obtained by a hash operation are made as continuous as possible in the storage space, thereby further improving the query performance.

[0122] It should be noted that this embodiment also provides an optional implementation, which is described in detail below.

[0123] The approximate member query method for dynamic k-mer datasets proposed in this embodiment mainly proposes a new storage architecture and core processes that rely on the architecture, including index construction and element query, and supports incremental addition and deletion of datasets. For the initial dataset, an index is first built to support subsequent query tasks. When a new dataset needs to be added, incremental updates can be achieved without relying on the initial dataset, avoiding redundant backup of the original dataset, thereby saving storage space.

[0124] This embodiment proposes an adaptive Bloom filter allocation strategy, that is, setting Bloom filter related parameters and then allocating an appropriate number of Bloom filters to each data set. At the same time, a grouped hash function is combined to ensure that as few Bloom filters as possible are accessed during the query phase without generating false negatives.

[0125] The approximate member query structure proposed in this embodiment has three levels of basic units, namely, segment (unit), group (unit), and block (unit):

[0126] Segment: A segment represents a logical group of k-mer datasets. Each segment contains a set of Bloom filters that require the same number of Bloom filters. This segmentation method can allocate enough and as small a space as possible for each k-mer dataset, thereby optimizing space utilization.

[0127] Group: The Bloom filter group of each segment is divided into multiple groups, and the grouping hash function g(.) is used to ensure that each k-mer dataset is queried only one Bloom filter in a k-mer query. This grouping strategy reduces the number of Bloom filters that need to be accessed during the query process, thereby improving query efficiency.

[0128] Block: To maximize the efficiency of machine word reading, each group consists of multiple blocks. The block organization does not use the classic Bloom filter matrix, but uses the form of interleaved Bloom filters. By interleaving the bits of multiple Bloom filters, the target bits obtained by a hash operation are as continuous as possible in the storage space, thereby further improving the query performance.

[0129] Figure 3 is a schematic diagram of an index hierarchical architecture of an embodiment of the present invention, such as Figure 3 As shown, segment 1 stores a k-mer data set that requires 1 Bloom filter, segment 2 stores a k-mer data set that requires 2 Bloom filters, and segment 3 stores a k-mer data set that requires 3 Bloom filters.

[0130] Segment 1 includes at most 1 group, segment 2 includes at most 2 groups, and segment 3 includes at most 3 groups. In other words, segment 2 has the ability to store the k-mer data set stored in segment 1, but it will cause space waste.

[0131] The number of groups per segment is equal to the number of Bloom filters needed for the stored k-mer dataset.

[0132] Assuming that a k-mer dataset requires two Bloom filters, they are stored in segment 2, and each group in segment 2 is assigned a Bloom filter to the k-mer dataset to store approximately half of the elements in the k-mer dataset. Assuming that there is an element x in the k-mer dataset, a grouping function is used to determine whether the element x is stored in the Bloom filter assigned to the k-mer dataset in group 1 or in the Bloom filter assigned to the k-mer dataset in group 2.

[0133] Each group can be divided into multiple blocks. The relative positions of Bloom filters assigned to the same k-mer dataset by different groups in the same segment are consistent.

[0134] Each block has a fixed size, which can be 64 bits, and each bit corresponds to at most one Bloom filter. In a 64-bit block, the Bloom filter matrix is ​​as follows: Figure 3 As shown in the figure, the five digits 00010 and 00000 encircled by the ellipse in the figure can be five Bloom filters at different positions of two hash functions, which need five read operations to obtain. The same row can be regarded as the same position, corresponding to the same hash function.

[0135] That is, the target bit of a hash is also a row in the block, such as Figure 3 As shown in the rectangular box in the figure, each block read operation can only obtain data from all positions of a column of the same Bloom filter. In this way, multiple read operations are required to determine whether a query element belongs to the block. Only after all Bloom filters are read can it be determined whether the block does not have the query element and the specific Bloom filter to which the query element belongs.

[0136] The interleaved Bloom filter strings together the data at the same hash position of different Bloom filters in an end-to-end manner. The data formed in this way is arranged in rows in the order of hash positions in the block. In this way, multiple Bloom filters can be queried with one read operation, and it can be determined whether multiple k-mer data sets contain the query element.

[0137] The specific process of index construction is as follows: First, data preprocessing is performed to estimate the cardinality of the input n k-mers set using the HyperLogLog algorithm.

[0138] HyperLogLog (HLL) is a probabilistic algorithm for estimating the number of distinct elements (cardinality) in a large data set. Its basic principle is based on logarithmic probability distribution and hash functions. When an element enters a data set, it is first mapped to a binary hash value through a hash function. Then, the number of consecutive zeros starting from the left of this hash value is observed. For example, the hash value is "00101", and the number of consecutive zeros is 2. By performing this operation on a large number of elements, the maximum value of these consecutive zeros is used to estimate the cardinality of the data set.

[0139] Specifically, HLL divides the data set into multiple buckets (usually a power of 2 buckets), each of which records the maximum number of consecutive zeros in the hash value of the elements in it. Then, based on the maximum values ​​in these buckets, a complex mathematical formula is used to calculate the estimated cardinality of the data set.

[0140] Then, the element cardinality b stored in the basic Bloom filter and the size m of the basic Bloom filter are determined, and the n sets are divided into different segments according to the number of Bloom filters required by the sets.

[0141] Segment L stores a set of L basic Bloom filters. Segment L is logically divided into L groups, that is, the sequence number L of segment L is the same as the number of Bloom filters it contains and the number of corresponding groups. Each k-mer dataset stored in segment L has one Bloom filter in each group. For a k-mer dataset S that requires L Bloom filters i When constructing the element x in , we first determine the group index of x stored in segment L. The specific calculation formula is:

[0142] g x =g(x)mod L

[0143] Store element x into group g x Assigned to the k-mer dataset S i In the Bloom filter, the specific storage method is to use k independent hash functions h0,…h k-1 , each hash function converts the k-mer dataset S i The element x in is randomly and uniformly mapped to an integer in {0,...,m-1}. The corresponding k positions in the Bloom filter are set to 1, that is, B[h0(x)],…, B[h k-1 (x)] is set to 1.

[0144] The specific process of element query is as follows: For the query element q, initialize M q An empty list used to store the dataset containing the query k-mer q.

[0145] Query segment by segment, query the current segment L, and use the same grouping hash function as in the construction phase to calculate the group index to determine the group g to be queried in the current segment q The specific calculation formula is:

[0146] g q =g(q)mod L

[0147] For group g q For each block in the block, get the k hash indexes h0(x),…h k-1 (x) k rows of bit vectors corresponding to the segment L, and perform bitwise AND operations on these bit vectors to generate a bit vector of equal length. The bit vectors obtained from each block are concatenated. Assuming that the number of sets stored in segment L is n, the final query result of segment L is a bit vector of length n, in which each bit indicates whether the corresponding set contains element q. Specifically, if a bit is 1, it means that the corresponding set contains q; if it is 0, it means that q is not contained.

[0148] Each segment queries a group and adds the set containing element q to the query result M q Continue to query the next segment.

[0149] The specific process of incrementally adding a data set is as follows: When inserting a k-mer data set s, first perform a cardinality estimation to determine the segment L to be stored. s .

[0150] Section L s Convert from interleaved Bloom filter form to Bloom filter matrix form to support insert operations.

[0151] In segment L s The insertion position can be the empty position left by the deletion operation or the newly created position for the k-mer dataset s.

[0152] Store the elements of the k-mer dataset s into segments L s The L assigned to it s A Bloom filter.

[0153] Convert the involved blocks from Bloom filter matrix form back to interleaved Bloom filter form to support subsequent queries.

[0154] The specific process of deleting a data set is as follows: delete the k-mer data set s and make sure that the k-mer data set s is stored in the index.

[0155] If the k-mer dataset s is stored in the index, then get the segment L stored in the k-mer dataset s from the metadata s .

[0156] Section Ls Convert from the interleaved Bloom filter form to the Bloom filter matrix form to support query operations.

[0157] Empty or delete all Bloom filters owned by the k-mer dataset s. If the number of deleted sets is small, keep the empty positions. If all sets in an entire block are deleted, release the space resources of the block.

[0158] Convert the involved blocks from the Bloom filter matrix form back to the interleaved Bloom filter form to support subsequent queries.

[0159] Figure 4 It is a schematic diagram of the index construction process of k-mer data in the embodiment mode of the present invention, as Figure 4 shown. The specific implementation of index construction includes the following steps:

[0160] S11: Use the HyperLogLog algorithm to estimate the cardinality of the input n k-mers sets.

[0161] S12: Determine the element cardinality b stored in the basic Bloom filter and the basic Bloom filter size m, and divide the n sets into corresponding segments. The specific division strategy is: for the k-mer dataset S with cardinality c i , if (L - 1)b < c < Lb, then divide the k-mer dataset S i into segment L.

[0162] S13: For the element x in the k-mer dataset S that requires L Bloom filters i , during construction, then determine the group index where x is stored in segment L through the grouped hash function, and then store it in the Bloom filter allocated to the k-mer dataset S i in this group.

[0163] Figure 5 It is a schematic diagram of the element query process of the k-mer data index in the embodiment mode of the present invention, as Figure 5 shown. The specific implementation steps of the element query process are as follows:

[0164] S21: For the query element q, initialize M q as an empty list, which is used to store the datasets containing k-mer q.

[0165] S22: Perform queries segment by segment.

[0166] S23: Query the current segment L. Segment L has L groups, and the sets stored in L have one Bloom filter in each group.

[0167] S24: Calculate the group index g of element q stored in L using the same grouping function as in the construction phase q , that is, if q is stored in L, it must be stored in group g q middle.

[0168] S25: Query g q All Bloom filters in , add the set containing element q to the query result M q Continue to query the next segment.

[0169] S26: After all segments are queried, the query result M is returned q .

[0170] Figure 6 Schematic diagram of the process of adding a data set by k-mer data indexing according to an embodiment of the present invention, such as Figure 6 As shown, the specific steps for incrementally adding a data set are as follows:

[0171] S31: insert k-mer dataset s,

[0172] S32: First, perform a cardinality estimation to determine the segment L to be stored s .

[0173] S33: segment L s Convert from interleaved Bloom filter form to Bloom filter matrix form.

[0174] S34: In segment L s The insertion position can be the empty position left by the deletion operation or the position newly created for the k-mer dataset s. s L s Each group will assign a Bloom filter to the k-mer dataset s, and the relative position of the k-mer dataset s in each group will remain consistent.

[0175] S35: Store the elements of the k-mer dataset s into the L assigned to it s A Bloom filter.

[0176] S36: Convert the involved blocks from the Bloom filter matrix form to the interleaved Bloom filter form to support subsequent queries.

[0177] Figure 7 Schematic diagram of a process of deleting a data set by k-mer data indexing according to an embodiment of the present invention, such as Figure 7 As shown, the specific steps to delete the data set are as follows:

[0178] S41: Delete the k-mer dataset s.

[0179] S42: Search the k-mer dataset s in the metadata to determine if the k-mer dataset s is stored in the index.

[0180] S43: If the k-mer dataset s is stored in the index, obtain the segment L stored in the k-mer dataset s from the metadata s .

[0181] S44: segment L s Convert from interleaved Bloom filter form to Bloom filter matrix form.

[0182] S45: Clear or destroy all Bloom filters corresponding to the set to achieve deletion.

[0183] S46: Convert the involved blocks from the Bloom filter matrix form back to the interleaved Bloom filter form to support subsequent queries.

[0184] Figure 8 Schematic diagram of an approximate member query index generation device for a dynamic k-mer data set according to an embodiment of the present invention. Figure 8 As shown, based on the above-mentioned method for generating approximate member query indexes for dynamic k-mer data sets provided by the embodiments of the present invention, the embodiments of the present invention also provide a device for generating approximate member query indexes for dynamic k-mer data sets, the device comprising:

[0185] A cardinality module 801 is used to determine the cardinality of elements of a target k-mer data set to be stored according to a cardinality estimation algorithm;

[0186] A selection module 802, connected to the cardinality module 801, is used to select a target first unit of the first level that needs to be stored according to the cardinality of the element, wherein the number of Bloom filters of the multiple first units of the first level is different, the capacity of the Bloom filters of the first units is the same, the Bloom filters are used to store the corresponding elements in the target k-mer data set, and the cardinality of the elements that can be stored in the first unit is not less than the cardinality of the elements of the corresponding k-mer data set;

[0187] A grouping module 803 is connected to the selection module 802, and is used to generate a second unit index of a second level of the target element according to the target element of the target k-mer data set through the grouping function of the target first unit, and divide the target element into the target second unit corresponding to the second unit index, wherein the first unit includes a plurality of second units, and the number of the second units of each k-mer data set stored in the first unit is the same as the number of Bloom filters required by the corresponding k-mer data set;

[0188] The allocation module 804 is connected to the above-mentioned grouping module 803, and is used to store the target element in the target third unit of the third level in the target second unit, and allocate it to the Bloom filter of the target k-mer data set, wherein the second unit includes multiple third units of fixed size, and the third unit is organized in the form of an interleaved Bloom filter. When querying an element, the third unit reads the same bit of different Bloom filters at one time according to the interleaved Bloom filter.

[0189] The above-mentioned approximate member query index generation device for dynamic k-mer data sets provided by the embodiment created by the present invention selects the target first unit of the first level that needs to be stored according to the element cardinality, and uses the first unit containing a fixed number of Bloom filters to store the k-mer data set with the element cardinality corresponding to the fixed number, so that sufficient and as small as possible space can be allocated to each k-mer data set to avoid space waste.

[0190] Through the grouping function of the target first unit, according to the target element of the target k-mer data set, the second unit index of the second level of the target element is generated, and the target element is divided into the target second unit corresponding to the second unit index. The number of second units stored in each k-mer data set is the same as the number of Bloom filters required by the corresponding k-mer data set. Each second unit stores a different set of elements of the k-mer data set. In this way, when querying, only one Bloom filter needs to be queried for an element to determine whether the element is in the second unit and the first unit to which it belongs.

[0191] The target element is stored in the target third unit of the third level in the target second unit, and is assigned to the Bloom filter of the target k-mer data set. By using the interleaved Bloom filter of the third unit, the same bit of different Bloom filters can be read at one time in the third unit according to the interleaved Bloom filter when querying the element, so as to determine the target k-mer number cluster where the query element is located at one time, avoiding the situation that the Bloom filter matrix can only read the data of one Bloom filter at a time, and the Bloom filter needs to be traversed to determine the k-mer data set to which the query element belongs.

[0192] This achieves the technical effect of supporting incremental updates and deletions to meet the usage requirements of k-mer data while ensuring that the space occupied is as small as possible and the number of queries is as small as possible.

[0193] Fig. 9 Schematic diagram of an approximate member query device for a dynamic k-mer data set according to an embodiment of the present invention. Fig. 9As shown, based on the above-mentioned approximate member query method for dynamic k-mer data sets provided by the embodiment of the invention, the embodiment of the invention also provides an approximate member query device for dynamic k-mer data sets, the device comprising:

[0194] A generating module 901 is used to generate a second unit index of a target element in a first unit of a second level by using a grouping function of a first unit of a first level and a target element to be queried, so as to identify a target second unit where the target element is located, wherein the target element is indexed level by level in a multi-level manner, and when being stored in the second level, a corresponding second unit index is generated by using the grouping function of the first unit to which it belongs, and is stored in the target second unit corresponding to the second unit index;

[0195] A determination module 902, connected to the generation module 901, is used to determine that the target element is stored in the target second unit of the first unit when the target second unit exists in the first unit;

[0196] An indexing module 903 is connected to the determination module 902, and is used to obtain an interleaved Bloom filter of a third unit of a third level in the target second unit, and obtain multiple hash indexes according to the interleaved Bloom filter, wherein the third unit is a component of the second unit to which it belongs, and when the target element is stored in the third unit, it will be assigned to the Bloom filter of the target k-mer data set to which it belongs, and the organization form of the third unit is an interleaved Bloom filter, and the number of bits of the hash index is the same as the number of Bloom filters of the target second unit, and the Bloom filters correspond to different k-mer data sets;

[0197] The query module 904 is connected to the index module 903 and is used to query the target k-mer data set to which the target element belongs according to the hash index.

[0198] The above-mentioned approximate member query device for dynamic k-mer data sets provided by the embodiment of the invention generates the second unit index of the target element in the second level in the first unit through the grouping function of the first unit of the first level and the target element to be queried, and directly generates the second unit index by using the grouping function of the first unit containing the Bloom filter with the same element cardinality. By querying whether the second unit index is the same, only one Bloom filter needs to be queried for a target element to determine whether the element is in the second unit and the first unit to which it belongs.

[0199] Obtain the interleaved Bloom filter of the third unit of the third level in the target second unit, obtain multiple hash indexes according to the interleaved Bloom filter, query the target k-mer data set to which the target element belongs, and use the interleaved Bloom filter of the third unit to read the same bit of different Bloom filters at one time according to the interleaved Bloom filter when querying the element in the third unit, so as to determine the target k-mer number cluster where the query element is located at one time, avoiding the use of the Bloom filter matrix that can only read the data of one Bloom filter at a time, and needing to traverse the Bloom filter to determine the k-mer data set to which the query element belongs.

[0200] This achieves the technical effect of supporting incremental updates and deletions to meet the usage requirements of k-mer data while ensuring that the space occupied is as small as possible and the number of queries is as small as possible.

[0201] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0202] The embodiments of the present invention further provide a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0203] The embodiment of the invention also provides an approximate member query system for dynamic k-mer data sets, comprising: an electronic device, the electronic device comprising at least one processor; and a memory connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to execute the method of the embodiment of the invention when executed by the at least one processor.

[0204] refer to Fig.10 , a block diagram of an electronic device that can be used as a server or client of an embodiment of the invention will now be described, which is an example of hardware devices that can be applied to various aspects of the invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or required herein.

[0205] like Fig.10As shown, the electronic device includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 to a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0206] Multiple components in the electronic device are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device, and the input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk, an optical disk. The communication unit 1009 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0207] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1002 and / or a communication unit 1009. In some embodiments, the computing unit 1001 may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0208] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0209] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0210] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0211] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0212] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A method for generating approximate membership query index for dynamic k-mer datasets, characterized in that: include: Determine the element cardinality of the target data set to be stored according to a cardinality estimation algorithm; Selecting a target first unit of the first level to be stored according to the element cardinality, wherein the number of Bloom filters of the multiple first units of the first level is different, the capacity of the Bloom filters of the first units is the same, the Bloom filters are used to store the corresponding elements in the target data set, and the element cardinality that can be stored in the first unit is not less than the element cardinality of the corresponding data set; Generate a second unit index of a second level of the target element according to the target element of the target data set through the grouping function of the target first unit, and divide the target element into target second units corresponding to the second unit index, wherein the first unit includes a plurality of second units, and the number of the second units of each data set stored in the first unit is the same as the number of Bloom filters required for the corresponding data set; The target element is stored in a target third unit of the third level in the target second unit, and assigned to the Bloom filter of the target data set, wherein the second unit includes multiple third units of fixed size, and the third unit is organized in the form of an interleaved Bloom filter. When querying an element, the third unit reads the same bit of different Bloom filters at one time according to the interleaved Bloom filter.

2. The method according to claim 1, characterized in that Selecting a target first unit of a first level to be stored according to the cardinality includes: The element cardinality of the target data set is compared with the element cardinality that can be stored in the first unit; When the element cardinality of the target data set is less than or equal to the element cardinality that can be stored in the comparison first unit and greater than the element cardinality that can be stored in the previous first unit, the comparison first unit is used as the target first unit; The first comparison unit is one of a plurality of first units, and the previous first unit is a first unit whose number of Bloom filters is less than that of the first comparison unit by a preset difference.

3. The method according to claim 1, characterized in that Storing the target element in a target third unit of a third level in the target second unit and assigning it to a Bloom filter of the target data set includes: storing the target element into a target third unit of a third level in the target second unit; The target element is hashed using multiple independent hash functions of the target third unit and mapped to a corresponding position in the corresponding Bloom filter, and the corresponding position is set to 1.

4. The method according to claim 1, characterized in that: The method further comprises: Receive a first data set to be inserted; Determine, according to the element cardinality of the first data set, a corresponding first insertion unit; Converting the organization form of the third unit inserted into the first unit from an interleaved Bloom filter to a Bloom filter matrix; Searching for a corresponding insertion position in the first insertion unit, wherein the insertion position is found through a Bloom filter; Inserting the first data set into the insertion position, and assigning a corresponding Bloom filter to the first data set; The organization form of the third unit inserted into the first unit is converted from a Bloom filter matrix to an interleaved Bloom filter.

5. The method according to claim 1, characterized in that The method further comprises: receiving a second data set to be deleted; Query the first unit corresponding to the second data set; Converting the organization form of the third unit belonging to the first unit from an interleaved Bloom filter to a Bloom filter matrix; When the element set of the second data set is a partial element set of the third unit, the Bloom filter corresponding to the second data set is deleted, and an empty position is reserved; In a case where the element set of the second data set is the entire element set of the third unit, deleting the Bloom filter corresponding to the second data set, and releasing the space resources of the third unit; The organization form of the third unit belonging to the first unit is converted from a Bloom filter matrix to an interleaved Bloom filter.

6. An approximate membership query method for dynamic k-mer datasets, characterized in that: include: Generate a second unit index of the second level of the target element in the first unit through a grouping function of the first unit of the first level and a target element to be queried, so as to identify the target second unit where the target element is located, wherein the target element is indexed level by level in a multi-level manner, and when being stored in the second level, the corresponding second unit index is generated by using the grouping function of the first unit to which it belongs, and is stored in the target second unit corresponding to the second unit index; In the case where the target second cell exists in the first cell, determining that the target element is stored in the target second cell of the first cell; Obtain an interleaved Bloom filter of a third unit of a third level in the target second unit, and obtain a plurality of hash indexes according to the interleaved Bloom filter, wherein the third unit is a component of the second unit to which it belongs, and when the target element is stored in the third unit, it will be assigned to the Bloom filter of the target data set to which it belongs, and the third unit is organized in an interleaved Bloom filter, and the number of bits of the hash index is the same as the number of Bloom filters of the target second unit, and the Bloom filters correspond to different data sets; The target data set to which the target element belongs is queried according to the hash index.

7. The method according to claim 6, characterized in that Querying a target data set to which the target element belongs according to the hash index includes: Performing a bitwise AND operation on the plurality of hash indexes to obtain a target index, wherein the hash index is data of the same bit of different Bloom filters, the hash index is a mapping of the same hash function to elements of different data sets, different bits of the hash index represent values ​​of different Bloom filters, and the values ​​represent whether the Bloom filter has the target element; The target data set to which the target element belongs is determined according to the target index.

8. The method according to claim 6, characterized in that After querying the target data set to which the target element belongs according to the hash index, the method further includes: storing the target data set in an output list; Continue to query whether the index of the target element exists in other first units; If the index of the target element exists in the first unit, obtain the corresponding target data set and store it in the output list; When all first units have completed traversal, the output list is output.

9. The method according to any one of claims 1 to 8, characterized in that The first unit of the first level is a segment unit, the segment unit includes a plurality of Bloom filters, the capacity of the Bloom filters of each segment unit is the same, one Bloom filter records a corresponding element in a data set, and the element cardinality of the data set that can be stored by the segment unit is greater than or equal to the element cardinality of the data set that needs to be stored; The second unit of the second level is a group unit, and the number of the group units of each data set stored in the segment unit is the same as the number of Bloom filters required by the corresponding data set; The third unit of the third level is a block unit, which is a data block with a fixed data size and is a component unit of the group unit. Different elements are stored in the form of an interleaved Bloom filter. When querying an element, the block unit reads all data at positions corresponding to the query element in different Bloom filters at one time according to the interleaved Bloom filters.

10. An approximate membership query system for dynamic k-mer datasets, comprising: A processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images