Object set matching method, electronic device, storage medium, and program product

Abstract

The present disclosure provides an object set matching method, an electronic device, a storage medium and a program product. The method of the present disclosure comprises: performing hierarchical analysis on a target object according to multiple attribute dimensions, and constructing a bit set of a bit map data block from the obtained attribute dimensions and attribute values; selecting a corresponding heterogeneous bit map container for storage according to the data density of each bit map data block in the bit set; in response to a set matching request containing at least one attribute condition, determining a target bit map data block associated with the matching request; and performing mixed bit operation on the target bit map data blocks belonging to different heterogeneous bit map container types by using a corresponding bit operation strategy determined according to the target heterogeneous bit map container type to which the target bit map data block belongs, to obtain a target bit set.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to object set matching methods, electronic devices, storage media, and program products. Background Technology

[0002] Existing set matching schemes often employ a single data structure to construct the bitmap index, such as dense bitmaps based on contiguous memory storage (e.g., BitSet) or sparse bitmaps based on block compression (e.g., RoaringBitmap) for data storage and computation. When faced with complex matching requirements involving extremely varying or dynamically changing attribute data densities, dense bitmaps based on contiguous storage generate massive amounts of invalid zero-bit storage when processing extremely sparse data (e.g., low-frequency feature attributes), resulting in severe memory waste. Conversely, sparse bitmaps based on block compression, when dealing with high-density data, introduce additional memory overhead and computational losses due to their internal block management, container encoding, and compression / decompression logic, leading to computational performance inferior to native dense bitmaps. Therefore, a set matching scheme that adapts to data density is needed. Summary of the Invention

[0003] This disclosure provides methods for matching sets of objects, electronic devices, storage media, and program products.

[0004] According to a first aspect of this disclosure, an object set matching method is provided. The method includes: performing hierarchical parsing of a target object according to multiple attribute dimensions; constructing a bit set of bitmap data blocks from the obtained attribute dimensions and attribute values; selecting a corresponding heterogeneous bitmap container for storage based on the data density of each bitmap data block in the bit set; determining the target bitmap data block associated with the matching request in response to a set matching request containing at least one attribute condition; and performing mixed bit operations on target bitmap data blocks belonging to different heterogeneous bitmap container types using a bit operation strategy determined based on the target heterogeneous bitmap container type to which the target bitmap data block belongs, thereby obtaining a target bit set.

[0005] According to the aforementioned disclosure, a hierarchical parsing method is used to construct independently managed bitmap data blocks. A heterogeneous bitmap container system, which can be dynamically selected based on data density, is introduced within the same system, enabling both sparse and dense data to achieve optimal storage structure and computational efficiency, while balancing high compression ratios and high access performance. By performing cross-container type hybrid bit operations, heterogeneous bitmap data blocks can directly perform collaborative computations without prior full data format conversion. This solution effectively addresses the problems of high storage overhead, large computational latency, and insufficient throughput in high-concurrency scenarios caused by a unified data structure, achieving fast and accurate matching of arbitrary attribute combinations in massive object sets.

[0006] In at least one embodiment of this disclosure, a corresponding heterogeneous bitmap container is selected for storage based on the data density of each bitmap data block in the bit set, including: calculating the non-zero bit data density of the bitmap data block; if the data density is greater than a preset density threshold, a dense bitmap container based on continuous bit storage is selected as the heterogeneous bitmap container for storage; wherein, the dense bitmap container is a bitmap container with a continuous memory array structure.

[0007] In at least one embodiment of this disclosure, a corresponding heterogeneous bitmap container is selected for storage based on the data density characteristics of each bitmap data block in the bit set, including: calculating the non-zero bit data density of the bitmap data block; if the data density is not greater than a preset density threshold, a sparse bitmap container based on block compression is selected as the heterogeneous bitmap container, and the bitmap data block is stored after block compression; wherein, the sparse bitmap container is a bitmap container that is divided according to a preset data block size and the sparse data block is compressed using in-container encoding.

[0008] In at least one embodiment of this disclosure, a target bit set is obtained by determining the corresponding bit operation strategy based on the target heterogeneous bitmap container type to which the target bitmap data block belongs, and performing hybrid bit operations on target bitmap data blocks belonging to different heterogeneous bitmap container types. This includes: identifying the heterogeneous bitmap container type to which the target bitmap data block belongs; if the target bitmap data block belongs to a sparse bitmap container based on block compression, then the first bit operation logic adapted to the block compression rules is used to directly perform bit matching calculation in the compressed state; if the target bitmap data block belongs to a dense bitmap container based on continuous bit storage, then the second bit operation logic is used to perform bit matching calculation on the target bitmap data block; and combining the bit matching calculation results corresponding to different heterogeneous bitmap container types according to the logical operation relationship contained in the set matching request to obtain the target bit set.

[0009] In at least one embodiment of this disclosure, when the logical operation relationship is an intersection operation, the combination operation method includes: determining the target bitmap data block belonging to the sparse bitmap container based on block compression as the query index source; traversing the query index source and extracting the set bit indexes therein; performing a matching search at the corresponding position in the target bitmap data block belonging to the dense bitmap container based on continuous bit storage according to the extracted set bit indexes; and taking the set bit indexes found in the dense bitmap container based on continuous bit storage as the intersection operation result.

[0010] In at least one embodiment of this disclosure, the target object is parsed hierarchically according to multiple attribute dimensions, and a bit set of bitmap data blocks is constructed from the obtained attribute dimensions and attribute values. This includes: parsing multiple attribute dimensions of the target object layer by layer according to a preset attribute hierarchy relationship, and determining the attribute values ​​of the attribute dimensions at each level; constructing bitmap data blocks using the corresponding attribute values ​​as indexes for each level of attribute dimension; and constructing a bit set from multiple bitmap data blocks at the same level or different levels.

[0011] In at least one embodiment of this disclosure, in response to a set matching request containing at least one attribute condition, determining the target bitmap data block associated with the matching request includes: performing semantic parsing on the set matching request to determine at least one target attribute dimension and the corresponding target attribute value; and searching in a stored heterogeneous bitmap container based on the target attribute dimension and the target attribute value to determine the target bitmap data block corresponding to the target attribute value.

[0012] In at least one embodiment of this disclosure, when the target object data is updated, the method further includes: monitoring the data density change of the stored bitmap data block; if the data density of the bitmap data block changes from being greater than a preset density threshold to not being greater than a preset density threshold, then the bitmap data block is migrated from a dense bitmap container based on continuous bit storage to a sparse bitmap container based on block compression; if the data density of the bitmap data block changes from not being greater than a preset density threshold to being greater than a preset density threshold, then the bitmap data block is migrated from a sparse bitmap container based on block compression to a dense bitmap container based on continuous bit storage.

[0013] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs the method described in the first aspect of any embodiment of this disclosure.

[0014] According to a third aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the method described in the first aspect of any embodiment of this disclosure.

[0015] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect of any embodiment of this disclosure. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 This is a flowchart illustrating the object set matching method provided in the embodiments of this disclosure.

[0018] Figure 2 This is a flowchart illustrating a high data density storage method provided in an embodiment of the present disclosure.

[0019] Figure 3 This is a flowchart illustrating a low data density storage method provided in an embodiment of the present disclosure.

[0020] Figure 4 This is a flowchart illustrating the target bit set generation method provided in the embodiments of this disclosure.

[0021] Figure 5 This is a flowchart illustrating the intersection operation method provided in an embodiment of the present disclosure.

[0022] Figure 6 This is a flowchart illustrating the bit set generation method provided in an embodiment of the present disclosure.

[0023] Figure 7 This is a flowchart illustrating the method for determining a target bitmap data block provided in an embodiment of this disclosure.

[0024] Figure 8 This is a flowchart illustrating the target object data update method provided in an embodiment of the present disclosure.

[0025] Figure 9 This is a schematic block diagram of an object set matching device according to one embodiment of the present disclosure.

[0026] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0027] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0028] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 This is a flowchart illustrating the object set matching method provided in an embodiment of this disclosure. Figure 1The method shown includes steps S101 to S104. This method can be applied to the server side (e.g., local server, cloud server, etc.).

[0030] Specifically, Figure 1 The method shown includes step S101: performing hierarchical parsing of the target object according to multiple attribute dimensions, and constructing a bit set of bitmap data blocks from the obtained attribute dimensions and attribute values.

[0031] It should be noted that the target object referred to here is the set of entities to be partitioned and matched. This entity can be a user account, device identifier, product identifier, or any uniquely identifiable data subject. Each target object possesses a globally unique object identifier, which forms the basis for the bit index in subsequent bitmap data blocks. Each target object has multiple attribute dimensions. An attribute dimension refers to a classification perspective used to describe the characteristics of the target object. For example, in a user profiling scenario, it could include region, brand preference, job category, etc., each constituting a different attribute dimension. Each attribute dimension contains several specific attribute values; for example, the region dimension could include attribute values ​​such as Beijing and Shanghai.

[0032] The bitmap data block mentioned here is the basic unit of data storage and computation. It is a binary vector indexed bit by bit. For a specific attribute value under a certain attribute dimension, the system constructs a bitmap data block, where the position index of each bit in the data block corresponds to the object identifier of a target object. If the corresponding bit of an object identifier in the data block is set to 1, it means that the target object has a corresponding attribute value; if it is 0, it means that there is no attribute value.

[0033] The bit set mentioned here refers to a logical set consisting of all bitmap data blocks corresponding to multiple attribute values ​​under multiple attribute dimensions. This set constitutes the data foundation for multi-dimensional attribute description of the entire target object set. The bit set constructed through hierarchical parsing decouples data from different attribute dimensions, avoiding data bloat caused by dimension combination explosion in traditional solutions.

[0034] In practical applications, the first step is to acquire the complete set of data for the target objects. Each target object contains its attribute values ​​across various attribute dimensions. Hierarchical parsing is then performed according to the preset attribute dimensions; that is, for each independent attribute dimension, all possible attribute values ​​under that dimension are extracted. For each pair of "attribute dimension: attribute value," a corresponding bitmap data block is constructed. The position index of each bit in this bitmap data block corresponds to an object identifier of the target object. The length of the bitmap data block is at least equal to or greater than the maximum value of object identifiers in the complete set of target objects to ensure that each object identifier has a unique corresponding bit. All target objects are traversed; if a target object matches the "attribute dimension: attribute value" combination, the bit corresponding to that target object identifier in the bitmap data block is set to 1; otherwise, it is set to 0. Thus, all bitmap data blocks corresponding to "attribute dimension: attribute value" combinations together constitute a bit set describing the multidimensional attribute characteristics of the target object. This hierarchical construction method allows data under each attribute dimension to be managed independently, greatly reducing the data coupling between different attribute dimensions.

[0035] Step S102: Select the corresponding heterogeneous bitmap container for storage based on the data density of each bitmap data block in the bit set.

[0036] Data density, as referred to here, is the fill ratio of set bits in a bitmap data block, used to measure the sparseness or density of that data block. Data density can be expressed as the number of "1" bits in the data block divided by the total number of bits in the data block. Data density characteristics serve as the basis for subsequent decisions regarding the selection of heterogeneous bitmap containers. Heterogeneous bitmap containers, as referred to here, are at least two physical storage containers provided by the system that differ fundamentally in their underlying data structures and storage modes. This scheme forms a heterogeneous storage system by simultaneously coexisting at least two different types of bitmap containers within the same system. These two container types coexist in the system, and the system chooses between them for each bitmap data block, rather than using only one type for the entire system.

[0037] Specifically, after the initial construction of the bitmap data blocks is completed, the system does not use a uniform data structure for storage. Instead, it first monitors and calculates the data density of each bitmap data block. The data density is specifically represented by the proportion of "1" bits in the bitmap data block.

[0038] Based on the calculated data density, a target container type is dynamically selected for storage within a container set containing at least two heterogeneous bitmap containers. These two heterogeneous bitmap containers have distinctly different underlying implementation logic and applicable scenarios; they coexist in the same system, allowing the system to select one based on data characteristics.

[0039] In practical applications, bitmap data density distribution is sometimes extremely uneven: under common attributes like "gender = male," bitmap data blocks may be very dense; while under long-tail attributes like "preference = a niche brand," bitmap data blocks are extremely sparse. If a single type of container is used, such as uniformly using a contiguous memory array for storage, sparse bitmaps will waste a lot of storage space and computational bandwidth; if a block-compressed container is used uniformly, accessing dense bitmaps requires decompression overhead each time, failing to achieve the efficient access performance of native arrays. This solution, by coexisting two heterogeneous containers in the same system, enables each type of data to obtain the most suitable storage and computation method for its own characteristics, balancing space efficiency and computational efficiency—something that a single container solution cannot achieve.

[0040] Step S103: In response to a set matching request containing at least one attribute condition, determine the target bitmap data block associated with the matching request.

[0041] After the system has stably stored all bitmap data blocks, it can receive external set matching requests. This request contains at least one attribute condition, such as "Region = Beijing" and "Brand Preference = Brand A". The system parses the matching request, extracting the target attribute dimensions and their corresponding values. Subsequently, the system searches the constructed bit set to quickly locate and determine the target bitmap data blocks stored using various heterogeneous bitmap containers that correspond to these target attribute values. These target bitmap data blocks are the direct operands for subsequent mixed bitwise operations.

[0042] Step S104: Using the bit operation strategy determined according to the target heterogeneous bitmap container type to which the target bitmap data block belongs, perform mixed bit operations on the target bitmap data blocks belonging to different heterogeneous bitmap container types to obtain the target bit set.

[0043] It should be noted that the hybrid bitwise operations mentioned here refer to the process of collaboratively performing logical bitwise operations (such as intersection, union, difference, etc.) on multiple bitmap data blocks belonging to at least two different heterogeneous bitmap container types during a single set matching calculation. This operation requires the adoption of appropriate bitwise operation strategies based on the characteristics of each container type, and finally merging the calculation results.

[0044] In practical applications, after the target bitmap data blocks are determined, since these target bitmap data blocks may belong to different types of heterogeneous bitmap containers, they cannot be processed using a single bit operation process and need to be processed using mixed bit operations.

[0045] First, the system identifies the specific heterogeneous bitmap container type to which each target bitmap data block belongs, thereby determining the appropriate bitwise operation strategy. The principle for formulating the bitwise operation strategy is to fully utilize the storage characteristics of each container to achieve the highest computational efficiency. For example, for sparse bitmap containers based on block compression, the bitwise operation strategy might be to perform operations directly in the compressed state, executing intersection, union, and other logic by directly manipulating the compressed integer array, avoiding the time and space overhead of decompression; for dense bitmap containers based on continuous bit storage, the bitwise operation strategy might be to directly perform efficient 64-bit word-aligned bitwise operations on its continuous bit array.

[0046] Subsequently, based on the logical relationships between the attribute conditions in the matching request (e.g., requiring multiple conditions to be met simultaneously, or any one condition to be met), the system collaboratively executes these bitwise operation strategies adapted to different containers, performs mixed bitwise operations on multiple target bitmap data blocks, and logically merges the intermediate operation results. Finally, the system calculates a result bitmap containing the identifiers of all target objects that meet the matching conditions, i.e., the target bit set.

[0047] Based on the publicly available solutions described above, by constructing independently managed bitmap data blocks through layered parsing and introducing a heterogeneous bitmap container system within the same system that can be dynamically selected based on data density, both sparse and dense data can achieve optimal storage structure and computational efficiency, balancing high compression ratios and high access performance. By performing hybrid bit operations across container types, heterogeneous bitmap data blocks can directly perform collaborative computations without prior full data format conversion. This solution effectively addresses the problems of high storage overhead, large computational latency, and insufficient throughput in high-concurrency scenarios caused by a unified data structure, achieving fast and accurate matching of arbitrary attribute combinations in massive object sets.

[0048] In one or more embodiments of this disclosure, such as Figure 2 This is a flowchart illustrating a high data density storage method provided in an embodiment of this disclosure. Figure 2 As shown, based on the data density of each bitmap data block in the bit set, a corresponding heterogeneous bitmap container is selected for storage, including: Step S201: Calculate the non-zero bit data density of the bitmap data block. Step S202: If the data density is greater than a preset density threshold, a dense bitmap container based on continuous bit storage is selected as the heterogeneous bitmap container for storage. Here, the dense bitmap container is a bitmap container with a continuous memory array structure.

[0049] It's important to note that the non-zero bit density mentioned here refers to the ratio between the number of bits set to "1" in a bitmap data block and the total number of bits contained in that bitmap data block. This ratio reflects the filling level of the bitmap data block and is a quantitative indicator for determining whether a bitmap data block is sparse or dense. The value of the non-zero bit density ranges from 0 to 1; the closer the value is to 1, the denser the bitmap data block, and the closer the value is to 0, the sparser the bitmap data block.

[0050] For example, the non-zero bit density can be calculated by traversing the bitmap data block and counting the number of set bits, and then dividing the count by the total number of bits in the bitmap data block.

[0051] After obtaining the non-zero bit data density, the non-zero bit data density is compared with a preset density threshold, and different container selection strategies are executed according to the comparison results.

[0052] When the calculated non-zero bit density of a bitmap data block exceeds a preset density threshold, it indicates that the proportion of set bits in the bitmap data block is high, and the data distribution is relatively dense. In this case, a dense bitmap container based on continuous bit storage is selected as the heterogeneous bitmap container for storing the bitmap data block. The underlying implementation of this dense bitmap container is a continuous memory array structure, that is, the entire bitmap data block is stored in memory as a continuous bit array.

[0053] Based on the aforementioned publicly available solutions, by allocating high-density bitmap data blocks exceeding a preset density threshold to dense bitmap containers based on contiguous bit storage, these data blocks can achieve an optimal storage structure and computational efficiency that matches their data distribution characteristics. This avoids the additional decompression overhead and access performance loss caused by using compressed storage for dense data in a single container solution. This adaptive storage strategy, which varies depending on data density, allows this solution to simultaneously address the high compression ratio requirements of sparse data and the high access performance requirements of dense data within the same system, solving the problem that a uniform storage structure cannot be used to specifically optimize heterogeneous density data.

[0054] In one or more embodiments of this disclosure, such as Figure 3 This is a schematic flowchart illustrating a low-data-density storage method provided in an embodiment of this disclosure. Figure 3As shown, based on the data density characteristics of each bitmap data block in the bit set, a corresponding heterogeneous bitmap container is selected for storage, including: Step S301: Calculate the non-zero bit data density of the bitmap data block. Step S302: If the data density is not greater than a preset density threshold, a sparse bitmap container based on block compression is selected as the heterogeneous bitmap container, and the bitmap data block is stored after block compression. The sparse bitmap container is a bitmap container that is divided according to a preset data block size and uses in-container encoding compression for the sparse data blocks.

[0055] In practical applications, when the comparison result shows that the density of the non-zero bit data is less than or equal to a preset density threshold, it indicates that the proportion of set bits in the bitmap data block is low, and the data distribution exhibits significant sparse characteristics. In this case, the system selects a sparse bitmap container based on block compression as the heterogeneous bitmap container for storing the bitmap data block.

[0056] First, based on a preset data block size, the bitmap data block is divided into multiple consecutive logical data blocks starting from the beginning. Each logical data block has the same bit length. For each logical data block, the system checks whether there are any set bits. If all bits in a logical data block are 0, an empty tag is generated for that block, indicating that the block does not contain valid data, and no physical space is allocated to store the actual bit data. If a logical data block contains at least one set bit, it is considered a sparse data block. The system performs in-container encoding compression on the sparse data block, converting its internal set bit information into a compact compressed encoding representation, and stores the compressed result in the sparse bitmap container. After the above processing, the entire bitmap data block is converted into a compact storage form composed of several empty tags and compressed encoded data blocks, completing the storage process after block compression.

[0057] Choosing a sparse bitmap container based on block compression to store low-density bitmap data blocks can significantly reduce memory usage. For highly sparse bitmap data blocks, the vast majority of bits are in a zero state, with only a very small number of bits set. If a contiguous memory array structure is used for storage, even if most of the space stores meaningless zero values, physical memory must still be allocated for all bits, resulting in a serious waste of memory resources. Especially when dealing with hundreds of millions of object identifier spaces and massive attribute dimensions, this waste will cause memory usage to expand rapidly, even exceeding the available memory capacity of a single node or cluster, making the system unable to operate normally.

[0058] Sparse bitmap containers based on block compression, through a block-based strategy, can merge large, continuous zero-value regions into empty tags, allocating storage space and encoding compression only for a small number of data blocks in the sparse regions that actually contain set bits. This approach changes the storage overhead from being proportional to the total length of the bitmap to a compression relationship proportional only to the number of set bits, significantly improving storage efficiency. This allows massive sparse bitmap data blocks that were previously impossible to maintain due to memory limitations to be efficiently loaded and stored in memory or cache. This provides a fundamental storage foundation for this solution to achieve real-time multi-dimensional attribute matching on a scale of hundreds of millions of objects.

[0059] Based on the publicly available solutions described above, by dividing sparse data blocks into logical data blocks, compressing and storing only blocks containing set bits, and retaining only the markers for blocks containing all zeros, storage efficiency is improved by orders of magnitude. This adaptive compression storage strategy, which adapts to different data densities, achieves both fast access to dense data and high compression ratio storage of sparse data within the same system, thus balancing the dual optimization goals of performance and space.

[0060] In one or more embodiments of this disclosure, such as Figure 4 This is a flowchart illustrating the target bit set generation method provided in an embodiment of this disclosure. Figure 4 As shown, the method utilizes the target heterogeneous bitmap container type to determine the corresponding bit operation strategy, and performs mixed bit operations on target bitmap data blocks belonging to different heterogeneous bitmap container types to obtain a target bit set. This includes: Step S401: Identifying the heterogeneous bitmap container type to which the target bitmap data block belongs. Step S402: If the target bitmap data block belongs to a sparse bitmap container based on block compression, the first bit operation logic adapted to the block compression rules is used to directly perform bit matching calculations in the compressed state. Step S403: If the target bitmap data block belongs to a dense bitmap container based on continuous bit storage, the second bit operation logic is used to perform bit matching calculations on the target bitmap data block. Step S404: Based on the logical operation relationships contained in the set matching request, the bit matching calculation results corresponding to different heterogeneous bitmap container types are combined to obtain the target bit set.

[0061] It should be noted that the block compression rules mentioned here refer to a set of encoding compression specifications followed by sparse bitmap containers based on block compression when constructing and storing bitmap data blocks. These rules include: dividing the original bitmap data block into multiple logical data blocks according to a preset block size; recording only empty markers for all-zero data blocks; and using a compact encoding method to record the position information of set bits for sparse data blocks containing set bits. These block compression rules determine the specific encoding format and data structure layout of the bitmap data blocks in the compressed state.

[0062] The first bit operation logic mentioned here refers to a bit-matching computation logic specifically adapted for sparse bitmap containers based on block compression. This logic can directly perform bit-matching operations on the compressed data structure while keeping the bitmap data block in a compressed state, without needing to decompress the bitmap data block before the operation. Here, "compressed state" refers to the storage state of the bitmap data block after block compression processing. In the compressed state, the original bit sequence of the bitmap data block has been converted into a compact representation composed of null markers and compressed data blocks, and its physical storage layout is fundamentally different from the original continuous bit sequence.

[0063] The second bitwise operation logic mentioned here refers to a bit-matching calculation logic specifically adapted for dense bitmap containers based on contiguous bit storage. This operation logic operates directly on the bit sequence in the contiguous memory array structure, utilizing direct memory addressing and processor word alignment operations to perform bit-matching calculations.

[0064] The logical operations mentioned here refer to the Boolean logical relationships between multiple attribute conditions contained in a set matching request. Logical operations include, but are not limited to, intersection, union, and difference operations. An intersection operation requires the target object to satisfy multiple attribute conditions simultaneously; a union operation requires the target object to satisfy at least one of multiple attribute conditions; and a difference operation requires the target object to satisfy one attribute condition but excludes objects that satisfy another attribute condition.

[0065] The combination operation mentioned here refers to the process of merging intermediate results obtained by performing bit matching calculations on target bitmap data blocks belonging to different heterogeneous bitmap container types using different bit operation logics, according to the logical operation relationship specified in the set matching request.

[0066] In practical applications, after determining the target bitmap data block associated with the set matching request, the first step is to identify which heterogeneous bitmap container type each target bitmap data block belongs to. Since the system has already allocated a corresponding heterogeneous bitmap container to each bitmap data block based on its data density characteristics during the data storage phase, each bitmap data block carries metadata identifiers for its container type. By reading these metadata identifiers, it can be determined whether the target bitmap data block is stored using a sparse bitmap container with block compression or a dense bitmap container with continuous bit storage. This identification operation is a prerequisite for subsequently assigning different bit operation strategies.

[0067] Furthermore, after identifying the container type of each target bitmap data block, the system assigns different bitwise operation logic according to different container types. Specifically: If the identification result indicates that the target bitmap data block belongs to a sparse bitmap container based on block compression, the system adopts the first-order operation logic adapted to the block compression rules to directly perform bit matching calculations on the target bitmap data block in compressed state. The specific working method of this first-order operation logic is as follows: Since the block compression rules divide the original bitmap data block into multiple logical data blocks and store the position information of the set bits in a compact encoding format, the first-order operation logic can directly parse this compact encoding structure. Without needing to restore the entire bitmap data block to the original bit sequence, bit matching is completed through direct comparison and operation of the compressed encoded data. For example, when performing an intersection operation between two sparse bitmap data blocks that are both in compressed state, the first-order operation logic can simultaneously traverse the set bit position indices recorded in each of the two compressed data blocks, directly finding the set of position indices that appear in both, without needing to decompress either data block. This direct operation method in compressed state avoids the decompression calculation overhead and temporary memory allocation overhead caused by completely restoring the sparse bitmap data block from the compressed encoding format to a continuous bit sequence, significantly improving the computational efficiency of sparse bitmap data.

[0068] If the identification result indicates that the target bitmap data block belongs to a dense bitmap container based on continuous bit storage, the system employs a second bitwise operation logic to perform bit matching calculations on the target bitmap data block. The specific operation of this second bitwise operation logic is as follows: since the underlying implementation of the dense bitmap container is a continuous memory array structure, with each bit arranged continuously in memory according to its index, the second bitwise operation logic directly performs bit matching operations on this continuous memory array. In one specific implementation, the second bitwise operation logic utilizes the processor word length to perform 64-bit aligned batch bitwise operations, loading a continuous bit segment of one processor word at a time and performing logical operations such as bitwise AND, bitwise OR, and bitwise NOT, thereby achieving extremely high computational throughput.

[0069] It should be noted that among the multiple target bitmap data blocks associated with a single set matching request, there may be cases where they belong to different heterogeneous bitmap container types. For example, a matching request may be associated with three target bitmap data blocks, two of which belong to sparse bitmap containers based on block compression, and the other to dense bitmap containers based on continuous bit storage. In this case, the system uses the first bit operation logic to process the target bitmap data block belonging to the sparse bitmap container, and the second bit operation logic to process the target bitmap data block belonging to the dense bitmap container. The two bit operation logics are executed collaboratively in the processing flow of the same request, producing bit matching calculation results corresponding to their respective types.

[0070] After obtaining the bit-matching calculation results for each target bitmap data block, the system reads the logical operation relationship contained in the set matching request and performs a combination operation on all the bit-matching calculation results obtained in the previous steps according to the logical operation relationship. The combination operation mentioned here is the specific operational means of implementing the logical operation relationship, and the logical operation relationship determines the execution method of the combination operation. Specifically, if the logical operation relationship contained in the set matching request is an intersection operation, the combination operation performs an intersection-union operation on each intermediate result, retaining only the object identifiers that appear in all intermediate results simultaneously; if the logical operation relationship is a union operation, the combination operation performs a union-union operation on each intermediate result, retaining the object identifiers that appear in any one intermediate result; if the logical operation relationship is a difference operation, the combination operation removes the object identifiers that appear in one intermediate result from another. When the logical operation relationship is a combination of the above relationships, the combination operation executes the corresponding merging operation step by step according to the priority or nesting order specified in the request.

[0071] In real-world applications, set matching requests often involve complex combinations of attribute conditions, making simple intersection or union operations insufficient to meet diverse matching needs. For example, in a demographic targeting scenario, a matching request might require selecting targets that meet the criteria of "location equal to Beijing" and "age greater than 25," while excluding those whose "occupation equals student." This request involves both intersection and difference operations. This solution supports multiple combination operations within a hybrid bitwise operation framework, enabling flexible responses to matching requests containing arbitrary Boolean logic combinations. Furthermore, the unified merging process for combination operations effectively manages the dependencies and order of intermediate results, preventing complex logical relationships from causing confusion or redundant calculations. After the combination operations are completed, the system obtains a final target bit set, where each set bit corresponds to an identifier of a target object that satisfies all matching conditions.

[0072] Based on the aforementioned publicly available solutions, by identifying the container type of the target bitmap data block and selecting a suitable bitwise operation logic, bitmap data blocks under different storage structures can participate in bit matching calculations in the optimal way. Direct operation in compressed state avoids the decompression overhead of sparse data, while direct operation on contiguous memory arrays leverages the advantages of dense data access and batch processing. By combining and merging the results of heterogeneous bitwise operations according to logical operation relationships, unified integration of bit matching calculation results from different container types is achieved. A hybrid bitwise operation architecture capable of co-computing with multiple heterogeneous container types is constructed, realizing efficient and flexible set matching on massive heterogeneous storage data.

[0073] In one or more embodiments of this disclosure, such as Figure 5This is a flowchart illustrating the intersection operation method provided in an embodiment of this disclosure. Figure 5 As shown, when the logical operation relationship is an intersection operation, the combination operation method includes: Step S501: Determine the target bitmap data block belonging to the sparse bitmap container based on block compression as the query index source. Step S502: Traverse the query index source and extract the set bit indexes. Step S503: According to the extracted set bit indexes, perform matching and searching at the corresponding positions in the target bitmap data block belonging to the dense bitmap container based on continuous bit storage. Step S504: Take the set bit indexes found in the dense bitmap container based on continuous bit storage as the intersection operation result.

[0074] Among the multiple target bitmap data blocks participating in the intersection operation, the system first identifies which target bitmap data blocks belong to the sparse bitmap container based on block compression. If at least one such sparse bitmap data block exists, it is determined as the query index source. If multiple target bitmap data blocks belong to the sparse bitmap container, the one with the lowest non-zero bit data density can be selected as the query index source.

[0075] The reason for choosing sparse bitmap data blocks as the query index source is that the sparse bitmap container based on block compression stores the position indices of all set bits in a compact encoded form under compressed conditions. The system can directly obtain the complete list of set bit indices without decompressing the entire bitmap data block. Furthermore, since the number of set bits in a sparse bitmap data block is extremely small, the number of iterations traversing these set bit indices is kept to a very low level. Using the side with the fewest set bits as the query index source minimizes the number of subsequent matching searches performed in the dense bitmap container, thereby reducing the overall computational load to the greatest extent.

[0076] The system iterates through the sparse bitmap data block identified as the source of the query index. Since this bitmap data block belongs to a sparse bitmap container based on block compression, its compressed data structure directly maintains a list of position indices for set bits or stores equivalent information in a compact encoding format. The system reads this data structure in compressed state, extracting the position index value of each set bit recorded therein, forming a set bit index sequence.

[0077] For each extracted set bit index, the system uses that set bit index as a lookup key to perform a matching search in other target bitmap data blocks participating in the intersection operation. This step specifically operates on target bitmap data blocks belonging to dense bitmap containers based on continuous bit storage.

[0078] Since the dense bitmap container is implemented as a contiguous memory array structure, it supports random bit access with constant time complexity. The system directly maps the set bit index value to the physical memory address of the corresponding bit in the contiguous memory array and reads the bit value at that address. If the value of the bit is 1, it means that the position index is also set in the target bitmap data block corresponding to the dense bitmap container, and the target object corresponding to the position index satisfies the currently checked attribute conditions. The set bit index is retained for the next round of matching search or as a candidate result. If the value of the bit is 0, it means that the position index is not set in the dense bitmap container, and the target object corresponding to the position index does not simultaneously satisfy all attribute conditions. The set bit index is discarded. If there are more than two target bitmap data blocks participating in the intersection operation, the system continues to perform matching searches in other target bitmap data blocks one by one with the currently retained set bit indexes, and only retains the set bit indexes that have been successfully matched in all checked bitmap data blocks each time.

[0079] After the item-by-item matching and searching steps described above, all the bit indices retained to the end constitute the result set of the intersection operation. Each bit index in this set corresponds to a target object that simultaneously satisfies all attribute conditions in the set matching request. The system outputs this set of bit indices as the result of the intersection operation, which can be used for subsequent combination operations or directly returned as the target bit set.

[0080] Based on the aforementioned publicly available solutions, it is known that in existing technologies, when all bitmap data blocks adopt a unified storage structure, intersection operations typically require traversing all bits of a complete bitmap data block, or simultaneously traversing all participating bitmap data blocks in a symmetrical manner. This symmetrical traversal strategy generates a large number of invalid bit-checking operations when dealing with the intersection operation between an extremely sparse bitmap data block and an extremely dense bitmap data block, resulting in a severe waste of computational resources. In this solution, an asymmetric traversal strategy is employed, using a sparse bitmap container based on block compression as the query index source and a dense bitmap container based on continuous bit storage as the search target. This compresses the number of iterations in the intersection operation from being proportional to the total length of the bitmap data blocks to being proportional only to the number of set bits in the sparse bitmap data block. When the number of set bits in the sparse bitmap data block is extremely small, the number of matching searches required is also extremely small, thus significantly reducing the computational complexity of the intersection operation. This optimization strategy can only be implemented on an architecture where heterogeneous bitmap containers coexist in the same system.

[0081] In one or more embodiments of this disclosure, such as Figure 6 This is a flowchart illustrating the bit set generation method provided in an embodiment of this disclosure. Figure 6As shown, the target object is parsed hierarchically according to multiple attribute dimensions, and a bit set of bitmap data blocks is constructed from the obtained attribute dimensions and attribute values. The steps include: Step S601: Parsing the multiple attribute dimensions of the target object layer by layer according to a preset attribute hierarchy to determine the attribute values ​​of each level's attribute dimensions. Step S602: For each level's attribute dimension, constructing a bitmap data block using the corresponding attribute value as an index. Step S603: Constructing a bit set from multiple bitmap data blocks of the same or different levels.

[0082] The preset attribute hierarchy mentioned here refers to the hierarchical relationship definition pre-configured by the system to describe the organizational structure between various attribute dimensions of the target object. This hierarchy specifies the order and logical hierarchy when the system parses multiple attribute dimensions. The preset attribute hierarchy can be set to classify attribute dimensions according to business scenarios, such as organizing attribute dimensions into categories like static attributes, dynamic behavioral attributes, and consumption preference attributes. It should be noted that this hierarchy describes the organizational logic of attribute dimensions during the parsing process, rather than a strict physical hierarchical dependency between attribute dimensions.

[0083] The layer-by-layer parsing mentioned here refers to the process by which the system, according to a preset attribute hierarchy, independently parses and processes the attribute dimensions contained in each level, extracting all possible attribute values ​​for each attribute dimension within that level. Layer-by-layer parsing ensures that the attribute dimension data of each level is extracted and processed independently, and the parsing processes of different levels are logically independent of each other.

[0084] In practical applications, after obtaining the complete set of data for the target object, multiple attribute dimensions are parsed layer by layer according to a preset attribute hierarchy. Specifically, the preset attribute hierarchy configuration is accessed first, which defines the hierarchy to which each attribute dimension belongs and the parsing order of each hierarchy. Following this parsing order, starting from the first hierarchy, the attribute dimensions contained in each hierarchy are processed sequentially.

[0085] For the currently parsing level, the system retrieves all attribute dimensions contained within that level. For each attribute dimension, it iterates through all attribute values ​​appearing under that attribute dimension in the entire target object set, associates that attribute dimension with that level, and records each attribute value appearing under that attribute dimension as the corresponding attribute value set for that level. After completing the parsing of the current level, the system moves to the next level and repeats the above process until all attribute dimensions in all levels have been parsed. Through layer-by-layer parsing, the system establishes a hierarchical mapping structure of attribute dimensions and attribute values, where each level corresponds to a set of attribute dimensions, and each attribute dimension corresponds to a set of attribute values.

[0086] After completing the layer-by-layer parsing and determining the attribute dimensions and attribute values ​​for each level, the system begins to construct bitmap data blocks. For each attribute value under each attribute dimension in each level, the system uses that attribute value as an index to construct a uniquely corresponding bitmap data block.

[0087] The specific construction process is as follows: First, determine the value range of the object identifier in the target object set. Allocate a bit sequence with a length sufficient to cover this value range as storage space for the bitmap data block, and initialize all bits of this bit sequence to zero. The system iterates through the data corresponding to this attribute dimension in the target object set. For each target object, it checks whether its value in this attribute dimension matches the attribute value currently being constructed. If they match, the system sets the bit position of the bit corresponding to the object identifier of the target object in the bitmap data block to 1; otherwise, the bit remains 0. The bit value of each bit in the bitmap data block directly represents whether the corresponding target object possesses the attribute value: a bit value of 1 indicates possession, and a bit value of 0 indicates non-possession.

[0088] The above construction operation is performed on all attribute values ​​under all attribute dimensions in all levels, and each "attribute dimension: attribute value" combination yields an independent bitmap data block.

[0089] After constructing all bitmap data blocks, the system incorporates the bitmap data blocks corresponding to all attribute values ​​under all attribute dimensions within the same level into the bitmap data block set for that level. Simultaneously, the system logically organizes and manages all bitmap data blocks from different levels in a unified manner, collectively forming a complete bit set. This bit set encompasses all set information of the entire target object set under all attribute dimensions and attribute value combinations, providing a complete data foundation for subsequent data density calculations, heterogeneous bitmap container selection, and set matching.

[0090] Based on the aforementioned publicly available solutions, it is known that by parsing multiple attribute dimensions of the target object layer by layer through a pre-defined attribute hierarchy, the multi-dimensional attribute data, which may originally have complex and overlapping relationships, is decoupled into independent bitmap data blocks organized hierarchically. Each bitmap data block is constructed using attribute values ​​as indexes, allowing each bitmap data block to independently describe a set of objects under a specific attribute condition. This approach solves the problem of data volume growing exponentially with the number of dimensions due to the Cartesian product combination of all attribute dimensions. Since each bitmap data block exists as an independent data unit after construction, the subsequent system can perform separate analysis and adaptive optimization of the data density characteristics of each bitmap data block without considering the combined effects between dimensions.

[0091] In one or more embodiments of this disclosure, such as Figure 7This is a flowchart illustrating the method for determining a target bitmap data block provided in an embodiment of this disclosure. Figure 7 As shown, in response to a set matching request containing at least one attribute condition, determining the target bitmap data block associated with the matching request includes: Step S701: Performing semantic parsing on the set matching request to determine at least one target attribute dimension and its corresponding target attribute value. Step S702: Based on the target attribute dimension and target attribute value, searching in the stored heterogeneous bitmap container to determine the target bitmap data block corresponding to the target attribute value.

[0092] In practical applications, a set matching request is received from an external source. This request contains at least one attribute condition. The attribute condition can be expressed as an equality condition stating that "the value of a certain attribute dimension is the value of a specified attribute," or it can be a compound conditional expression containing logical operators. For example, the attribute condition "the value of the geographic attribute is Beijing" means that the target object must have the value "Beijing" in the geographic attribute dimension.

[0093] The system performs semantic parsing on the matching request for this set. The parsing process is as follows: First, it identifies all attribute conditions contained in the request and breaks down the complex conditional expression into several atomic conditions based on logical operators. Second, for each atomic condition, it extracts the specified attribute dimension name as the target attribute dimension. Third, it extracts the specified attribute value from the atomic condition as the target attribute value corresponding to that target attribute dimension. The semantics expressed by this atomic condition are: the value of the target object on the target attribute dimension should be equal to the target attribute value. Finally, if there are logical operations between the atomic conditions, the system records these logical operations for use in subsequent steps. After parsing, the system obtains a structured query description consisting of at least one pair of "target attribute dimension: target attribute value" combinations.

[0094] After obtaining a structured query description, a lookup operation is performed within the stored heterogeneous bitmap container system. Specifically, the system uses each pair of "target attribute dimension: target attribute value" combinations as a lookup index and performs a match in the index mapping table maintained by the heterogeneous bitmap container system. This index mapping table records the mapping relationship between each pair of "attribute dimension: attribute value" combinations and the storage location of its corresponding bitmap data block. This storage location includes the heterogeneous bitmap container type to which the bitmap data block belongs and its specific location information within that container.

[0095] Furthermore, based on the search results, a bitmap data block uniquely corresponding to each target attribute value is identified, and these bitmap data blocks are marked as target bitmap data blocks. Since the bitmap data blocks corresponding to different attribute values ​​have been allocated to different types of heterogeneous bitmap containers during the storage phase, the target bitmap data blocks identified in this step may belong to different types of heterogeneous bitmap containers.

[0096] Based on the aforementioned publicly available solutions, semantic parsing transforms set matching requests into structured query elements, which are then searched within a stored heterogeneous bitmap container system using a unified index. This allows target bitmap data blocks belonging to different heterogeneous bitmap container types to be retrieved and identified simultaneously and efficiently. This solution breaks away from the homogeneous data retrieval pattern inherent in existing technologies due to their unified storage structure, establishing a data access mechanism capable of transparent searching within heterogeneous storage systems. It provides essential retrieval support for cross-container type hybrid bit operations.

[0097] In one or more embodiments of this disclosure, such as Figure 8 This is a flowchart illustrating the target object data update method provided in an embodiment of this disclosure. Figure 8 As shown, when the target object data is updated, the method further includes: Step S801: Monitoring the data density change of the stored bitmap data block. Step S802: If the data density of the bitmap data block changes from being greater than a preset density threshold to not being greater than a preset density threshold, then the bitmap data block is migrated from a dense bitmap container based on continuous bit storage to a sparse bitmap container based on block compression. Step S803: If the data density of the bitmap data block changes from not being greater than a preset density threshold to being greater than a preset density threshold, then the bitmap data block is migrated from a sparse bitmap container based on block compression to a dense bitmap container based on continuous bit storage.

[0098] The system monitors for changes to target object data. When an operation to add, delete, or modify target object data occurs, the system determines the attribute dimensions and attribute values ​​affected by the change, and then locates the affected stored bitmap data block. The system marks the affected bitmap data block as an object to be checked for density changes.

[0099] For bitmap data blocks marked as pending inspection, data density changes are monitored. For example, the historical non-zero bit data density values ​​of the bitmap data block are obtained before the target object data is updated; after the target object data is updated, the current non-zero bit data density of the bitmap data block is recalculated. The non-zero bit data density before the update is compared with the non-zero bit data density after the update to determine whether the non-zero bit data density of the bitmap data block has crossed a preset density threshold.

[0100] Specifically, crossing the preset density threshold includes two scenarios. The first scenario is: the non-zero bit density before the update is greater than the preset density threshold, while the non-zero bit density after the update becomes less than or equal to the preset density threshold, i.e., changing from a dense type to a sparse type. The second scenario is: the non-zero bit density before the update is less than or equal to the preset density threshold, while the non-zero bit density after the update becomes greater than the preset density threshold, i.e., changing from a sparse type to a dense type. If the non-zero bit density before and after the update is both greater than the preset density threshold, or both are less than or equal to the preset density threshold, it indicates that the density type has not changed, and no container migration operation is triggered; only the data content of the bitmap data block is updated.

[0101] The following will describe different situations through specific embodiments.

[0102] The first scenario involves migrating from a dense bitmap container to a sparse bitmap container. Specifically, if monitoring results indicate that the density of non-zero bits in a bitmap data block changes from being greater than a preset density threshold to being no greater than a preset density threshold (i.e., the density type of the bitmap data block changes from dense to sparse), the system migrates the bitmap data block from a dense bitmap container based on continuous bit storage to a sparse bitmap container based on block compression. The specific migration process includes: First, reading the complete bit sequence data of the bitmap data block from the source container. The source container (i.e., the container before the change) is a dense bitmap container based on continuous bit storage, and its underlying data structure is a contiguous memory array. All bit information of the bitmap data block is obtained through direct memory read. Second, the system performs format conversion on the bitmap data block according to the data format of the target container. The target container is a sparse bitmap container based on block compression. The system divides the bitmap data block's bit sequence into multiple logical data blocks according to a preset block size. It checks the set bits in each logical data block, generates null tags for all-zero data blocks, and performs in-container encoding compression on sparse data blocks containing set bits to form a compact, compressed representation. Finally, the format-converted compressed data is written into the target container, releasing the contiguous memory space originally occupied by the bitmap data block in the source container, and updating the container type in the bitmap data block's metadata identifier to a sparse bitmap container based on block compression.

[0103] The second scenario: migration from a sparse bitmap container to a dense bitmap container. Specifically, if monitoring results indicate that the density of non-zero bits in a bitmap data block changes from no more than a preset density threshold to more than a preset density threshold, meaning the density type of the bitmap data block changes from sparse to dense, the system migrates the bitmap data block from a sparse bitmap container based on block compression to a dense bitmap container based on continuous bit storage.

[0104] The migration process is as follows: First, the compressed data of the bitmap data block is read from the source container (i.e., the original container before the change). The source container is a sparse bitmap container based on block compression; the compressed encoded data and null marker information corresponding to the bitmap data block are read. Second, the compressed data is decompressed to restore it to a complete bit sequence representation. Third, the bitmap data block is format-converted according to the data format of the target container. The target container is a dense bitmap container based on continuous bit storage. The system allocates a contiguous physical memory space and writes the decompressed bit sequence into this contiguous memory space in index order, forming a continuous memory array structure for storing the bitmap data block. Finally, the system writes the format-converted data into the target container, releases the compressed data storage space originally occupied by the bitmap data block in the source container, and updates the container type in the metadata identifier of the bitmap data block to a dense bitmap container based on continuous bit storage.

[0105] It should be noted that for the scenarios in Case 1 and Case 2 where migration is not triggered—that is, when the density of non-zero bits remains within the same type range before and after the update—no container migration operation is performed. For bitmap data blocks that remain in a dense type, the system directly modifies the set state of the corresponding bits in the contiguous memory array; for bitmap data blocks that remain in a sparse type, the system updates the position index record of the corresponding set bit in a compressed state, or re-encodes the affected logical data blocks according to the changes.

[0106] This embodiment establishes a dynamic container migration mechanism based on data density change monitoring, enabling the system to maintain the heterogeneous bitmap container type used by each bitmap data block and match its current actual data density characteristics during the continuous updating of target object data.

[0107] Based on the publicly available solutions described above, through dynamic migration, the system can continuously monitor the data density changes of each bitmap data block during runtime and automatically trigger container migration when a qualitative change occurs in the density type. This ensures that the bitmap data block is always stored using a heterogeneous bitmap container that optimally matches its current data density characteristics. In other words, throughout the entire data lifecycle, the system can provide storage and computation efficiency adapted to the current density characteristics of each bitmap data block, guaranteeing both high access performance for dense data and maintaining a high compression ratio for sparse data, thus achieving a dynamic optimal balance between storage resource utilization and computational efficiency.

[0108] In another embodiment of this disclosure, to improve the parallel processing efficiency of query matching, the identifier space of the target object is divided into multiple partitions, each partition corresponding to a continuous subrange of the identifier space. A Bloom filter is established for each partition to record the object identifiers present in that partition. Before performing mixed bitwise operations, the Bloom filter is used to prune each partition, filtering out partitions that do not contain any objects in the target bitmap data block, and performing mixed bitwise operations across container types only on the partitions determined to be matched. In the bit matching calculation within the partition, Single Instruction Multiple Data (SIMD) instructions are used to accelerate the 64-bit word aligned bitwise operations to further improve computational throughput.

[0109] In another embodiment of this disclosure, to optimize storage resource utilization and accelerate access to hot data, a heterogeneous multi-level caching system bound to container type is established. Specifically, decompressed copies of bitmap data blocks using dense bitmap containers based on contiguous bit storage and accessed more frequently than a preset hotness threshold are stored in the local memory layer; compressed data using sparse bitmap containers based on block compression is stored in the distributed cache layer; and persistent data of the full bitmap data blocks is stored in the cold storage layer. A hybrid cache replacement strategy based on Least Recently Used (LRU) and Least Frequently Used (LFU) is adopted to perform data migration and backfilling between the local memory layer, the distributed cache layer, and the cold storage layer, so that hot data remains in the high-speed cache, while cold data is slumped into low-cost storage.

[0110] In another embodiment of this disclosure, to support stable processing in high-concurrency scenarios, a partition scheduling thread pool is constructed based on the aforementioned partitions. An independent scheduling thread is allocated to each partition to execute pipelined task processing, including decompression, bitwise operations, merging, and result writing back. When the number of tasks to be processed in the task queue exceeds a preset backpressure threshold, a backpressure mechanism is triggered to control the rate at which upstream matching requests are delivered, ensuring stable system operation with low latency and high throughput.

[0111] In another embodiment of this disclosure, a monitoring and recovery mechanism is employed to ensure data reliability in the event of a system failure. Consistency checks are periodically performed between cached data and backend persistent snapshots. Each change operation to a bitmap data block is recorded using a Write-Ahead Log (WAL), and incremental snapshots are generated periodically. When a node fails, the bitmap data blocks are quickly restored to a consistent state by replaying the WAL and loading the most recent snapshot.

[0112] Based on any of the above embodiments, this disclosure also provides an object set matching device. Figure 9 This is a schematic block diagram illustrating the structure of an object set matching device according to one embodiment of this disclosure. Figure 9 As shown, the object set matching device includes: a receiving module 901, used to perform hierarchical parsing of the target object according to multiple attribute dimensions, and construct a bit set of bitmap data blocks from the obtained attribute dimensions and attribute values; a selection module 902, used to select the corresponding heterogeneous bitmap container for storage according to the data density of each bitmap data block in the bit set; a determining module 903, used to determine the target bitmap data block associated with the matching request in response to a set matching request containing at least one attribute condition; and a calculation module 904, used to perform mixed bit operations on target bitmap data blocks belonging to different heterogeneous bitmap container types by determining the corresponding bit operation strategy according to the target heterogeneous bitmap container type to which the target bitmap data block belongs, to obtain the target bit set.

[0113] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0114] The entity executing the information sending method in the specific embodiments of this disclosure may be an electronic device such as a server (including a local server or a cloud computing platform).

[0115] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the object set matching method of any of the embodiments described above.

[0116] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure.

[0117] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0118] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0119] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0120] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0121] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0122] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0128] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for matching object sets, characterized in that, The method includes: The target object is parsed hierarchically according to multiple attribute dimensions, and the bit set of the bitmap data block is constructed from the obtained attribute dimensions and attribute values; Based on the data density of each bitmap data block in the bit set, select the corresponding heterogeneous bitmap container for storage; In response to a set matching request containing at least one attribute condition, determine the target bitmap data block associated with the matching request; By determining the corresponding bit operation strategy based on the target heterogeneous bitmap container type to which the target bitmap data block belongs, hybrid bit operations are performed on the target bitmap data blocks belonging to different heterogeneous bitmap container types to obtain the target bit set.

2. The object set matching method according to claim 1, characterized in that, The step of selecting a corresponding heterogeneous bitmap container for storage based on the data density of each bitmap data block in the bit set includes: Calculate the non-zero bit data density of the bitmap data block; If the data density is greater than a preset density threshold, then a dense bitmap container based on continuous bit storage is selected as the heterogeneous bitmap container for storage. The dense bitmap container is a bitmap container with a contiguous memory array structure.

3. The object set matching method according to claim 1, characterized in that, The step of selecting a corresponding heterogeneous bitmap container for storage based on the data density characteristics of each bitmap data block in the bit set includes: Calculate the non-zero bit data density of the bitmap data block; If the data density is not greater than a preset density threshold, then a sparse bitmap container based on block compression is selected as the heterogeneous bitmap container, and the bitmap data block is stored after being compressed in blocks. The sparse bitmap container is a bitmap container that is divided according to a preset data block size and the sparse data blocks are compressed using in-container encoding.

4. The object set matching method according to claim 1, characterized in that, The step involves determining the corresponding bit operation strategy based on the target heterogeneous bitmap container type to which the target bitmap data block belongs, and performing mixed bit operations on the target bitmap data blocks belonging to different heterogeneous bitmap container types to obtain a target bit set, including: Identify the heterogeneous bitmap container type to which the target bitmap data block belongs; If the target bitmap data block belongs to a sparse bitmap container based on block compression, then the first bit operation logic adapted to the block compression rule is used to directly perform bit matching calculation in the compressed state. If the target bitmap data block belongs to a dense bitmap container based on continuous bit storage, then the second bit operation logic is used to perform bit matching calculation on the target bitmap data block; Based on the logical operation relationships contained in the set matching request, the bit matching calculation results corresponding to different heterogeneous bitmap container types are combined to obtain the target bit set.

5. The object set matching method according to claim 4, characterized in that, When the logical operation relationship is an intersection operation, the combination operation methods include: Identify the target bitmap data block belonging to the sparse bitmap container based on block compression as the source of the query index; Traverse the query index source and extract the set bit indexes; Based on the extracted bit index, a matching search is performed at the corresponding position in the target bitmap data block belonging to the dense bitmap container based on continuous bit storage; The set of set bit indices found in the dense bitmap container based on continuous bit storage is used as the result of the intersection operation.

6. The object set matching method according to claim 1, characterized in that, The step of performing hierarchical parsing of the target object according to multiple attribute dimensions, and constructing a bit set of bitmap data blocks from the obtained attribute dimensions and attribute values, includes: According to the preset attribute hierarchy, the multiple attribute dimensions of the target object are parsed layer by layer to determine the attribute values ​​of each attribute dimension at each level. For each level of attribute dimension, the bitmap data block is constructed using the corresponding attribute value as an index; The bit set is constructed from multiple bitmap data blocks at the same or different levels.

7. The object set matching method according to claim 1, characterized in that, The step of determining the target bitmap data block associated with the matching request in response to a set matching request containing at least one attribute condition includes: The set matching request is semantically parsed to determine at least one target attribute dimension and its corresponding target attribute value; Based on the target attribute dimension and target attribute value, a search is performed in the stored heterogeneous bitmap container to determine the target bitmap data block corresponding to the target attribute value.

8. The object set matching method according to claim 1, characterized in that, When the target object data is updated, the method further includes: Monitor changes in the data density of stored bitmap data blocks; If the data density of the bitmap data block changes from being greater than a preset density threshold to being no greater than a preset density threshold, then the bitmap data block is migrated from a dense bitmap container based on continuous bit storage to a sparse bitmap container based on block compression. If the data density of the bitmap data block changes from not greater than a preset density threshold to greater than a preset density threshold, then the bitmap data block is migrated from a sparse bitmap container based on block compression to a dense bitmap container based on continuous bit storage.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as, A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A readable storage medium storing executable instructions, the executable instructions being executed by a processor according to any one of claims 1 to 8.

Images