Data query caching method and related device

Abstract

Embodiments of the present application provide a data query caching method and related equipment. The data query caching method comprises: establishing a corresponding global bitmap index for each target data identifier, the target data identifier being a data identifier participating in rule matching; constructing a corresponding cache key-value pair structure for each target data identifier according to the global bitmap index; in response to a rule query request, extracting each field condition contained therein, and obtaining a corresponding data identifier bitmap according to the cache key-value pair structure; filtering each data identifier bitmap using logical operations to generate a candidate target set; and performing final business rule determination based on the candidate target set. The technical solution of the embodiments of the present application can quickly locate a data identifier set meeting the conditions in the rule query stage by establishing a global bitmap index in combination with a cache key-value pair structure, thereby reducing repeated access to the underlying storage system, improving rule execution efficiency, and reducing system resource consumption.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a data query caching method and related equipment. Background Technology

[0002] In rule engine query scenarios based on multidimensional data, existing caches are usually built up gradually by relying on historical accesses. When a rule is executed for the first time, it still needs to access the underlying storage, resulting in high response latency. At the same time, batch queries of multiple fields are prone to generating a large number of duplicate retrievals, increasing storage pressure. High cardinality fields are also prone to causing the number of cache keys to expand, affecting cache space utilization. Summary of the Invention

[0003] The embodiments of this application provide a data query caching method and related devices, which can at least to some extent organize query results based on a unified index and efficiently filter and prune candidates for multi-field conditions during rule execution.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a data query caching method is provided, comprising: establishing a corresponding global bitmap index for each target data identifier, wherein the target data identifier is a data identifier participating in rule matching; constructing a corresponding cache key-value pair structure for each target data identifier according to the global bitmap index; in response to a rule query request, extracting the field conditions contained therein, and obtaining the corresponding data identifier bitmap according to the cache key-value pair structure; performing filtering processing on each data identifier bitmap using logical operations to generate a candidate target set; and performing a final business rule determination based on the candidate target set.

[0006] According to one aspect of the embodiments of this application, a data query caching device is provided, the data query caching device comprising: a bitmap index building module, configured to build a corresponding global bitmap index for each target data identifier, wherein the target data identifier is a data identifier participating in rule matching; a key-value pair structure building module, configured to construct a corresponding cache key-value pair structure for each target data identifier based on the global bitmap index; a field condition extraction module, configured to extract each field condition contained therein in response to a rule query request, and obtain a corresponding data identifier bitmap based on the cache key-value pair structure; a bitmap filtering processing module, configured to perform filtering processing on each data identifier bitmap using logical operations to generate a candidate target set; and a business rule determination module, configured to perform a final business rule determination based on the candidate target set.

[0007] According to one aspect of the embodiments of this application, a computer program product is provided, including one or more computer programs that, when executed by one or more processors, implement the steps of the data query caching method as described above.

[0008] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the data query caching method as described in the above embodiments.

[0009] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the data query caching method as described in the above embodiments.

[0010] In some embodiments of this application, by establishing a global bitmap index and combining it with a cached key-value pair structure, this method can quickly locate the set of data identifiers that meet the conditions during the rule query stage, reduce repeated access to the underlying storage system, improve rule execution efficiency, and reduce system resource consumption.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0013] Figure 2 The diagram shows a flowchart of a data query caching method provided in an embodiment of this application.

[0014] Figure 3 It shows according to Figure 2 A flowchart illustrating a specific implementation of step S300 in the data query caching method shown in the corresponding embodiment.

[0015] Figure 4 It shows according to Figure 2 A flowchart illustrating a specific implementation of step S400 in the data query caching method shown in the corresponding embodiment.

[0016] Figure 5 A schematic diagram of the structure of a data query caching device provided in an embodiment of this application is shown.

[0017] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0020] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0021] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0022] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0023] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0024] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0025] Users can interact with server 105 via network 104 using terminal devices to receive or send messages, etc. Server 105 can be a server providing various services. For example, a user can upload target data identifiers, rule query requests, etc. to server 105 using terminal device 103 (or terminal device 101 or 102). Server 105 can establish a corresponding global bitmap index for each target data identifier, where the target data identifier is the data identifier participating in rule matching; based on the global bitmap index, it constructs a corresponding cache key-value pair structure for each target data identifier; in response to the rule query request, it extracts the field conditions contained therein, obtains the corresponding data identifier bitmap based on the cache key-value pair structure; it performs filtering processing on each data identifier bitmap using logical operations to generate a candidate target set; and based on the candidate target set, it executes the final business rule determination.

[0026] It should be noted that the data query caching method provided in this application embodiment is generally executed by server 105, and correspondingly, the data query caching device is generally set in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the data query caching scheme provided in this application embodiment.

[0027] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 2 A flowchart of a data query caching method according to an embodiment of this application is shown. This data query caching method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this data query caching method includes at least the following: S100, establish a corresponding global bitmap index for each target data identifier, wherein the target data identifier is the data identifier participating in rule matching.

[0028] S200, construct a corresponding cache key-value pair structure for each of the target data identifiers based on the global bitmap index.

[0029] S300, in response to the rule query request, extract the field conditions contained therein, and obtain the corresponding data identifier bitmap according to the cache key-value pair structure.

[0030] S400, using logical operations to filter each of the data identifier bitmaps to generate a candidate target set.

[0031] S500, based on the candidate target set, perform the final business rule determination.

[0032] The embodiments of this application construct a global bitmap index and a cached key-value pair structure corresponding to the target data identifier. During the rule query stage, bitmap logical operations are used to quickly generate a candidate target set, and business rule judgment is performed based on the candidate target set. This reduces the number of full data matching and storage accesses, improves rule execution efficiency, reduces system resource consumption, and enhances query response capabilities in large-scale data scenarios.

[0033] In S100, the target data identifier refers to the unique identifier of the data object participating in the rule matching process. The target data identifier can be a file digest value, file fingerprint, user identifier, device identifier, account identifier, transaction identifier, sample identifier, or other data identifier that can uniquely represent the business object.

[0034] To enable rapid calculations in subsequent rule query processes, a unified bitmap indexing system is first established for the target data identifiers participating in rule matching. Specifically, this step obtains all target data identifiers to be matched and establishes a corresponding bitmap position mapping relationship for each target data identifier, ensuring that each target data identifier corresponds to a unique bitmap position.

[0035] In some embodiments, bitmap positions can be generated using sequential numbering; in other embodiments, they can be generated using hash mapping, interval partitioning, fragmented allocation, or dynamically extended numbering. By establishing a unified global bitmap index, data sets corresponding to different field conditions can all be mapped to the same bitmap coordinate system, thereby providing a unified data foundation for subsequent bitmap operations.

[0036] Specifically, in some embodiments, the specific implementation of step S100 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S100 in the data query caching method shown in the corresponding embodiment, step S100 in the data query caching method may include the following steps: Assign a corresponding continuous bitmap position to each target data identifier.

[0037] A global bitmap index is generated based on the mapping relationship between each target data identifier and the position of the continuous bitmap.

[0038] This embodiment assigns a unique and continuous bitmap position to each target data identifier and establishes a unified mapping relationship, which can form a standardized bitmap coordinate system. This enables data sets corresponding to different field conditions to perform fast logical operations in the same dimension, improves bitmap calculation efficiency and index management efficiency, and provides a unified data foundation for subsequent cache construction and candidate set generation.

[0039] In this embodiment, the global bitmap index serves as the data infrastructure of the entire cache query system. Its function is to establish a unified positional mapping relationship for all target data identifiers participating in rule matching, thereby ensuring that all subsequently generated bitmaps of various data identifiers can be represented and calculated based on the same coordinate system. The global bitmap index can be understood as a correspondence table between target data identifiers and bitmap positions. This correspondence table records the unique position of each target data identifier in the bitmap and is used in subsequent bitmap generation, bitmap operations, and bitmap deserialization processes.

[0040] Specifically, the first step is to obtain the set of target data identifiers for rule matching. These target data identifiers can originate from the data warehouse, sample database, file database, log database, object storage system, or real-time data stream system managed by the rule engine. Target data identifiers can be MD5 values, SHA values, file fingerprints, device identifiers, account identifiers, user identifiers, business object identifiers, or other data identifiers that can uniquely represent the target object. After obtaining the set of target data identifiers, a corresponding consecutive bitmap position is assigned to each target data identifier in the set. Here, a consecutive bitmap position refers to the position number arranged sequentially in a preset order within the bitmap space. For example, bitmap positions can be assigned sequentially in the order of 0, 1, 2, 3…N, or in the order of 1, 2, 3, 4…N.

[0041] In some embodiments, target data identifiers can be assigned consecutive numbers according to the order in which they enter the system. When a new target data identifier enters the system, the system automatically appends a new bitmap position after the currently assigned position and establishes a corresponding mapping relationship. This method is suitable for application scenarios where the target data scale continues to grow and the update frequency is high.

[0042] In other embodiments, the target data identifiers can be sorted lexicographically, and consecutive bitmap positions can be allocated sequentially based on the sorting results. By pre-sorting the target data identifiers, the global bitmap index can be kept consistent across different nodes, which is beneficial for bitmap synchronization and bitmap sharing in a distributed environment.

[0043] In some other embodiments, continuous bitmap positions can be allocated according to preset partitioning rules. For example, different data partitions can be created based on business type, data source, risk level, time interval, or organization, and bitmap positions within each partition can be established using a sequential numbering method. This approach can reduce the management complexity of a single bitmap space in ultra-large-scale data scenarios and improve index expansion capabilities.

[0044] In other embodiments, a combination of hash mapping and continuous position mapping can be used to establish bitmap positions. Specifically, the target data identifier is first hashed, and the partition to which it belongs is determined based on the hash result. Then, continuous numbering is performed within the corresponding partition. This method can improve the index building efficiency in massive data scenarios while ensuring bitmap continuity.

[0045] After completing the bitmap position allocation, a global bitmap index is generated based on the mapping relationship between the target data identifier and the continuous bitmap positions.

[0046] Specifically, a one-to-one correspondence is established between target data identifiers and bitmap positions. Each target data identifier corresponds to only one bitmap position, and each bitmap position corresponds to only one target data identifier, thus forming a bidirectional mapping structure. In subsequent applications, the corresponding bitmap position can be quickly located using the target data identifier, and the corresponding target data identifier can be quickly reconstructed using the bitmap position.

[0047] In some embodiments, the global bitmap index can be stored using a mapping table structure. The mapping table records the target data identifier field and the bitmap location field, enabling fast lookup through key-value mapping relationships.

[0048] In other embodiments, the global bitmap index can also be stored using a hash index structure. The system uses the target data identifier as the index key and stores the corresponding bitmap position as the index value in the hash structure to improve the efficiency of bitmap position lookup.

[0049] In other embodiments, the global bitmap index can also be stored using data structures such as B-tree index, skip list index, LSM tree index, or inverted index, thereby adapting to data environments of different sizes and access patterns.

[0050] In some embodiments, to adapt to scenarios where the number of target data identifiers changes dynamically, dynamic maintenance operations can be performed on the global bitmap index. When a new target data identifier enters the system, new consecutive bitmap positions are automatically allocated and the mapping relationship is updated; when some target data identifiers become invalid or are deleted, the original bitmap positions can be retained and invalidation flags can be set, or the global bitmap index can be regenerated through an index reconstruction mechanism. In some embodiments, to avoid frequent bitmap structure reconstruction, a logical invalidation method is preferably used to manage deleted target data identifiers, that is, the corresponding bitmap positions are retained but they do not participate in subsequent rule matching calculations.

[0051] In distributed deployment scenarios, the global bitmap index can also be managed using a master node maintenance and slave node synchronization approach. The master node is responsible for bitmap position allocation and mapping relationship maintenance, while slave nodes periodically retrieve the latest index data, thereby ensuring that the bitmap coordinate system used by different computing nodes remains consistent. In another implementation, a consistency protocol can be used to maintain the global bitmap index, enabling multiple nodes to jointly complete index management and synchronization updates.

[0052] The global bitmap index established in the above manner provides a unified positional reference for the data identifier bitmaps in the subsequent cached key-value pair structure, enabling the data sets corresponding to any field condition to be mapped to the same bitmap space. The subsequently obtained data identifier bitmaps can then be directly subjected to bitwise logical operations without requiring additional data alignment processing, thereby significantly reducing the computational complexity of the rule query process and improving data filtering and rule execution efficiency.

[0053] In S200, the cache key-value pair structure is used to record the data identifier set information corresponding to different query conditions, where the cache key is used to represent the query condition and the cache value is used to represent the data identifier set that satisfies the corresponding query condition.

[0054] This step constructs cache entries based on the field information involved in rule execution and converts target data identifiers that meet the corresponding field conditions into corresponding data identifier bitmaps for storage based on the global bitmap index. Each bit in the data identifier bitmap corresponds to a target data identifier in the global bitmap index. When the corresponding target data identifier meets the current field condition, its corresponding bit is marked as valid; otherwise, it is marked as invalid. The cache key-value pair structure can be stored in an in-memory cache system, a distributed cache system, a local high-speed cache, or other data storage structures that support fast access. By pre-establishing the cache key-value pair structure, subsequent rule queries can directly utilize the cache to retrieve the target data set without frequently accessing the underlying storage system.

[0055] Specifically, in some embodiments, the specific implementation of step S200 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S200 in the data query caching method shown in the corresponding embodiment, step S200 in the data query caching method may include the following steps: For each rule, a corresponding cache key is generated for the field conditions involved. The cache key is composed of at least a rule identifier, a field name, and a field value bitmap.

[0056] Based on the global bitmap index, a corresponding cache value is generated, which forms a cache key-value pair structure with the cache key. The cache value is a data identifier bitmap.

[0057] This embodiment combines rule identifiers, field names, and field value bitmaps into a cache key, and uses the data identifier bitmap as the cache value. This can control the size of cache entries while ensuring the accuracy of query positioning, avoid the expansion of the number of keys caused by directly creating keys with the original field values, and improve the data reuse capability under multi-field conditions.

[0058] In this embodiment, field conditions refer to the conditions used to filter target data identifiers during rule querying, typically including field name, field value, and necessary rule context information. To ensure that the cache key can uniquely identify a field condition, the cache key is at least composed of a rule identifier, field name, and a field value bitmap. The rule identifier distinguishes the cache space corresponding to different rules, the field name distinguishes different business attributes, and the field value bitmap converts the original field values ​​into a bitmap representation. The field value bitmap can be understood as a compressed bitmap representation of the field value set, used to replace directly using the original field values ​​to construct the cache key, thereby avoiding the generation of a large number of discrete key values ​​from high-cardinality segments. The field value bitmap can be generated using hash bitmap mapping, compressed bitmap encoding, segmented bitmap encoding, Bloom filter bitmap, or other methods that can map the field value set into a bitmap representation.

[0059] In some embodiments, the rule identifier can be a rule number, rule name, rule version number, rule template number, or a combination thereof. By introducing a rule identifier, different rules, even those involving the same field name and the same field value, can form different cache keys, thereby avoiding cross-rule false hits. The field name can be a risk level, file type, source channel, behavior category, time interval, geographic attribute, device attribute, or other business fields used for rule matching. The field name, as an important component of the cache key, is used to ensure logical isolation between different field conditions. The field value bitmap can be automatically generated based on the range of field values. For low-cardinality fields, multiple possible values ​​can be mapped to a limited number of bitmap positions; for high-cardinality fields, the set of field values ​​can be compressed and encoded to form a compact bitmap representation. This supports single-value conditions, as well as multi-value conditions, range conditions, enumeration conditions, or composite conditions. If a field condition contains multiple values, the system can merge the bitmaps corresponding to the multiple values ​​to form a unified field value bitmap for subsequent fast retrieval.

[0060] After generating the cache key, a corresponding cache value is generated based on the global bitmap index, and the cache value and cache key form a cache key-value pair structure. This cache value is a data identifier bitmap, which records the set of target data identifiers that satisfy the conditions of the field represented by the cache key. Specifically, based on the hit results corresponding to the field conditions, the position of the corresponding target data identifier can be found in the global bitmap index, and these positions are marked as valid identifiers in the bitmap, forming the cache value. The length of the cache value is consistent with the global bitmap index, so cache values ​​corresponding to different field conditions can be directly compared and combined within the same bitmap coordinate system.

[0061] In some embodiments, cached values ​​can be generated on the fly based on the hit set corresponding to field conditions and written to the cache after the first query, so that subsequent queries with the same conditions can be directly reused. In other embodiments, cached values ​​can also be built in advance during system initialization, rule publishing, data preheating, or batch synchronization, thereby improving the first query hit rate and reducing latency caused by cold starts. In still other embodiments, the cache key-value pair structure can be organized in multiple levels. For example, it can be categorized by rule identifier in the first-level index, by field name in the second-level index, and by field value bitmap in the third-level index, thereby improving cache lookup efficiency. The system can also manage the cache key-value pair structure hierarchically according to query frequency, cache size, field cardinality, and expiration time to adapt to different business scenarios.

[0062] Once the cached values ​​are constructed, the cache key-value pair structure can be stored in an in-memory cache, distributed cache, local cache, or hybrid caching environment. Frequently used field conditions can be preferentially retained in the high-speed cache layer; less frequent field conditions can be stored in a larger cache layer. In this way, the caching system can meet high-concurrency access requirements while controlling cache resource consumption.

[0063] In this embodiment, the core significance of the cached key-value pair structure lies in using rule identifiers, field names, and field value bitmaps to jointly determine query conditions, with a data identifier bitmap serving as the unified output result. Thus, when performing rule queries subsequently, the system can quickly obtain the corresponding data identifier bitmap based solely on the cached key, without needing to re-retrieve the underlying storage or repeatedly calculate the hit set. This significantly improves rule query efficiency and provides a stable data foundation for subsequent logical operation filtering processes.

[0064] In S300, a rule query request refers to a query request used to describe target filtering conditions, which typically contains one or more field conditions. Field conditions may include attribute conditions, source conditions, behavior conditions, risk conditions, time conditions, or other conditional information used to describe the characteristics of the target data.

[0065] This step first parses the rule query request, identifies the fields it contains, and then searches for the corresponding cached item in the cache key-value pair structure based on the extracted fields. If a corresponding cached item exists, the data identifier bitmap represented by the corresponding cached value is read. For rule query requests containing multiple fields, the data identifier bitmap corresponding to each field is obtained separately.

[0066] In some embodiments, the data identifier bitmaps for each field condition can be acquired serially; in other embodiments, a parallel acquisition method can be used to load the bitmaps corresponding to multiple field conditions simultaneously, so as to further improve the rule query response speed.

[0067] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2 According to the detailed description of step S300 in the data query caching method shown in the corresponding embodiment, step S300 in the data query caching method may include the following steps: S310, in response to the rule query request, parse the rule query request, extract the field conditions contained therein, and obtain the specific query conditions.

[0068] S320, based on the specific query conditions, obtain the corresponding data identifier bitmap according to the cache key-value pair structure.

[0069] This embodiment parses the rule query request and extracts the specific query conditions, enabling the system to directly locate cached items according to field conditions, reducing irrelevant searches and redundant calculations, and improving cache reading efficiency and rule query response speed.

[0070] In S310, a rule query request refers to the request information used to trigger data filtering and rule matching. It can be passed in the form of structured messages, semi-structured messages, interface parameters, rule expressions, or query commands. To ensure that the subsequent cache reading process has explicit input, it is necessary to identify the conditions of each field from this request and convert them into executable query conditions. Specific query conditions include not only field names and values, but also logical relationships, comparison relationships, value ranges, enumeration sets, and multi-condition combinations between fields. It should be noted that the parsing in this step refers to the process of syntax recognition, field decomposition, condition classification, and logical structure extraction of the rule query request. Its purpose is to convert the externally input rule request into a condition expression form that the system can directly use for querying.

[0071] This step, when parsing rule query requests, can employ methods such as rule template parsing, expression tree parsing, condition item decomposition, structured field reading, or interface parameter mapping. If the rule query request itself is stored in structured field format, the field names, field values, and logical symbols can be read directly. If the rule query request is a string expression or script expression, lexical and semantic analysis can be performed first, and then it can be decomposed into several field conditions. If the rule query request contains nested logical structures, the system can expand it into a hierarchical condition tree or a normalized condition set to ensure that subsequent cached queries can be executed in a unified format. Through the above parsing process, the system obtains the specific query conditions, thereby converting the original request into standardized condition input.

[0072] In some embodiments, a rule query request may contain multiple field conditions, such as risk level conditions, file type conditions, source conditions, behavior conditions, and time conditions. During parsing, it is necessary to identify the field name, field value, and constraint type corresponding to each field condition and register them into the specific query condition set. If there are AND, OR, or NOT logical relationships between field conditions, these relationships should be retained to determine how to retrieve the corresponding data identifier bitmap from the cached key-value pair structure later. For example, for multiple field conditions that must be satisfied simultaneously, the corresponding bitmap can be retrieved according to intersection logic during the query phase; for cases where any field condition is satisfied, the corresponding bitmap can be read according to union logic. If the field conditions include range conditions or set conditions, they can be normalized into multiple equivalent conditions or converted into a unified field value bitmap representation to facilitate matching with the cached key-value pair structure.

[0073] In other embodiments, rule query requests can be validated to ensure the completeness and executability of the extracted field conditions. This validation process includes field validity checks, value range validity checks, logical relationship integrity checks, and rule version consistency checks. If missing field conditions, abnormal field values, or logical errors are found, subsequent cached queries can be refused, or the corresponding conditions can be marked as requiring supplementation before further processing. This validation process prevents erroneous conditions from entering the cache reading stage, improving the stability and accuracy of rule queries.

[0074] In some embodiments, after S310, the method further includes: In response to a partial field condition not being matched in the cached key-value pair structure, the data identifier bitmap corresponding to the missing field condition is retrieved from the preset storage system.

[0075] The acquired data identifier bitmap and the data identifier bitmap obtained from the cache key-value pair structure are used together to perform the logical operation.

[0076] In the event of a cache miss, this embodiment can supplement the corresponding bitmap from the preset storage system and participate in logical operations together with the cache hit bitmap, thereby ensuring the integrity of the query results and taking into account cache efficiency, data real-time performance, and system scalability.

[0077] In this embodiment, the preset storage system refers to the underlying data storage unit used to store the correspondence between target data identifiers and field attributes. This storage system can be a relational database, a distributed key-value database, a document database, a search engine, an object storage system, an offline data warehouse, or a real-time streaming storage system. This storage system is used to supplement the return of data identifier bitmaps corresponding to the missing field conditions when the cache cannot directly provide the target result, thus ensuring the complete execution of the rule query. After parsing the rule query request and obtaining the specific query conditions, the system will access the cached key-value pair structure item by item according to the specific query conditions. If it finds that some field conditions do not match the cached key-value pair structure, it determines that the data identifier bitmaps corresponding to those field conditions need to be retrieved from the source.

[0078] Specifically, if a field condition fails to hit the cache, a corresponding storage query condition is first generated based on that field condition. This storage query condition can remain consistent with the original field condition or be normalized according to the query syntax supported by the storage system. Subsequently, a query request is sent to the preset storage system to retrieve the data identifier bitmap corresponding to the missed field condition. The construction method of this data identifier bitmap is consistent with that of the bitmap in the cache, and its bitmap length is consistent with the global bitmap index, ensuring that bitmaps from different sources can directly participate in subsequent calculations. For the case of a single field failure, only the bitmap corresponding to that single field condition is retrieved from the source; for the case of multiple field conditions failing simultaneously, the source can be retrieved item by item, or multiple missed field conditions can be merged and retrieved in batches to reduce the number of accesses and improve processing efficiency.

[0079] In some embodiments, different origin retrieval methods can be selected based on the type of the missed field condition. If the amount of data corresponding to the field condition is small, an exact query can be used to directly return the target data identifier bitmap from the storage system. If the amount of data corresponding to the field condition is large, the target data identifier set that meets the condition can be read from the storage system first, and then converted into bitmap form based on the global bitmap index. If the storage system supports bitmap indexes, inverted indexes, or pre-aggregated results, the system can also directly read the bitmap results to reduce conversion overhead. This can adapt to application scenarios with different storage architectures and different data scales.

[0080] In other embodiments, asynchronous origin pull processing can also be performed on missed field conditions. Specifically, the current query request first performs partial calculations using the cached bitmap that has already been hit, while simultaneously submitting the missed field conditions to a background task. The background task then retrieves the corresponding data identifier bitmap from a preset storage system and, upon completion, populates the cache key-value pair structure. This approach is suitable for high-concurrency scenarios, reducing the waiting time for foreground requests and gradually improving the cache hit rate for subsequent requests with the same field conditions. For origin pull results, the system can also decide whether to write back to the cache based on the popularity of the field conditions, query frequency, storage cost, and expiration period, thereby achieving a dynamic balance between caching and storage.

[0081] In some embodiments, a priority strategy can be set for missed field conditions. For high-frequency field conditions, frequently used field conditions in rule execution, or field conditions that have a significant impact on the results, priority can be given to retrieving from the preset storage system, and these conditions can immediately participate in subsequent logical operations after the retrieval is completed. For low-frequency field conditions, delayed retrieval, batch retrieval, or phased retrieval methods can be used to control storage access pressure. If multiple field conditions correspond to different storage shards, queries can be initiated in parallel by shard, thereby shortening the overall retrieval time.

[0082] After retrieving the bitmap of the data identifier corresponding to the missed field condition, this bitmap is used together with the bitmap of the data identifier obtained from the cache key-value pair structure to perform logical operations. These logical operations involve bitwise combination of multiple bitmaps to produce a set of data identifiers that simultaneously satisfy multiple field conditions. Since both the cache hit bitmap and the origin bitmap are built based on the same global bitmap index, they are naturally aligned in bitmap position, allowing direct execution of bitwise AND, bitwise OR, bitwise NOT, or combinations thereof operations without additional data format conversion or bitmap rearrangement.

[0083] In some embodiments, the source of the cache-hit bitmap and the source-fetched bitmap can be marked separately before being sent to the logic operation module for processing. This approach facilitates subsequent statistics on cache hit rate, analysis of source-fetching costs, and optimization of cache preheating strategies. In other embodiments, the cache bitmap and the source-fetched bitmap can be merged and organized before calculating the candidate target set in a unified manner to reduce intermediate state storage overhead. If the rule query request contains multiple field conditions, some of which come from the cache and some from the storage system, all bitmaps can be included in the same logic operation process to ensure the consistency and completeness of the results.

[0084] In this embodiment, the purpose of introducing the origin cache miss mechanism is to enable the system to complete rule queries without relying on a full cache hit. Even if some field conditions are not yet cached, the corresponding bitmap can still be supplemented through a preset storage system, and then the candidate target set can be calculated together with the cache results. This retains the advantages of cache acceleration while avoiding query failures caused by cache cold start, cache missing, or the appearance of new fields. At the same time, the origin cache miss results can also promote the gradual improvement of the cache, making subsequent queries with the same conditions more likely to hit the cache, thus forming a closed loop of continuous optimization.

[0085] In step S320, based on the specific query conditions, the corresponding data identifier bitmap is obtained according to the cache key-value pair structure. This cache key-value pair structure is a pre-constructed intermediate result storage structure, where the cache key uniquely identifies a certain type of field condition, and the cache value represents the set of data identifiers that satisfy that condition. The specific query conditions are matched against the cache key, and the corresponding cache value, i.e., the data identifier bitmap, is read from the cache. Since the cache key already contains information such as rule identifiers, field names, and field value bitmaps, the system can quickly locate cached items based on the current query conditions without needing to access the underlying data storage again for a full retrieval.

[0086] In some embodiments, the cached key-value pair structure can be queried item by item according to field conditions. That is, a retrieval key is first constructed for each individual field condition, and then the corresponding data identifier bitmap is read using that retrieval key; if the rule query request contains multiple field conditions, the data identifier bitmaps corresponding to each field condition can be read separately, and these bitmaps can be sent to the subsequent logic operation module for processing. This method is suitable for scenarios with few field conditions and clear query paths.

[0087] In other embodiments, specific query conditions can be standardized into a unified set of query keys before batch caching and matching of this set of query keys. This approach is suitable for scenarios with many field conditions, complex rule expressions, or high concurrency, reducing the number of index accesses per query and improving overall throughput. The query order can also be determined based on the priority, access frequency, or historical hit rate of field conditions. For example, cached values ​​corresponding to high-frequency fields can be read first, followed by cached values ​​corresponding to low-frequency fields, to reduce average query latency.

[0088] In some embodiments, if a specific query condition does not have a corresponding cached item in the cached key-value pair structure, the condition can be marked as a cache miss and handled by a subsequent compensation mechanism. The compensation mechanism can be to retrieve the corresponding data identifier bitmap from the storage system in a subsequent step, or it can trigger cache preheating, asynchronous origin pull, incremental build, or missing item backfilling to ensure the integrity of the query results. For the cache-hit portion, the cached value is directly read and entered into the logical operation stage, thereby reducing the access pressure on the storage system.

[0089] In S400, logical operation refers to the process of combining and filtering multiple data sets based on the logical relationship between corresponding bits in a bitmap.

[0090] Based on the logical relationships between the fields in the rule query request, bitwise AND, bitwise OR, bitwise NOT, bitwise XOR, or combinations thereof can be used for processing. Specifically, bitwise AND is used to obtain data sets that simultaneously satisfy multiple field conditions; bitwise OR is used to obtain data sets that satisfy any field condition; and bitwise NOT is used to exclude data sets corresponding to specific conditions. After logical operation processing, a target candidate bitmap is obtained. Subsequently, the target candidate bitmap is converted into a corresponding target data identifier set based on the global bitmap index, thereby generating a candidate target set. The target data identifiers in the candidate target set represent data objects that have already met the preset conditions in the rule query request; therefore, their number is significantly reduced compared to the full data set.

[0091] Specifically, in some embodiments, the specific implementation of step S400 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 2According to the detailed description of step S400 in the data query caching method shown in the corresponding embodiment, step S400 in the data query caching method may include the following steps: S410, perform bitwise logical operations on each of the data identifier bitmaps to generate target candidate bitmaps.

[0092] S420, the target candidate bitmap is deserialized based on the global bitmap index to generate a candidate target set.

[0093] This embodiment performs bitwise logical operations on multiple data identifier bitmaps and combines them with global bitmap indexes for deserialization, which can quickly generate a set of candidate targets with less computational overhead, reduce the full matching range, and improve rule execution efficiency and cache pruning accuracy.

[0094] In this embodiment, bitwise logical operations refer to combining and operating on bit values ​​at the same position in multiple data identifier bitmaps according to a preset logical relationship to generate a bitmap that can represent the joint hit result of multiple conditions. The target candidate bitmap refers to the intermediate bitmap result obtained after logical operations, where each bit corresponds to a global bitmap position of a target data identifier, used to indicate whether the target data identifier simultaneously satisfies the preset field conditions in the rule query request. Deserialization processing refers to the process of converting the valid bit positions in the target candidate bitmap back into a specific set of data identifiers. This conversion relies on the mapping relationship between bitmap positions and target data identifiers established by the aforementioned global bitmap index.

[0095] In S410, if multiple field conditions in a rule query request belong to a relationship that must be satisfied simultaneously, a bitwise AND operation is performed on each data identifier bitmap; if the field conditions in a rule query request belong to a relationship where satisfying any one condition is sufficient, a bitwise OR operation is performed; if the rule query request contains exclusion conditions, a bitwise NOT operation or a combination operation is performed; if the rule expression contains multiple levels of logical relationships, the corresponding bitmap operations are performed layer by layer according to the order of the logical expression tree. Through the above bitwise logical operations, the data sets corresponding to multiple field conditions are merged, filtered, or excluded at the bitmap level, thereby obtaining a target candidate bitmap that retains only candidate objects. Since bitmap operations are performed on a bit-by-bit basis, they have high processing efficiency and are suitable for large-scale target data identifier filtering scenarios.

[0096] In some embodiments, before performing bitwise logical operations, the validity and length consistency of each data identifier bitmap can be checked to ensure that each bitmap is constructed based on the same global bitmap index, avoiding calculation errors caused by inconsistent bitmap lengths. If a bitmap length is found to be inconsistent with the global bitmap index, the bitmap can be padded, truncated, or remapped before participating in the operation. In other embodiments, the bitmap can be computed in parallel by fragments, performing bitwise logical operations on different fragments of the bitmap separately, and then summing the results of each fragment to generate a complete target candidate bitmap, thereby improving processing throughput and reducing the computational pressure per operation.

[0097] In S420, the core of deserialization is to map the valid bit positions in the bitmap back to the actual data identifier.

[0098] Specifically, the system can iterate through the valid bitmap positions in the candidate bitmap, where a valid bitmap position is a position with a valid identifier, indicating that the corresponding target data identifier satisfies all field conditions involved in the current calculation or meets a preset logical relationship. The system queries the data identifier corresponding to the valid bitmap position based on the global bitmap index, and adds the query result as the actual data identifier to the candidate target set. After iterating through all valid bitmap positions, a complete set of candidate targets is obtained.

[0099] In some embodiments, deserialization can employ a sequential scanning method to read the valid bit positions in the target candidate bitmap bit by bit and map them one by one to a data identifier. This method is simple to implement and suitable for scenarios with small to medium-sized candidate bitmaps. In other embodiments, deserialization can also employ skip-bit scanning, compressed bitmap decoding, segmented batch mapping, or parallel traversal to improve mapping efficiency for large-scale bitmaps. If the target candidate bitmap uses a compressed storage format, such as a block compression format or a sparse bitmap format, the compressed block can be decoded first, and then the valid bit positions can be extracted and mapped. This reduces storage usage while maintaining high deserialization efficiency. In some embodiments, the correspondence between bitmap positions and real data identifiers can also be recorded synchronously during deserialization for subsequent rule execution, result backtracking, or audit tracing. For duplicate bitmap positions, deduplication can be performed before mapping to avoid duplicate data identifiers in the candidate target set. For invalid bits or empty bits, no mapping is performed, thereby ensuring that the candidate target set only contains real data identifiers that meet the filtering conditions.

[0100] The processing flow of this embodiment can be further integrated with cache hit status. When multiple data identifier bitmaps come from cache hit results and storage-back-to-origin results respectively, bitwise logical operations can still be performed within the same bitmap coordinate system to uniformly generate target candidate bitmaps. Subsequently, deserialization processing is performed based on the global bitmap index to obtain a set of candidate targets. This ensures that even if some field conditions come from different sources, the final set of candidate targets still has consistent bitmap semantics and executability.

[0101] Specifically, in some embodiments, the specific implementation of step S420 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S420 in the data query caching method shown in the corresponding embodiment, step S420 in the data query caching method may include the following steps: Traverse the valid bitmap positions in the target candidate bitmap, where each valid bitmap position represents a data identifier that simultaneously satisfies all field conditions.

[0102] Based on the global bitmap index, the effective bitmap position index is mapped to the corresponding data identifier to obtain the real data identifier.

[0103] All real data identifiers are combined to generate a set of candidate targets.

[0104] This embodiment maps each valid bit in the target candidate bitmap to a real data identifier, which can accurately recover the hit result and ensure the integrity, uniqueness and traceability of the candidate target set.

[0105] In this embodiment, a valid bitmap position refers to a bitmap position in the target candidate bitmap where the bit value is a valid identifier. A valid identifier indicates that the corresponding target data identifier simultaneously satisfies all field conditions currently involved in the logical operation, or satisfies a preset joint filtering relationship. A real data identifier refers to the specific target object identifier that corresponds one-to-one with a valid bitmap position; it can be an MD5 value, file digest value, device identifier, account identifier, user identifier, transaction identifier, or other business object identifier. The purpose of deserializing the target candidate bitmap based on the global bitmap index is to restore the compressed filtering results at the bitmap level into a set of data identifiers that can be directly used for business rule determination.

[0106] Specifically, during the traversal, the bit values ​​in the target candidate bitmap are checked bit by bit according to the storage order, index order, or fragmentation order of the bitmap, and the positions in the valid identification state are identified. The determination of valid bitmap positions can be achieved by directly reading the bit values, or by block-level scanning, sparse bit scanning, scanning after decoding compressed blocks, or parallel segmented scanning. For target candidate bitmaps stored in compressed format, the compressed blocks can be expanded first, and then valid bit identification can be performed on the expanded bitmap. This method can improve traversal efficiency while ensuring the accuracy of the results.

[0107] In some embodiments, a sequential traversal can be performed on the target candidate bitmap to determine whether the current bit is a valid identifier. If the current bit value is a valid identifier, the corresponding position is recorded as a valid bitmap position; if the current bit value is an invalid identifier, the position is skipped and not proceeded to the subsequent mapping process. This method is suitable for scenarios where the bitmap size is small or the candidate hit results are relatively dense, and it is easy to implement and maintain.

[0108] In other embodiments, skip traversal or sparse traversal can be performed on the target candidate bitmap to directly locate the interval positions between consecutive valid bits. This method is suitable for scenarios where candidate results are relatively sparse and can reduce the computational overhead caused by scanning invalid bits. The system can also perform traversal according to bitmap fragments and then summarize the valid bitmap positions of each fragment to support parallel computing and distributed processing.

[0109] In some other embodiments, valid bitmap positions can be temporarily cached or listed during the traversal process for unified mapping later, avoiding repeated access to the global bitmap index. For duplicate valid bitmap positions, the system can directly deduplicate them during the traversal phase or perform deduplication during the mapping phase, thereby ensuring that the subsequently generated set of real data identifiers does not contain duplicates.

[0110] Once a valid bitmap location is identified, it is mapped to a corresponding data identifier based on the global bitmap index, thus obtaining the actual data identifier. This mapping relationship is a pre-established correspondence between bitmap locations and target data identifiers, allowing the system to quickly retrieve the corresponding target data identifier based on the bitmap location. The mapping can be implemented using methods such as mapping table retrieval, hash index lookup, array direct access, sharded index positioning, or distributed index lookup. Because there is a definite correspondence between bitmap locations in the global bitmap index and target data identifiers, this mapping process is deterministic and repeatable.

[0111] In some embodiments, the global bitmap index is constructed using a sequential numbering method, which allows direct indexing to the corresponding target data identifier based on the valid bitmap position without additional calculation. This method is suitable for scenarios with stable indexes and a large number of data identifiers, and can improve mapping speed.

[0112] In other embodiments, the global bitmap index is constructed using a sharding mapping approach. This involves first determining the shard to which the valid bitmap belongs based on its location, and then querying the target data identifier within the corresponding shard. This approach is suitable for ultra-large-scale data scenarios, reduces the storage pressure of a single index structure, and facilitates distributed deployment.

[0113] In some other embodiments, after mapping bitmap locations to actual data identifiers is completed, the validity of the mapping results can be verified simultaneously. For example, if the data identifier corresponding to a certain bitmap location has become invalid, deleted, or replaced, that location can be ignored, or updated to the latest data identifier according to version control rules, to ensure that the candidate target set is consistent with the current business data status.

[0114] After mapping is completed at all valid bitmap locations, all real data identifiers are combined to generate a candidate target set. It should be noted that the aforementioned combination refers to aggregating multiple real data identifiers into sets, lists, queues, arrays, or other iterable structures to form a data set that can be directly used for subsequent business rule determination. During the combination process, deduplication, sorting, sharding and merging, or priority reorganization can be performed on the real data identifiers to meet the needs of different business processing strategies. If the business scenario requires the results to maintain their original order, the traversal order can be preserved; if the business scenario requires output according to risk level, time order, or rule priority, the candidate target set can also be reorganized according to a preset sorting rule.

[0115] In some embodiments, the candidate target set can be used as input data for the final business rule determination, allowing the rule engine to perform subsequent verification item by item. Since the candidate target set has already undergone bitmap filtering and deserialization recovery, its size is usually significantly smaller than the original full dataset, which can effectively reduce the computational cost and storage access cost of subsequent rule matching.

[0116] In other embodiments, the candidate target set can be used in conjunction with rule context information, field attribute information, or dynamic attribute tables to perform more refined business rule determinations in subsequent steps. In this way, the candidate target set is not only the result of bitmap filtering but also the data foundation for business decision-making.

[0117] Through the above method, this embodiment realizes a complete recovery process from the target candidate bitmap to the real data identifier set. This process is based on a global bitmap index, uses the effective bitmap position as an intermediate link, and outputs the real data identifier. It can maintain the accuracy and consistency of the results while ensuring computational efficiency, providing stable input for subsequent final business rule determination.

[0118] In S500, business rule determination refers to the process of further analyzing and judging target data in the candidate target set according to the pre-configured rule logic of the business system. This can include risk identification, anomaly detection, threat analysis, data classification, policy matching, access control, alarm triggering, or other business processing. In specific implementation, the target data identifiers in the candidate target set can be obtained one by one, and the business data, attribute data, behavioral data, or other extended information associated with the target data identifiers can be read. Then, rule matching processing is performed according to preset rules.

[0119] In some embodiments, business rule determination can be performed using a rule engine; in other embodiments, it can be combined with machine learning models, expert rule models, statistical analysis models, or hybrid decision models to complete the determination process. Since the aforementioned steps have already pre-screened the candidate target set through bitmap logic operations, the final data size involved in business rule determination is significantly reduced, thereby effectively reducing system computational overhead and improving rule execution efficiency.

[0120] In some embodiments, after S500, the method further includes: Obtain a pre-constructed bitmap fragment mapping relationship, which records the correspondence between bitmap index fragments and data ranges of the target data identifier set.

[0121] Based on the bitmap fragmentation mapping relationship and the dynamic attribute table, attribute filtering processing is performed on the candidate target set. The dynamic attribute table stores attribute information of the target data identifier.

[0122] This embodiment introduces bitmap fragmentation mapping relationships and dynamic attribute tables, which can further refine the filtering by fragmentation and attributes after the candidate target set is generated, thereby improving the filtering accuracy, query flexibility and efficiency of large-scale data processing.

[0123] In this embodiment, the bitmap sharding mapping relationship refers to the segmented management relationship formed for the global bitmap index, which records the correspondence between bitmap index shards and data ranges of the target data identifier set. Bitmap index sharding refers to dividing the global bitmap index into several continuous or logically continuous index segments according to preset rules, with each index segment corresponding to a set of target data identifier data ranges. A data range refers to the distribution segment of target data identifiers in business, time, source, risk, or other dimensions, used to characterize the attribute aggregation characteristics of the target data identifiers within that shard. A dynamic attribute table is an attribute storage structure that can be updated in real-time or near real-time with changes in business data. It stores attribute information corresponding to the target data identifiers, such as field values, risk levels, category tags, timestamps, status information, source information, behavioral characteristics, or other business attributes. By introducing the bitmap sharding mapping relationship and the dynamic attribute table, the system can continue to perform secondary filtering based on ranges and attributes after the candidate target set is formed, thereby further improving the accuracy and relevance of the final rule determination.

[0124] Specifically, the bitmap sharding mapping relationship can be pre-generated during system initialization or dynamically updated during data growth, bitmap expansion, partition adjustment, or index reconstruction. The bitmap sharding mapping relationship includes at least the bitmap shard number, shard start position, shard end position, and the target data identifier data range information corresponding to the shard.

[0125] During sharding, the global bitmap index can be divided into multiple shards based on the number of target data identifiers, update frequency, business classification, popularity distribution, or storage structure. Sharding methods can include equal-length sharding, unequal-length sharding, sharding by business domain, sharding by time window, sharding by popularity, or sharding by hash range, to adapt to different business scenarios. The role of bitmap sharding mapping is that when the candidate target set is large, the system can quickly locate the bitmap range to which the candidate result belongs, thereby narrowing down the attribute filtering range and improving subsequent processing efficiency.

[0126] In some embodiments, bitmap shard mapping relationships are stored in the form of a mapping table. Each record in the mapping table corresponds to a bitmap shard and records the range of target data identifier numbers covered by the shard and its associated data interval information. The shard segments involved in a candidate target set can be directly determined by querying the mapping table, thereby achieving interval-level filtering. This method is suitable for scenarios with relatively stable index structures and well-defined shard boundaries. In other embodiments, bitmap shard mapping relationships can also be stored using tree indexes, segment page indexes, interval indexes, or multi-level directory indexes to adapt to large-scale distributed deployments and frequent update scenarios. For scenarios with massive target data identifiers, the shard mapping relationships can also be distributed and stored across multiple nodes to reduce the access pressure on a single node and improve concurrent processing capabilities.

[0127] After obtaining the bitmap fragment mapping relationship, attribute filtering processing is performed on the candidate target set based on the bitmap fragment mapping relationship and the dynamic attribute table. This attribute filtering processing means that, on the premise that the candidate target set has already met the basic field conditions in the rule query request, further filtering is performed based on the attribute information of the target data identifier to remove objects that do not meet the refined business requirements.

[0128] Specifically, the data range corresponding to the candidate target set can be determined first based on the bitmap segmentation mapping relationship. Then, the attribute information in the dynamic attribute table is read within this data range, and filtering is performed according to preset attribute conditions. Attribute conditions can include single attribute conditions, combined attribute conditions, range attribute conditions, enumerated attribute conditions, threshold attribute conditions, or hierarchical attribute conditions. In this way, fine-grained processing can be completed within a relatively small data range, avoiding indiscriminate searching of all candidate target sets.

[0129] In some embodiments, the dynamic attribute table can be read segment by segment. That is, the segment to which the candidate target set belongs is first determined based on the bitmap segmentation mapping relationship, and then the attribute records in the dynamic attribute table are read only for the data range corresponding to that segment, and attribute matching is performed. This method can reduce the invalid scanning range of the attribute table and is suitable for scenarios where the candidate target set is large but the attributes are widely distributed. In another embodiment, the system can first establish an attribute index on the dynamic attribute table, and then use the attribute index to quickly filter the candidate target set. The attribute index can be a hash index, inverted index, B-tree index, Trie index, composite index, or distributed index. With the attribute index, the system can directly locate the satisfying objects in the candidate target set based on attribute conditions, thereby reducing attribute query time.

[0130] It should be noted that the dynamic attribute table supports a real-time update mechanism. When the attributes of a target data identifier change, the corresponding record in the dynamic attribute table can be updated immediately or nearly in real-time, and the partition status in the bitmap sharding mapping relationship can be maintained synchronously. This ensures that the attribute filtering results are consistent with the current business status and avoids misjudgments caused by using outdated attribute information. If changes in certain attributes will cause changes in sharding assignment, the data range assignment of the corresponding target data identifier can also be adjusted synchronously to maintain the validity of the sharding mapping relationship. In some implementations, version management can also be implemented for attribute change records to allow for backtracking of the attribute status of the candidate target set at different points in time, meeting the needs of auditing, tracking, and result verification.

[0131] In attribute filtering, different filtering logics can be adopted based on the complexity of the business rules. For simple rules, filtering can be performed directly based on a single attribute condition; for complex rules, multiple attribute conditions can be combined and filtered according to preset logical relationships, such as satisfying multiple attribute conditions simultaneously, satisfying any attribute condition, excluding specific attribute conditions, or satisfying multi-level nested attribute conditions. Bitmap sharding mapping relationships and dynamic attribute tables can also be used in conjunction, i.e., first narrowing the candidate range by shards, then refining the filtering by attribute conditions, and finally outputting the final candidate results. This process can reduce attribute access costs while ensuring filtering accuracy, and is especially suitable for scenarios with large target data identification scales, many attribute dimensions, and frequent rule changes.

[0132] In other embodiments, attribute filtering can also be combined with a caching mechanism. If certain attribute records in the dynamic attribute table are frequently accessed, an attribute cache can be established for high-frequency attributes, and cached results can be read first during attribute filtering. If attribute information changes less frequently, some stable attributes can be pre-calculated to reduce real-time filtering pressure. For candidate target sets spanning multiple shards, the dynamic attribute tables of multiple shards can be read in parallel, and then the filtering results of each shard can be merged to improve throughput. For expired or missing attribute records, fault tolerance processing can be implemented, such as skipping the record, returning to the source to complete it, or marking it as an object to be verified, to ensure the stability of the filtering process.

[0133] By employing the above method, after forming a candidate target set through bitmap logical operations, attribute filtering is further performed using bitmap fragmentation mapping relationships and dynamic attribute tables. This ensures that the candidate results not only meet the basic field conditions but also satisfy more granular attribute requirements. This step effectively improves the accuracy of the final business rule determination, reduces false positives, and enhances the method's adaptability to complex business scenarios and dynamic data environments.

[0134] To enable those skilled in the art to better understand the technical solution of this application, a malicious file detection rule engine scenario is used as an example for illustration. In this scenario, real-time rule matching needs to be performed on hundreds of millions of MD5 file features to identify potentially risky files.

[0135] Specifically, the system first retrieves all MD5 hashes involved in rule matching from the threat intelligence database and assigns a unique and consecutive bitmap position to each MD5 hash, establishing a global bitmap index. This global bitmap index records the one-to-one correspondence between MD5 hashes and bitmap positions. For example, the first MD5 hash corresponds to bitmap position 0, the second MD5 hash corresponds to bitmap position 1, and so on. All subsequent cached data is constructed based on this unified bitmap coordinate system.

[0136] The system then parses all rules in the rule base. For example, rule R1 contains: risk=high, file_type=exe, source=download, etc. The system generates a cache key based on the rule identifier, field name, and field value bitmap, and generates a corresponding data identifier bitmap as the cache value based on the global bitmap index, thus forming a cache key-value pair structure.

[0137] To reduce initial access latency, the system further constructs a rule dependency graph. This graph records the co-occurrence relationships of each field condition during historical rule execution. For example, system statistics show that `risk=high` and `file_type=exe` co-occur with a probability of 92%. Simultaneously, `file_type=exe` and `source=download` co-occur with a probability of 87%.

[0138] When the system detects that a user has started executing a query with the condition risk=high, it predicts based on the rule dependency graph that the user is very likely to access the data corresponding to file_type=exe and source=download. Therefore, it loads the relevant MD5 set from the storage system in advance and generates the corresponding bitmap, which is then written to the cache to achieve predictive bitmap preloading.

[0139] During continuous operation, the system further monitors the frequency of rule execution. When it detects that the combination of risk=high AND file_type=exe AND source=download is accessed frequently, the background automatically generates a composite bitmap (Composite_Bitmap) and constructs the corresponding composite cache key.

[0140] The composite cache key is represented in the form of ruleID + (risk_filetype_source) + Composite_Bitmap. Subsequent executions of the same rule do not require recalculating the intersection of multiple bitmaps; instead, the composite cache result is directly hit.

[0141] When a new rule query request is received, the system parses the rule expression, extracts the field conditions it contains, and retrieves the corresponding data identifier bitmap based on the cached key-value pair structure. If some field conditions are not cached, the system retrieves the corresponding MD5 set from the underlying threat intelligence database and converts it into a bitmap structure. During the construction of cached values, the system calculates the bitmap sparsity in real time. When a field condition corresponds to a large number of MD5s, the system uses a standard global bitmap for storage. When a field condition corresponds to a very small number of MD5s, such as only a few dozen MD5s, while the global bitmap size reaches hundreds of millions of bits, the system automatically switches the cache structure to a Roaring Bitmap or a compact inverted index. In this case, the cache structure is upgraded from a fixed-length bitmap to a sparse structure. To ensure that different structures can participate in rule calculation uniformly, the system is configured with a heterogeneous intersection calculator.

[0142] When performing calculations for a standard bitmap ∩ a Roaring Bitmap or a Roaring Bitmap ∩ an inverted index, the heterogeneous intersection calculator automatically performs format conversion and unified operations. The system then performs bitwise logical operations. For a dataset that meets all conditions, an AND operation is performed: CandidateSet_bitmap = Bitmap_risk AND Bitmap_filetype AND Bitmap_source to obtain the target candidate bitmap.

[0143] The system further traverses the valid bit positions in the target candidate bitmap and recovers the corresponding real MD5 based on the global bitmap index to generate a CandidateSet of candidate targets.

[0144] Considering the continuous updating of the threat intelligence database, the system adopts a dual-region map incremental architecture.

[0145] The cached value is expanded into three parts: Base_Bitmap, Increment_Bitmap, and Tombstone_Bitmap.

[0146] When a new MD5 hash meets a certain field condition, the cache is not rebuilt; instead, it is written directly to Increment_Bitmap.

[0147] When a certain MD5 attribute becomes invalid, it is written to Tombstone_Bitmap.

[0148] When the rule is executed, (Base_Bitmap OR Increment_Bitmap) AND (NOTTombstone_Bitmap) is dynamically calculated to obtain the latest valid bitmap.

[0149] Therefore, there is no need to frequently delete or regenerate the cache.

[0150] When the size of the incremental bitmap reaches a preset threshold, the background thread asynchronously executes the compaction operation, merging the baseline bitmap and the incremental bitmap to generate a new Base_Bitmap, and clearing the incremental area.

[0151] Finally, the system combines bitmap fragmentation mapping relationships and dynamic attribute tables to further filter the CandidateSet by attribute. For example, it retains only samples newly added in the last 30 days, samples from a specified region, or samples with a specific risk level. After filtering, the remaining MD5 set is input into the rule engine to perform the final risk assessment and output the detection results.

[0152] This embodiment achieves efficient reuse of rule query results through global bitmap indexing, cached key-value pair structure, multi-dimensional bitmap intersection pruning, and a CandidateSet generation mechanism. It also introduces rule dependency graph prediction preloading, composite bitmap caching, sparsity-aware adaptive compression, and incremental lock-free update mechanisms, enabling the system to achieve high cache hit rate, high space utilization, and high data consistency. In scenarios involving massive MD5 rule matching, it can significantly reduce storage access pressure, decrease rule execution latency, and improve the overall throughput of the rule engine.

[0153] The following describes an apparatus embodiment of this application, which can be used to execute the data query caching method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the data query caching method described above in this application.

[0154] Figure 5 A block diagram of a data query caching device according to an embodiment of this application is shown.

[0155] Reference Figure 5 As shown, a data query caching device 500 according to an embodiment of this application includes: a bitmap index building module 510, a key-value pair structure building module 520, a field condition extraction module 530, a bitmap filtering processing module 540, and a business rule determination module 550.

[0156] The bitmap index building module 510 is used to build a corresponding global bitmap index for each target data identifier, wherein the target data identifier is the data identifier participating in rule matching; the key-value pair structure building module 520 is used to build a corresponding cached key-value pair structure for each target data identifier based on the global bitmap index; the field condition extraction module 530 is used to extract each field condition contained in the rule query request in response to the rule query request, and obtain the corresponding data identifier bitmap based on the cached key-value pair structure; the bitmap filtering processing module 540 is used to perform filtering processing on each data identifier bitmap using logical operations to generate a candidate target set; and the business rule determination module 550 is used to perform the final business rule determination based on the candidate target set.

[0157] Optionally, the bitmap index building module 510 specifically includes: a bitmap position allocation submodule, used to allocate a corresponding continuous bitmap position for each target data identifier; and a bitmap index generation submodule, used to generate a global bitmap index according to the mapping relationship between each target data identifier and the continuous bitmap position.

[0158] Optionally, the key-value pair structure establishment module 520 specifically includes: a cache key generation submodule, used to generate corresponding cache keys for the field conditions involved in each rule, wherein the cache key is at least composed of a rule identifier, a field name, and a field value bitmap; and a cache value generation submodule, used to generate corresponding cache values ​​according to the global bitmap index, which together with the cache key form a cache key-value pair structure, wherein the cache value is a data identifier bitmap.

[0159] Optionally, the field condition extraction module 530 specifically includes: a rule query request parsing submodule, used to respond to a rule query request, parse the rule query request, extract the field conditions contained therein, and obtain specific query conditions; and a data identifier bitmap acquisition submodule, used to obtain the corresponding data identifier bitmap based on the specific query conditions and the cache key-value pair structure.

[0160] Optionally, the apparatus further includes: a cache miss submodule, configured to, in response to a partial field condition in each of the field conditions missing a cache key-value pair structure, obtain a data identifier bitmap corresponding to the missed field condition from a preset storage system; and a bitmap-key-value pair submodule, configured to use the obtained data identifier bitmap together with the data identifier bitmap obtained from the cache key-value pair structure to perform the logical operation.

[0161] Optionally, the bitmap filtering processing module 540 specifically includes: a bitwise logic operation submodule, used to perform bitwise logic operations on each of the data identifier bitmaps to generate target candidate bitmaps; and a deserialization processing submodule, used to perform deserialization processing on the target candidate bitmaps based on the global bitmap index to generate a candidate target set.

[0162] Optionally, the deserialization processing submodule specifically includes: a valid bitmap position traversal unit, used to traverse the valid bitmap positions in the target candidate bitmap, wherein the valid bitmap position is the bitmap position in the target candidate bitmap with a bit value of a valid identifier, representing a data identifier that simultaneously satisfies all field conditions; a valid bitmap position mapping unit, used to map the valid bitmap positions to the corresponding data identifiers based on the global bitmap index, to obtain real data identifiers; and a real data identifier combination unit, used to combine all real data identifiers to generate a candidate target set.

[0163] Optionally, the device further includes: a sharding mapping relationship submodule, used to obtain a pre-constructed bitmap sharding mapping relationship, wherein the bitmap sharding mapping relationship records the correspondence between bitmap index shards and data ranges of the set of target data identifiers; and an attribute filtering processing submodule, used to perform attribute filtering processing on the candidate target set based on the bitmap sharding mapping relationship and a dynamic attribute table, wherein the dynamic attribute table stores attribute information of the target data identifiers.

[0164] The embodiments of this application construct a global bitmap index and a cached key-value pair structure corresponding to the target data identifier. During the rule query stage, bitmap logical operations are used to quickly generate a candidate target set, and business rule judgment is performed based on the candidate target set. This reduces the number of full data matching and storage accesses, improves rule execution efficiency, reduces system resource consumption, and enhances query response capabilities in large-scale data scenarios.

[0165] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0166] It should be noted that, Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 6As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1803. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via bus 1804. An Input / Output (I / O) interface 1805 is also connected to bus 1804.

[0168] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0169] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0170] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0172] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0173] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0174] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0175] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0176] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0177] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data query caching method, characterized in that, include: A corresponding global bitmap index is established for each target data identifier, where the target data identifier is the data identifier participating in rule matching; Based on the global bitmap index, construct a corresponding cache key-value pair structure for each of the target data identifiers; In response to a rule query request, extract the field conditions contained therein, and obtain the corresponding data identifier bitmap based on the cache key-value pair structure; Logical operations are used to filter each of the data identifier bitmaps to generate a candidate target set; Based on the set of candidate targets, the final business rule determination is performed.

2. The data query caching method as described in claim 1, characterized in that, The step of establishing a corresponding global bitmap index for each target data identifier specifically includes: Assign a corresponding consecutive bitmap position to each target data identifier; A global bitmap index is generated based on the mapping relationship between each target data identifier and the position of the continuous bitmap.

3. The data query caching method as described in claim 1, characterized in that, The step of constructing a corresponding cached key-value pair structure for each target data identifier based on the global bitmap index specifically includes: For each rule, a corresponding cache key is generated for the field conditions involved. The cache key is at least composed of a rule identifier, a field name, and a bitmap of field values. Based on the global bitmap index, a corresponding cache value is generated, which forms a cache key-value pair structure with the cache key. The cache value is a data identifier bitmap.

4. The data query caching method as described in claim 1, characterized in that, In response to a rule query request, the process of extracting the field conditions contained therein and obtaining the corresponding data identifier bitmap based on the cached key-value pair structure specifically includes: In response to a rule query request, the rule query request is parsed, and the field conditions contained therein are extracted to obtain the specific query conditions; Based on the specific query conditions, the corresponding data identifier bitmap is obtained according to the cache key-value pair structure.

5. The data query caching method as described in claim 4, characterized in that, After responding to a rule query request, parsing the rule query request, extracting the field conditions contained therein, and obtaining the specific query conditions, the method further includes: In response to a partial field condition in each of the stated field conditions not being found in the cached key-value pair structure, a data identifier bitmap corresponding to the missing field condition is obtained from a preset storage system. The acquired data identifier bitmap and the data identifier bitmap obtained from the cache key-value pair structure are used together to perform the logical operation.

6. The data query caching method as described in claim 1, characterized in that, The step of filtering each of the data identifier bitmaps using logical operations to generate a candidate target set specifically includes: Perform bitwise logical operations on each of the data identifier bitmaps to generate target candidate bitmaps; The target candidate bitmap is deserialized based on the global bitmap index to generate a candidate target set.

7. A data query caching device, characterized in that, The data query caching device includes: The bitmap index building module is used to build a corresponding global bitmap index for each target data identifier, wherein the target data identifier is the data identifier that participates in rule matching. The key-value pair structure building module is used to construct a corresponding cache key-value pair structure for each of the target data identifiers based on the global bitmap index; The field condition extraction module is used to extract the field conditions contained in the rule query request in response to the rule query request, and obtain the corresponding data identifier bitmap according to the cache key-value pair structure; The bitmap filtering processing module is used to filter each of the data identifier bitmaps using logical operations to generate a candidate target set; The business rule determination module is used to perform the final business rule determination based on the candidate target set.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data query caching method as described in any one of claims 1 to 8.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the data query caching method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, It includes one or more computer programs that, when executed by one or more processors, implement the steps of the data query caching method according to any one of claims 1 to 8.

Images