Data processing method and device, equipment and storage medium

Abstract

The application discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method obtains a first rule to be stored, the first rule comprising a first matching condition and first rule content; generates storage strategy information corresponding to the first matching condition in a rule storage structure table in the memory, the storage strategy information comprising first partition information for indicating storage of the first matching condition in a disk, the first partition information being partition information of a first partition in the disk; determines a first condition node corresponding to the first partition and a first rule node configured for the first condition node from a rule network graph of the disk, and stores the first matching condition in the first condition node and the first rule content in the first rule node. Thus, in a large-scale data storage scenario, disk storage is used instead of memory storage, and the problem of high cost of memory storage rules can be effectively solved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, device and storage medium. Background Technology

[0002] A rule consists of two parts: rule content and matching conditions. The rule content refers to the part of the storage engine that defines the execution logic, while the matching conditions are the part of the storage engine used to determine whether the rule is triggered. Current rule storage primarily uses memory. In scenarios involving large-scale rule storage, constructing a memory structure according to the storage engine logic would incur significant cost overhead. For example, uncompressed key-value pairs or full indexes would grow linearly with the number of rules, consuming substantial memory resources and increasing storage costs. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium that can solve the problem of high cost of memory storage rules.

[0004] In a first aspect, embodiments of this application provide a data processing method, which may include:

[0005] Retrieve the first rule to be stored, which includes the first matching condition and the content of the first rule;

[0006] In the rule storage structure table in memory, storage policy information corresponding to the first matching condition is generated. The storage policy information includes the first partition information used to indicate the storage of the first matching condition on the disk. The first partition information is the partition information of the first partition on the disk.

[0007] Determine the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the disk's rule network graph;

[0008] The first matching condition is stored in the first condition node, and the content of the first rule is stored in the first rule node.

[0009] Secondly, embodiments of this application provide a data processing apparatus, which may include:

[0010] The acquisition module is used to acquire the first rule to be stored, which includes the first matching condition and the content of the first rule.

[0011] The generation module is used to generate storage policy information corresponding to the first matching condition in the rule storage structure table in memory. The storage policy information includes the first partition information used to indicate the storage of the first matching condition on the disk. The first partition information is the partition information of the first partition on the disk.

[0012] The determination module is used to determine the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk.

[0013] The storage module is used to store the first matching condition in the first condition node and the content of the first rule in the first rule node.

[0014] Thirdly, embodiments of this application provide a computer device, which includes: a processor and a memory storing computer program instructions;

[0015] When the processor executes computer program instructions, it implements the data processing method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data processing method as described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the data processing method as shown in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the data processing method as described in the first aspect.

[0019] The data processing method, apparatus, device, and storage medium of this application embodiment can obtain a first rule to be stored, the first rule including a first matching condition and a first rule content; generate storage strategy information corresponding to the first matching condition in a rule storage structure table in memory, the storage strategy information including first partition information for indicating the storage of the first matching condition on disk, the first partition information being the partition information of the first partition on disk; determine a first condition node corresponding to the first partition and a first rule node configured for the first condition node from the rule network graph of the disk, and store the first matching condition and the first rule content in the first condition node. Thus, in large-scale data storage scenarios, replacing memory storage with disk storage effectively solves the problem of high cost of memory-stored rules, and applying the rule network graph to disk condition storage allows the relationship between nodes to be used as pointers, thereby accelerating the rule matching speed. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram of a rule storage structure table for a data processing method provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram of a rule network diagram for a data processing method provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart of a rule matching method according to an embodiment of the data processing method provided in this application;

[0025] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0029] The acquisition, storage, use, and processing of data (including but not limited to features and information mentioned in this document) in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0030] Rule storage engines are widely used for storing rules. For example, the Drools rule storage engine can parse a written configuration file into a network structure of nodes and edges using a syntax tree, enabling efficient rule matching. However, Drools is a rule matching solution designed for in-memory storage. In practical applications, the cost of storing large-scale rules is high. Disk-based solutions can effectively reduce costs, and general-purpose key-value in-memory databases still suffer from complex modeling issues in permission-related scenarios. Therefore, existing rule storage systems primarily rely on in-memory queries, placing significant pressure on memory. When deploying large-scale rules, distributed caching needs to be built, and requests need to be routed to the relevant rule cache machines, resulting in extremely high deployment complexity and cost.

[0031] To address the aforementioned pain points, this application proposes a data processing method, apparatus, device, and storage medium for disk storage and matching of large-scale rules.

[0032] The following will be combined with the appendix Figures 1 to 6 This application describes in detail the data processing methods, apparatus, computer equipment, and storage media of the embodiments. It should be noted that these embodiments are not intended to limit the scope of this application.

[0033] Combination Figure 1 This application provides a schematic flowchart of a data processing method.

[0034] like Figure 1 As shown, this data processing method can be applied to computer equipment, and the data processing method may specifically include the following steps:

[0035] Step 110: Obtain the first rule to be stored, which includes a first matching condition and the content of the first rule; Step 120: Generate storage strategy information corresponding to the first matching condition in the rule storage structure table in memory, which includes the first partition information used to indicate the storage of the first matching condition on the disk, where the first partition information is the partition information of the first partition on the disk; Step 130: Determine the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk; Step 140: Store the first matching condition in the first condition node and the first rule content in the first rule node.

[0036] In this way, in large-scale data storage scenarios, replacing memory storage with disk storage effectively solves the problem of high rule costs in memory storage. Furthermore, applying the rule network graph to disk conditional storage allows the relationship between nodes to be used as pointers, thereby accelerating the rule matching speed.

[0037] The steps described above are explained in detail below.

[0038] First, regarding step 110, the rules in this embodiment include rule content and matching conditions. The rule content refers to the part of the rule engine that defines the execution logic, which can consist of an action list or an output expression. The matching conditions refer to the part of the rule engine used to determine whether a rule is triggered, and can include conditional expressions or logical combinations. For ease of expression, in this embodiment, the rule can include a first rule, the matching conditions can include a first matching condition, and the rule content can include the first rule content.

[0039] For example, if the first rule is "block transaction requests where the merchant number is in the merchant number blacklist and the acquiring institution number is in the acquiring institution blacklist under the payment system", then the first matching condition of this first rule is "IF((system number IS payment system)&&(merchant number IN [merchant number blacklist])&&(acquiring institution IN [acquiring institution blacklist]))", and the content of the first rule is "reject the request".

[0040] It should be noted that the matching condition of the rule can be at least one. For example, in the above exemplary example, the matching condition is three: IF((System Number IS Payment System), (Merchant Number IN [Merchant Number Blacklist List]), (Acquiring Institution IN [Acquiring Institution Blacklist List])), and the matching pattern of the three conditions is "&&", that is, the relationship between them.

[0041] Secondly, step 120 is involved, such as Figure 2As shown, the rule storage structure table in this application is the core table used for dynamically managing permission rules, service logic, or query conditions. The rule storage structure table may include a memory area, a field data area, an initial field storage area, and a rule storage area.

[0042] The memory area, which can be located in memory, is used to record rule modifications and the version numbers of edge nodes in the rule network graph. Matching conditions and rule content can be stored in a key-value format. Recording rule modifications can include changes to matching conditions and rule content, including at least one of the following: deletion and updating. If deletion occurs, the number of deletions is recorded. Based on this, the usage count of condition nodes is counted, and during merging, the member function (use_count) can be subtracted accordingly to adjust the usage count of condition nodes. The version number is used for data synchronization between memory and disk. The version file corresponding to the version number can store the storage key-value pairs of rules. The key-value pairs can be partition and condition node key-value pairs, i.e., (fd + offset). Here, fd can be used to identify the partition, and offset can be used to specify the specific location of the condition node within the partition. Thus, the version number after a compaction operation can be located based on the key-value pairs in the version file. During rule matching, if the matching conditions to be matched have been modified in the memory area, the data in the memory area can be used first. Users can also cache the most frequently accessed rules in the memory area and perform cache replacement based on Least Recently Used (LRU). This allows rules in the memory area to be used preferentially during rule matching. When the modified data in the memory area reaches a specified size, the data will be categorized by field and merged into the corresponding field data area.

[0043] The field data area comprises M partitions on the disk containing the rules. These partitions are divided according to fields, such as partitions for field 1, field 2, ..., field M, where M is an integer greater than 1. Each partition can include files for condition nodes, edge nodes, and rule nodes, used to construct and update the rule network graph on the disk. Condition nodes can include nodes storing deterministic condition data and nodes storing non-deterministic condition data. Each partition corresponds to a compact log (compact_log), which records the partition's version number. The version file corresponding to this version number stores the offset of the condition nodes within that partition after each compaction.

[0044] The initial fields section includes the initial matching conditions for the rules, such as the initial matching conditions for the first rule. During rule matching, the fields included in the request can be iterated to see if matching of a particular rule's matching conditions can begin.

[0045] The rule storage area can be located on a disk and can be used to store rule network graphs. In this embodiment, after a rule is written, it is parsed into a combination of multiple condition nodes and edge nodes, and finally points to a rule node in the rule data area.

[0046] Based on this, in some embodiments of this application, if the first rule to be stored has partitions, there is no need to create partitions and it can be reused directly, reducing storage resources. Based on this, the rule storage structure table may include a field data area, which includes multiple partitions in the disk that store rules. The multiple partitions are divided according to fields. Step 120 may specifically include steps 1201 and 1202, as shown below.

[0047] Step 1201: Match the first field in the first matching condition with the fields of each partition in the multiple partitions.

[0048] Step 1202: If a partition that matches the first field is included among multiple partitions, the partition that matches the first field is determined as the first partition, and the partition information of the first partition is determined as the first partition information.

[0049] For example, the first field can be (merchant number), which is matched against the fields of each of the multiple partitions. If there is a partition 2 that includes (merchant number) among the multiple partitions, then it is identified as the first partition, and the partition number 2 of that partition is identified as the first partition information.

[0050] Therefore, by sharing partitions, we can avoid creating partitions repeatedly within the field data area, thus saving storage resources and reducing storage costs.

[0051] Furthermore, if there is no partition for the first field in the original rule storage result table, a partition can be created on the disk and the partition of the field data area can be updated. Based on this, step 120 may also include steps 1203 and 1204.

[0052] Step 1203: If the partition that matches the first field is not included in the multiple partitions, create a new partition on the disk that corresponds to the first field.

[0053] Step 1204: Add the new partition as the first partition to the field data area, and determine the partition information of the new partition as the first partition information.

[0054] For example, taking (merchant number) as the first field, if the partition does not include the (merchant number) among the multiple partitions, a new partition X corresponding to the (merchant number) can be added to the disk, and the new partition X can be added to the field data area. Similarly, the new partition X is used as the first partition, and the partition number X of the new partition is determined as the first partition information.

[0055] Therefore, when there is no partition, a new partition can be added to the field data area, which can improve the flexibility of storage rules.

[0056] Next, regarding step 130, the rule network graph in this embodiment is a rule network graph constructed based on the rule storage structure table. This rule network graph can be a directed acyclic graph. The matching conditions of the rules are abstracted into condition nodes, the rule content of the rules is abstracted into rule nodes, and the matching pattern of the matching conditions is abstracted into edges between condition nodes. The matching pattern can include combinations of operations such as AND and OR.

[0057] The rule network diagram in the embodiments of this application is as follows: Figure 3 As shown, a condition node can include the field name, field value, matching pattern, second link information, boundary point pointer, and user attribute information. The boundary point pointer is used to locate the boundary point of the partition where the condition node resides. The user attribute information (use_count) indicates the number of times the condition node is used; if this count is zero, it can be discarded and removed from the rule network graph. The second link information records the first rule node associated with the first rule node in the rule network graph.

[0058] Boundary points include version numbers and first link information. The version number is used for data synchronization between memory and disk. The version file corresponding to the version number can store key-value pairs of rules. These key-value pairs can be partition and condition node key-value pairs, i.e., (fd + offset). Here, fd can be used to identify the partition, and offset can be used to specify the specific location of the condition node within the partition. Thus, the version number after a certain compaction operation can be located based on the key-value pairs in the version file. In this embodiment, the first link information is used to record the dependency relationship between every two condition nodes in the rule network graph. For example... Figure 3 As shown, boundary points can record the dependency relationship between condition node 1 and condition node 2, and also the dependency relationship between condition node 1 and condition node 3.

[0059] If there is no next condition node linked to a condition node, the rule node connected to it can be determined through the second link information stored in the condition node. (This can still be referenced.) Figure 3Rule node 1 can store rule content related to condition nodes 1 and 2. For example, if condition node 1 stores "(System ID IS Payment System)" and condition node 2 stores "(Merchant ID IN [Merchant ID Blacklist List]")", then the rule is IF((System ID IS Payment System) &&(Merchant ID IN [Merchant ID Blacklist List]), and the rule content stored in rule node 1 is "Reject". Similarly, if condition 1 stores "((System ID IS Payment System)" and condition node 3 stores "(Acquiring Institution IN [Acquiring Institution Blacklist List])")", then the rule is IF((System ID IS Payment System) &&(Acquiring Institution IN [Acquiring Institution Blacklist List])), and the rule content stored in rule node 2 is "Allow". If the rule content stored in rule node 2 is "Reject", then the second link information in condition node 3 can be changed, that is, the second link information can be changed from condition node 3 associating with rule node 2 to condition node 3 associating with rule node 1, and rule node 1 can be deleted from the rule network graph.

[0060] In some embodiments of this application, when there is partitioning, it is necessary to further determine whether there is a first condition node for storing the first matching condition. If there is, the first condition node can be reused; if not, it can be added. Based on this, the first condition node can be determined in the following way.

[0061] Step 130 specifically includes steps 1301 and 1302, or steps 1303 or 1304.

[0062] Step 1301: Match the conditional data in the first partition with the first conditional data of the first matching condition.

[0063] For example, the first condition data of the first matching condition can be ([Merchant Number Blacklist List]), that is, the merchant number blacklist involved in the merchant number blacklist list, and ([Merchant Number Blacklist List]) can be matched with the condition data in the first partition.

[0064] Step 1302: If the condition data in the first partition includes second condition data that matches the first condition data, the second condition node storing the second condition data is determined as the first condition node.

[0065] For example, the first condition data can be ([Merchant ID Blacklist List]) and the Merchant ID Blacklist List includes 1000 merchant IDs, and the second condition data can be data that completely includes the first condition data.

[0066] Step 1303: If the condition data in the first partition does not include condition data that matches the first condition data, create a new condition node corresponding to the first partition in the rule network graph, and determine the new condition node as the first condition node.

[0067] For example, if the first partition does not include the merchant number blacklist, a new condition node corresponding to the first partition is created in the rule network graph, and the new condition node is determined as the first condition node.

[0068] It should be noted that if a new condition node corresponding to the first partition is created in the rule network graph, the records of each partition in the rule storage structure table can be updated according to the updated rule network graph.

[0069] Step 1304: If the condition data in the first partition includes third condition data that matches at least part of the condition data in the first condition data, the third condition node storing the third condition data is determined as the first condition node.

[0070] For example, the first condition data can be ([Merchant ID Blacklist List]) and the merchant ID blacklist list includes 1000 merchant IDs. The third condition data can also be ([Merchant ID Blacklist List]), except that the merchant ID blacklist list includes 500 merchant IDs. In this case, the third condition node storing the third condition data can also be determined as the first condition node.

[0071] In this embodiment, when the rule network graph is updated, such as by adding or deleting condition nodes, the rules are modified during matching based on the new compressed log content. Condition nodes are sorted in order after each compaction. During the process of determining the first condition node, a binary search can be used to check if the required condition node already exists. If it does, the original condition node in the rule network graph is reused; if it does not exist, a new condition node is appended to the end of the rule network graph file.

[0072] It should be noted that in this embodiment, field values ​​with different matching conditions will exist in a condition node. For example, the first matching condition of the first rule is "IF((System Number IS Payment System)&&(Merchant Number IN[Merchant Number Blacklist List])&&(Acquiring Institution IN[Acquiring Institution Blacklist List]))". The field values ​​of the fields, such as ((System Number IS Payment System)), will be stored in condition node 1, the field values ​​of the fields, such as (Merchant Number IN[Merchant Number Blacklist List])), will be stored in condition node 2, and the field values ​​of the fields, such as (Acquiring Institution IN[Acquiring Institution Blacklist List])), will be stored in condition node 3. Furthermore, condition node 1 and / or the edge nodes in the rule network graph will store the second link information and the matching pattern of the three condition nodes, i.e., the sum. The second link information can record the dependency relationship from condition node 1, condition node 2 to condition node 3. In this way, when matching rules, if condition node 1 is found, condition node 1, condition node 2 and condition node 3 can be found based on the second link information stored in the edge nodes of the rule network graph and the matching pattern of the three condition nodes.

[0073] Furthermore, the above example illustrates the situation where condition node 1, condition node 2, and condition node 3 are located in the same partition. In practice, in the embodiments of this application, condition node 1, condition node 2, and condition node 3 can be distributed across different partitions. In this case, the relationship between the three nodes can also be stored in the manner described above. What is particularly noteworthy is that it is not limited to storing the second link information and the matching pattern of the three condition nodes in the edge nodes of one partition. The edge nodes of each partition involved in different partitions can store the second link information and the matching pattern of the three condition nodes.

[0074] In some embodiments of this application, condition node 1 may be an original node in the regular network graph, and condition node 2 may be a node added in the manner described above. In this case, it is necessary to store the second link information and the matching pattern with condition node 2 in condition node 1. Based on this, the first condition node in the embodiments of this application may include a first node and a second node, where the first node is an original node in the regular network graph, and the second node is a newly added node in the regular network graph. The first node and the second node may be nodes in the same partition or nodes in different partitions. Based on this, during the storage process, the edge nodes can be updated to modify the first link information, i.e., the pointer, in the edge nodes. The data processing method may also include step 150, updating the first link information in the edge nodes of the regular network graph. The first link information is used to record the dependency relationship between every two conditions in the regular network graph. The first link information includes the dependency relationship between the first node and the second node. Wherein, when the first node and the second node are nodes in the same partition, the edge nodes involved in step 150 may be the boundary points of the partitions where the first node and the second node are located. When the first node and the second node are nodes in different partitions, the edge nodes involved in step 150 may include the boundary points of the partitions where the first node and the second node are located.

[0075] In some embodiments of this application, when there is a partition, it is necessary to further determine whether there is a first rule node for storing the content of the first rule. If there is, the first rule node can be reused; if not, it can be added. Based on this, the first rule node can be determined in the following way. Based on this, step 130 specifically includes step 1305, step 1306, or step 1307.

[0076] Step 1305: If the rule content of the second rule node configured for the second condition node includes the content of the first rule, then the second rule node is determined as the first rule node. Here, the second rule node configured for the second condition node refers to a rule node configured with a link relationship to the second condition node.

[0077] Step 1306: If the rule content of the second rule node does not include the first rule content, create a new rule node configured as a new condition node in the rule network graph, and designate the new rule node as the first rule node. Step 1307: If the rule content of the second rule node does not include the first rule content, traverse all other rule nodes in the rule network graph except the second rule node; if a third rule node exists among the other rule nodes that matches the first rule content, designate the third rule node as the first rule node.

[0078] For example, the first rule content can be "reject". In this case, if the second rule node configured for the second condition node does not include the first rule content, then other rule nodes in the rule network graph can be traversed to see if they include "reject". If there is a third rule node among other rule nodes that matches the first rule content, then the third rule node can be reused as the first rule node.

[0079] Here, if there is no third rule node among the other rule nodes that matches the content of the first rule, the first rule node can be determined using the method described in step 1306 above.

[0080] It should be noted that during the rule storage process, links are set between conditions and rule content. When the first rule node is a newly added rule node or a third rule node, a second link is added to the first condition node. This second link is used to record the first rule node associated with the first rule node in the rule network graph.

[0081] In this way, by sharing rule nodes, we can avoid creating rule nodes repeatedly, save storage resources, and reduce storage costs.

[0082] Then, regarding step 140, in this embodiment of the application, regardless of whether the first condition data of the first matching condition is completely reused from the original data in the rule network graph, completely newly added data, or partially newly added data, the first condition data of the first matching condition can be stored in the first condition node in the following manner.

[0083] In this embodiment, condition nodes are stored in order according to field names and matching patterns to support subsequent binary search. For definite value condition data, storage can be either inline or pointer-based, depending on the comparison between the definite value condition data and the definite value. Specifically, the first data in the definite value condition data that is less than or equal to a preset definite value can be stored inline, i.e., a short string is stored in the condition node. The second data in the definite value condition data that is greater than the preset definite value can be stored using pointers, i.e., a pointer to a long string is stored in the condition node. For non-definite value condition data, reference storage can be used, storing the regular expression of the non-definite value condition data in the first condition node. These will be explained separately below.

[0084] In some embodiments of this application, the first condition data of the first matching condition includes field values ​​and matching patterns, and the first condition node is stored according to the field values ​​and matching patterns. Based on this, step 140 may specifically include:

[0085] If the first condition data of the first matching condition is a definite value condition data, the first condition data is stored in the first condition node in ascending order of field values.

[0086] If the first condition data of the first matching condition is a non-deterministic condition data, the regular expression of the non-deterministic condition data is stored in the first condition node, and the non-deterministic condition data is stored in the expression library.

[0087] Furthermore, the determination value condition data in this application embodiment includes first data that is less than or equal to a preset determination value and second data that is greater than the preset determination value. Based on this, the step of storing the first condition data in the first condition node in ascending order of field values ​​can specifically include:

[0088] In the first condition node, the first data and the pointer data corresponding to the second data are stored in ascending order of field values. The pointer data is used to indicate the node where the second data is stored.

[0089] In this embodiment, the proposed disk storage rule references the LSMtree concept, converting all disk writes into sequential append writes and periodically merging condition nodes to clean up redundant ones. Condition nodes must be of fixed length to ensure in-place overwrite during data modification without affecting other data. If a condition node becomes longer, modifications to it may affect surrounding nodes. MySQL's B+ tree solves the problem of unequal data length through page splitting, but this incurs additional write and write overhead. Therefore, in this embodiment, only field values ​​and boundary points can potentially become longer in rule matching scenarios. When modifying a field value in a matching condition, only the boundary point pointer of the condition node storing that matching condition needs to be modified, thus discarding the original data and appending it directly to the existing data. Edge nodes without a pointed-to condition node or condition nodes without a pointed-to condition node can be considered discarded condition nodes. Discarded condition nodes are asynchronously deleted in the background at regular intervals, and the data is compressed.

[0090] Based on this, the above content will be further explained in conjunction with the rule network diagram. In this embodiment, modifications to condition nodes are preferentially stored and written into memory in key-value format. The key is the partition (filename fd of the original file) + the offset of the original data file, and the value is the specific condition node. The first link information of the newly added condition node, the boundary point pointer corresponding to the newly added or adjusted edge node, and the second link information corresponding to the newly added rule node are written to the append data area in the condition node. When the file size of the append data area reaches a threshold, it is then refreshed to the data partition corresponding to the field. Based on this, when a rule with storage is obtained, the initial field of the matching condition is appended to the corresponding file in the initial field area. If there is no corresponding file for the field, a new file is created. The writing of condition nodes first checks whether there is a corresponding partition in the field data area and whether there is a corresponding condition node in that partition. If there is, it is reused directly; otherwise, a new condition node is appended to the end of the corresponding file in the initial field area.

[0091] To further optimize rule matching speed, the condition nodes are ordered in this embodiment. While the first condition node corresponding to a rule cannot be jumped to by determining its position, for exact match patterns, binary search of ordered data segments can optimize query speed. Specifically, the condition nodes are sorted by match pattern and field value, and will be described in ascending order later. If the match pattern is equal to the sign, the offset nodes of the matched condition nodes are offset1 to offset2. M Here, it is expected that when a rule query request requests an offset within a range... k When a match fails, request the field value compared to the offset. k When the value of the field to be matched in a condition node is large, subsequent values ​​located at the offset will no longer be matched. k ~offset M The condition node matching. Based on the above, the embodiments of this application provide the following optimization process for the data that needs to be queried with a definite match. That is, during compaction, the definite value condition data and the non-definite value condition data are stored separately. For the condition node storing the definite value condition data, a definite value matching file header is added after the 20-byte ordered file length identifier field, as shown below: [{begin1,end1},{begin2,end2},{begin3,end3}]. Here, the definite value matching only has three types: less than, equal to, and greater than, so the file header length is determined. When the algorithm performs node matching, it should prune these three intervals each time. That is, under the interval of a definite matching algorithm, if the offset k Matching failed; no further offset matching is needed. k ~offsetM data.

[0092] It should be noted that the embodiments of this application also provide the method of deleting useless condition nodes in the rule network graph to reduce storage resources. Based on this, the data processing method in the embodiments of this application may further include:

[0093] Record the number of times each condition node is used in the rule network graph;

[0094] Obtain discarded condition nodes in the rule network graph whose usage frequency is less than or equal to a preset threshold;

[0095] Remove obsolete condition nodes from the rule network graph and update the version number and first link information of the edge nodes in the rule network graph. The version number is used for data synchronization between memory and disk, and the first link information is used to record the dependency relationship between every two condition nodes in the rule network graph.

[0096] For example, the number of deletions is recorded in memory. During compaction, the use_count is decremented accordingly. When the use_count reaches 0, the condition node is marked as obsolete, and the corresponding data is deleted during compaction.

[0097] This application embodiment also provides a rule query process, that is, during the query, the edge node information pointed to by the edge_ptr of the condition node is found, and all reachable condition nodes are traversed concurrently. Based on the matching mode, field name and field value stored in the condition node, the condition node that can successfully match the rule query request is found and the traversal continues until all condition nodes to be traversed have been traversed. Finally, the rule content of all rule nodes queried during the query process is returned. Based on this, the following combines... Figure 4 The process of rule query in the embodiments of this application is explained in detail below.

[0098] After step 140, the data processing method in this embodiment may further include steps 41 to 411, as shown below.

[0099] Step 41: Based on the rule query request, create a result set and a list of condition nodes.

[0100] Step 42: Traverse the initial field storage area in the rule storage structure table to obtain the initial target matching conditions in the initial field storage area that correspond to the fields in the rule query request.

[0101] In this embodiment, all fields to be queried are assigned a specific sequence number, such as field 1 to field 50. After accessing the initial field storage area, the query for matching condition nodes begins from the field 1 file. Rules are also created in the same order of field 1 to field 50, which reduces the area of ​​disk scanning.

[0102] Step 43: Determine the fourth condition node in the disk's rule network graph to store the target matching conditions.

[0103] Step 44: Based on the first link information in the edge node corresponding to the fourth condition node in the regular network graph, determine the N fifth condition nodes in the regular network graph that have a dependency relationship with the fourth condition node, where N is an integer greater than 1.

[0104] Step 45: Add the fourth condition node and N fifth condition nodes to the condition node list.

[0105] Step 46: Determine whether the version number of the partition containing the i-th condition node in the condition node list has been modified. If the version number of the partition containing the i-th condition node in the condition node list has not been modified, proceed to step 47. If the version number of the partition containing the i-th condition node in the condition node list has been modified, proceed to step 48.

[0106] Step 47: If the version number of the edge node in the rule network graph recorded in the memory area of ​​the rule storage structure table has been modified, add the target rule content corresponding to the target matching condition in memory to the result set.

[0107] Step 48: Update the version number of the edge node in the rule network graph according to the version number of the partition where the fourth condition node is located and the version number of the edge node in the rule network graph recorded in the memory area.

[0108] Step 49: Update the first link information in the edge node according to the updated version number of the edge node in the rule network graph;

[0109] Step 410: Adjust the list of condition nodes according to the updated first link information;

[0110] Step 411: Based on the adjusted list of condition nodes, traverse the target rule nodes associated with each condition node in the adjusted list of condition nodes, and add the target rule content stored in the target rule nodes to the result set.

[0111] Furthermore, in this embodiment, the reuse of condition nodes can effectively compress the storage space occupied by rules, but it also brings problems during querying. When a condition node matches successfully, not all the pointers of the first link information (edge_ptr) in the edge nodes can be added to the condition node list. Only pointers with the same identifier (rule_id) as the current access rule can be added to the condition node list. This means that a hash table of rule_id needs to be maintained to ensure that all unnecessary query requests can be deleted through this hash table when accessing the boundary point later. Considering that the allocation of rule_id is continuous and non-repeating, allocating a block of memory of a certain size and using a bitmap can effectively achieve deduplication. Based on this, the edge node in this embodiment also includes a rule identifier hash, which is used to record the rule identifier corresponding to each condition node in the rule network graph. Based on this, the above-mentioned step of determining the N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node according to the first link information in the edge node corresponding to the fourth condition node in the rule network graph can specifically include:

[0112] Based on the first link information and rule identifier hash in the edge node corresponding to the fourth condition node in the rule network graph, determine the N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node.

[0113] Among them, the N fifth condition nodes have the same rule identifier as the fourth condition nodes.

[0114] It should be noted that the version number of the first target partition file can be determined to have changed as follows: If the version number of the partition containing the i-th condition node is greater than the version number of the edge node in the rule network graph, it is determined that the version number of the partition containing the i-th condition node has been modified.

[0115] It should be noted that, after retrieving the first rule condition to be modified through the index, in the initial field area, the corresponding file is found based on the field name to be indexed, and all condition nodes in the file are traversed. Matching condition nodes are queried, and the edge nodes pointed to by these condition nodes are recursively traversed until all reachable nodes have been traversed. Finally, all rule nodes queried during the traversal are returned. For obsolete condition nodes, a corresponding compaction version number and a set of data offsets deleted during this compaction can be stored in the compact_log. When an obsolete condition node is found, the version log content modified by querying the offset pointed to by the edge node can be used to determine the obsolete condition node. The location of the compressed data can be shown below:

[0116] v1:[offset1,offset2,...,offset N ] ...

[0118] v k :[offset1,offset2,...,offset M ]

[0119] Each entry in the version log consists of a defined start, end, and offset. The specific size of each entry is determined by the user-defined file size. For a given data query, if the fourth merge deletes the data at index 5 and inserts data at index 7 that was originally appended to the end of the file (assuming a total of 1 million data entries), the corresponding additions in the compact_log are as follows: v4:0-4:0, 5:del, 6:+1, 7-999999:0, 1000000:+9999993. During the query, the condition node is indexed based on the edge node. If the edge node record version is v3, the query will return all compressed log entries after v3, i.e., v4. A binary search is used to find the positional changes of the data at the corresponding location. When updating the offset, the version number is updated to v4. The compression of each field region is independent, thanks to the compact_log. Dependent nodes can modify their pointers via the compact_log during subsequent queries.

[0120] At this point, all modifications, additions, and deletions to condition nodes can take effect in real time. This approach comes at a cost: variable-length edge node data and field value node data must be compressed through periodic file rewriting. This compression requires traversing all valid data in the condition node file, identifying all currently used boundary points and field value nodes, rewriting this data in a new file, updating the offset of this data, and finally deleting the old file. Variable-length rule data areas also need to be discarded using the same method. However, sacrificing space for speed optimization is always worthwhile for disk space, aligning with the optimization strategies of many industry best practices, including Kafka and LevelDB. It also conforms to the LSMTree algorithm, converting random writes into append writes and periodically discarding the original data. All data in the file is stored in 4KB blocks (the minimum page size allocated by the operating system) to avoid random writes across pages. The calculation of whether multiple reachable nodes can be matched can be optimized through concurrent queries.

[0121] Therefore, a disk storage solution can replace the widely used memory storage solution, solving the problems of high memory overhead and complex distributed deployment of existing rule-based solutions while maximizing performance. Based on rule network graph-based condition node modeling, the rule network graph modeling can be applied to disk condition storage, using file offsets as pointers to accelerate matching. Replacing the memory solution with this approach in large-scale rule storage scenarios can effectively reduce storage costs.

[0122] Based on the same inventive concept, this application also provides a data processing device. (Specifically combined with...) Figure 5 Please provide a detailed explanation.

[0123] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0124] In some embodiments of this application, Figure 5 The data processing device shown can be installed in a computer device. For example... Figure 5 As shown, the data processing device 50 may specifically include:

[0125] The acquisition module 501 is used to acquire the first rule to be stored, which includes the first matching condition and the content of the first rule.

[0126] The generation module 502 is used to generate storage strategy information corresponding to the first matching condition in the rule storage structure table in memory. The storage strategy information includes the first partition information used to indicate the storage of the first matching condition on the disk. The first partition information is the partition information of the first partition on the disk.

[0127] The determination module 503 is used to determine the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk.

[0128] Storage module 504 is used to store the first matching condition in the first condition node and the first rule content in the first rule node.

[0129] Thus, the data processing device in this application embodiment can effectively solve the problem of high rule cost in memory storage by replacing memory storage with disk storage in large-scale data storage scenarios. Furthermore, by applying the rule network graph to disk condition storage, the relationship between nodes can be used as pointers, thereby accelerating the rule matching speed.

[0130] The data processing device 50 in the embodiments of this application will be described in detail below.

[0131] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a matching module, used to match the first field in the first matching condition with the field of each partition in the multiple partitions when the rule storage structure table includes a field data area, the field data area includes multiple partitions in the disk that store rules, and the multiple partitions are divided according to fields;

[0132] The determination module 503 can also be used to determine the partition that matches the first field as the first partition when there are multiple partitions, and to determine the partition information of the first partition as the first partition information.

[0133] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a creation module, used to create a new partition corresponding to the first field in the disk when there is no partition matching the first field among multiple partitions;

[0134] In this embodiment, the data processing device 50 may further include an adding module, which is used to add the newly added partition as the first partition to the field data area, and to determine the partition information of the newly added partition as the first partition information.

[0135] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a matching module for matching the conditional data in the first partition with the first conditional data of the first matching condition;

[0136] The determining module 503 can also be used to determine the second condition node storing the second condition data as the first condition node when the condition data in the first partition includes second condition data that matches the first condition data.

[0137] The determination module 503 can also be used to create a new condition node corresponding to the first partition in the rule network graph when the condition data in the first partition does not include condition data that matches the first condition data, and to determine the new condition node as the first condition node.

[0138] The determining module 503 can also be used to determine the third condition node storing the third condition data as the first condition node when the condition data in the first partition includes third condition data that matches at least part of the condition data in the first condition data.

[0139] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include an update module, used to update the first link information in the edge nodes of the regular network graph when the first condition node includes a first node and a second node, the first node is an original node in the regular network graph, the second node is a newly added node in the regular network graph, and the first node and the second node are nodes in the same partition or nodes in different partitions. The first link information is used to record the dependency relationship between every two conditions in the regular network graph, and the first link information includes the dependency relationship between the first node and the second node.

[0140] In one or more optional embodiments, the determining module 503 may also be used to determine the second rule node as the first rule node when the rule content of the second rule node configured for the second condition node includes the first rule content.

[0141] In this embodiment of the application, the data processing device 50 may further include a creation module, which is used to create a new rule node configured as a new condition node in the rule network graph when the rule content of the second rule node does not include the first rule content, and to determine the new rule node as the first rule node.

[0142] In this embodiment, the data processing device 50 may further include a traversal module, used to traverse other rule nodes in the rule network graph other than the second rule node when the rule content of the second rule node does not include the first rule content; the determination module 503 may further be used to determine the third rule node as the first rule node when there is a third rule node among the other rule nodes that matches the first rule content.

[0143] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a new module, which is used to add second link information to the first condition node when the first rule node is a new rule node or a third rule node. The second link information is used to record the first rule node associated with the first rule node in the rule network graph.

[0144] In one or more optional embodiments, the first condition data of the first matching condition includes field values ​​and matching patterns, and the first condition node is stored according to the field values ​​and matching patterns;

[0145] In this embodiment of the application, the data processing device 50 may further include a sorting module, which is used to store the first condition data in the first condition node in ascending order of field values ​​when the first condition data of the first matching condition is a definite value condition data.

[0146] Storage module 504 can also be used to store regular expressions of non-deterministic condition data in the first condition node and store the non-deterministic condition data in the expression library when the first condition data of the first matching condition is non-deterministic condition data.

[0147] In one or more optional embodiments, the storage module 504 may be specifically used to, when the determined value condition data includes first data less than or equal to a preset determined value and second data greater than a preset determined value, sequentially store the first data and pointer data corresponding to the second data in the first condition node in ascending order of field values, wherein the pointer data is used to indicate the node storing the second data.

[0148] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a recording module for recording the number of times each condition node is used in the rule network graph;

[0149] The acquisition module 501 can also be used to acquire discarded condition nodes in the rule network graph whose number of uses is less than or equal to a preset threshold.

[0150] In this embodiment, the data processing device 50 may further include a processing module for deleting obsolete condition nodes from the rule network graph and updating the version number and first link information of the edge nodes in the rule network graph. The version number is used for data synchronization between memory and disk, and the first link information is used to record the dependency relationship between every two condition nodes in the rule network graph.

[0151] In one or more optional embodiments, the rule storage structure table includes an initial field area and a memory area; wherein, the initial field area includes the initial matching conditions of the first rule; and the memory area is used to record rule modifications and version numbers of edge nodes in the rule network graph.

[0152] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include a creation module for creating a result set and a list of condition nodes according to the rule query request;

[0153] In this embodiment of the application, the data processing device 50 may further include a traversal module for traversing the initial field storage area in the rule storage structure table; the acquisition module 501 may further be used to acquire the initial target matching conditions in the initial field storage area that correspond to the fields in the rule query request.

[0154] The determination module 503 can also be used to determine the fourth condition node for storing target matching conditions in the rule network graph of the disk;

[0155] The determination module 503 can also be used to determine N fifth condition nodes in the regular network graph that have a dependency relationship with the fourth condition node, based on the first link information in the edge node corresponding to the fourth condition node in the regular network graph.

[0156] In this embodiment of the application, the data processing device 50 may further include an adding module for adding the fourth condition node and N fifth condition nodes to the condition node list.

[0157] The module can also be used to add the target rule content corresponding to the target matching condition in memory to the result set when the version number of the partition where the i-th condition node in the condition node list is located has not been modified, but the version number of the edge node in the rule network graph recorded in the memory area of ​​the rule storage structure table has been modified.

[0158] In one or more optional embodiments, the data processing device 50 in this application embodiment may further include an update module, which is used to update the version number of the edge node in the rule network graph according to the version number of the partition where the i-th condition node is located and the version number of the edge node in the rule network graph recorded in the memory area when the version number of the partition where the i-th condition node is located is modified.

[0159] The update module can also be used to update the first link information in the edge node according to the updated version number of the edge node in the rule network graph;

[0160] In this embodiment of the application, the data processing device 50 may further include an adjustment module, used to adjust the condition node list according to the updated first link information;

[0161] In this embodiment of the application, the data processing device 50 may further include a traversal module, which is used to traverse the target rule nodes associated with each condition node in the adjusted condition node list according to the adjusted condition node list.

[0162] In this embodiment, the data processing device 50 may further include an adding module for adding the target rule content stored in the target rule node to the result set.

[0163] In one or more optional embodiments, the determining module 503 can also be used to determine, in the case that the edge node further includes a rule identifier hash, which is used to record the rule identifier corresponding to each condition node in the rule network graph, N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node based on the first link information in the edge node corresponding to the fourth condition node in the rule network graph and the rule identifier hash.

[0164] Among them, the N fifth condition nodes have the same rule identifier as the fourth condition nodes.

[0165] Based on the same inventive concept, this application also provides a computer device. (Specifically combined with...) Figure 6 Please provide a detailed explanation.

[0166] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0167] like Figure 6 As shown, the computer device may include at least one of the following as described in the embodiments of this application: an electronic device, a server. The computer device may include a processor 601 and a memory 602 storing computer program instructions.

[0168] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0169] Memory 602 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory. In a particular embodiment, memory 602 includes solid-state storage (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0170] The processor 601 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0171] In one example, the computer device may also include a communication interface 603 and a bus 610. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0172] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0173] Bus 610 includes hardware, software, or both, that couples components of a flow control device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0174] The computer device can execute the data processing method described in the embodiments of this application, thereby achieving the combination Figures 1 to 5 The data processing methods and apparatus described.

[0175] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the data processing methods in the above embodiments.

[0176] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0177] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0178] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0179] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the first rule to be stored, the first rule including the first matching condition and the content of the first rule; Storage policy information corresponding to the first matching condition is generated in the rule storage structure table in memory. The storage policy information includes the first partition information for indicating the storage of the first matching condition on the disk. The first partition information is the partition information of the first partition on the disk. The rule storage structure table is the core table for dynamically managing permission rules. The rule storage structure table includes an initial field area and a memory area. The initial field area includes the initial matching condition of the first rule. The memory area is used to record rule modifications and version numbers of edge nodes in the rule network graph; The first condition node corresponding to the first partition and the first rule node configured for the first condition node are determined from the rule network graph of the disk. The rule network graph is constructed based on the rule storage structure table. The rule network graph is a directed acyclic graph. The matching conditions of the rule are abstracted into condition nodes. The rule content of the rule is abstracted into rule nodes. The matching pattern of the matching condition is abstracted into the edge between condition nodes. The matching pattern includes a combination of AND and OR operations. The first matching condition is stored in the first condition node, and the first rule content is stored in the first rule node.

2. The method according to claim 1, characterized in that, The rule storage structure table includes a field data area, which includes multiple partitions on the disk that store rules, and the multiple partitions are divided according to fields; The step of generating storage strategy information corresponding to the first matching condition in the rule storage structure table in memory includes: Match the first field in the first matching condition with the fields of each of the plurality of partitions; If a partition that matches the first field is included among the plurality of partitions, the partition that matches the first field is determined as the first partition, and the partition information of the first partition is determined as the first partition information.

3. The method according to claim 2, characterized in that, The method further includes: If the plurality of partitions does not include a partition that matches the first field, a new partition corresponding to the first field is created on the disk; The newly added partition is added to the field data area as the first partition, and the partition information of the newly added partition is determined as the first partition information.

4. The method according to claim 1, characterized in that, Determining the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk includes: Match the conditional data in the first partition with the first conditional data of the first matching condition; If the condition data in the first partition includes second condition data that matches the first condition data, the second condition node storing the second condition data is determined as the first condition node; If the condition data in the first partition does not include condition data that matches the first condition data, a new condition node corresponding to the first partition is created in the rule network graph, and the new condition node is determined as the first condition node. If the condition data in the first partition includes third condition data that matches at least a portion of the condition data in the first condition data, the third condition node storing the third condition data is determined as the first condition node.

5. The method according to claim 4, characterized in that, The first condition node includes a first node and a second node. The first node is an original node in the rule network graph, and the second node is a newly added node in the rule network graph. The first node and the second node are nodes in the same partition or nodes in different partitions. The method further includes: Update the first link information in the edge nodes of the rule network graph. The first link information is used to record the dependency relationship between every two conditions in the rule network graph. The first link information includes the dependency relationship between the first node and the second node.

6. The method according to claim 4, characterized in that, Determining the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk includes: If the rule content of the second rule node configured for the second condition node includes the first rule content, the second rule node is determined as the first rule node; If the rule content of the second rule node does not include the first rule content, a new rule node is created in the rule network graph as configured for the new condition node, and the new rule node is identified as the first rule node. If the rule content of the second rule node does not include the first rule content, traverse all other rule nodes in the rule network graph except for the second rule node; if there is a third rule node among the other rule nodes that matches the first rule content, determine the third rule node as the first rule node.

7. The method according to claim 6, characterized in that, The method further includes: When the first rule node is the newly added rule node or the third rule node, a second link information is added to the first condition node. The second link information is used to record the first rule node associated with the first rule node in the rule network graph.

8. The method according to claim 1, characterized in that, The first condition data of the first matching condition includes field values ​​and matching patterns, and the first condition node is stored according to the field values ​​and the matching patterns; The step of storing the first matching condition in the first condition node includes: When the first condition data of the first matching condition is a definite value condition data, the first condition data is stored in the first condition node in ascending order of field values; When the first condition data of the first matching condition is a non-deterministic condition data, the regular expression of the non-deterministic condition data is stored in the first condition node, and the non-deterministic condition data is stored in the expression library.

9. The method according to claim 8, characterized in that, The determined value condition data includes first data that is less than or equal to a preset determined value and second data that is greater than the preset determined value; storing the first condition data in the first condition node in ascending order of field values ​​includes: In the first condition node, the first data and the pointer data corresponding to the second data are stored sequentially in ascending order of field values. The pointer data is used to indicate the node where the second data is stored.

10. The method according to claim 1, characterized in that, After storing the first matching condition in the first condition node and the first rule content in the first rule node, the method further includes: Record the number of times each condition node is used in the rule network graph; Obtain discarded condition nodes in the rule network graph whose usage frequency is less than or equal to a preset threshold; The discarded condition node is deleted from the rule network graph, and the version number and first link information of the edge node in the rule network graph are updated. The version number is used for data synchronization between the memory and the disk, and the first link information is used to record the dependency relationship between every two condition nodes in the rule network graph.

11. The method according to claim 1, characterized in that, The method further includes: Based on the rule query request, create a result set and a list of condition nodes; Traverse the initial field storage area in the rule storage structure table to obtain the initial target matching conditions in the initial field storage area that correspond to the fields in the rule query request. In the rule network graph of the disk, a fourth condition node is determined for storing the target matching conditions; Based on the first link information in the edge node corresponding to the fourth condition node in the rule network graph, determine N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node; Add the fourth condition node and the N fifth condition nodes to the condition node list; If the version number of the partition containing the i-th condition node in the condition node list has not been modified, and the version number of the edge node in the rule network graph recorded in the memory area of ​​the rule storage structure table has been modified, the target rule content corresponding to the target matching condition in the memory is added to the result set.

12. The method according to claim 11, characterized in that, The method further includes: If the version number of the partition where the i-th condition node is located is modified, update the version number of the edge node in the rule network graph according to the version number of the partition where the fourth condition node is located and the version number of the edge node in the rule network graph recorded in the memory area. Update the first link information in the edge node according to the updated version number of the edge node in the network graph according to the rules; Adjust the list of condition nodes according to the updated first link information; Based on the adjusted list of condition nodes, traverse the target rule nodes associated with each condition node in the adjusted list of condition nodes, and add the target rule content stored in the target rule nodes to the result set.

13. The method according to claim 11 or 12, characterized in that, The edge node also includes a rule identifier hash, which is used to record the rule identifier corresponding to each condition node in the rule network graph; The step of determining N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node based on the first link information in the edge nodes corresponding to the fourth condition node includes: Based on the first link information in the edge node corresponding to the fourth condition node in the rule network graph and the rule identifier hash, determine N fifth condition nodes in the rule network graph that have a dependency relationship with the fourth condition node; Among them, the N fifth condition nodes have the same rule identifier as the fourth condition node.

14. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a first rule to be stored, the first rule including a first matching condition and the content of the first rule; A generation module is used to generate storage policy information corresponding to the first matching condition in a rule storage structure table in memory. The storage policy information includes first partition information for indicating the storage of the first matching condition on the disk. The first partition information is the partition information of the first partition on the disk. The rule storage structure table is a core table for dynamically managing permission rules. The rule storage structure table includes an initial field area and a memory area. The initial field area includes the initial matching condition of the first rule. The memory area is used to record rule modifications and version numbers of edge nodes in the rule network graph; The determination module is used to determine the first condition node corresponding to the first partition and the first rule node configured for the first condition node from the rule network graph of the disk. The rule network graph is constructed according to the rule storage structure table. The rule network graph is a directed acyclic graph. The matching conditions of the rule are abstracted into condition nodes. The rule content of the rule is abstracted into rule nodes. The matching pattern of the matching condition is abstracted into the edge between the condition nodes. The matching pattern includes a combination of AND and OR operations. The storage module is used to store the first matching condition in the first condition node and the first rule content in the first rule node.

15. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-13.

Images