A general rule-based filtering method, apparatus, computing device, and readable storage medium

Abstract

This invention proposes a general rule-based filtering method, apparatus, computing device, and readable storage medium. The method includes the following steps: defining meta-rules for filtering, wherein the meta-rules include at least one attribute category among attribute name, attribute storage location, attribute value type, attribute filtering operation set, and attribute selectable value range; extracting multiple attribute values ​​from the original data and importing them into the attribute categories of the meta-rules for storage; selecting at least one meta-rule and defining filtering logic for the meta-rule to form a filtering dimension; performing set operations on multiple filtering dimensions to form a filtering unit; performing multiple set operations on multiple filtering units to form a filtering model with a tree structure; and batch executing multiple filtering models to obtain a target set. This invention can improve the execution efficiency of general filtering model algorithms required for refined operations and alleviate the problem of increasingly slow target user extraction.

Description

Technical Field

[0001] This invention relates to the field of big data and data analysis, and more specifically, to a general rule-based filtering method, apparatus, computing device, and readable storage medium. Background Technology

[0002] Currently, to achieve truly refined operations and personalized operations for each user in different scenarios, it's essential to first acquire a large amount of data, such as user attributes, behaviors, lifecycles, and usage scenarios. Based on this data, algorithms are used to calculate and match corresponding strategies, which are then deployed. The algorithms are further optimized and iterated based on the effectiveness and feedback of these strategies. Moreover, as the granularity of operations increases, the associated operational costs grow exponentially. All of this is absolutely impossible to achieve manually; therefore, it's necessary to consider transforming it into machine algorithms to reduce costs and improve efficiency.

[0003] The industry standard practice is to establish a general rule-based filtering system based on a multi-dimensional attribute matrix. This allows operations personnel to solidify operational strategies into algorithms for refined business operations. However, as the level of refinement increases and the number of filtering models grows, the filtering process becomes increasingly slow. Summary of the Invention

[0004] In view of the above problems, the present invention provides a general rule filtering method, apparatus, computing device and readable storage medium to solve the problem that the filtering process is becoming increasingly slow as the number of filtering models increases in the prior art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a general rule-based filtering method, comprising the following steps: defining meta-rules for filtering, wherein the meta-rules include at least one attribute category among attribute name, attribute storage location, attribute value type, attribute filtering operation set, and attribute selectable value range; extracting multiple attribute values ​​from the original data and importing them into the attribute categories of the meta-rules for storage; selecting at least one meta-rule and defining filtering logic for the meta-rule to form a filtering dimension; performing set operations on multiple filtering dimensions to form a filtering unit; performing multiple set operations on multiple filtering units to form a filtering model with a tree structure; and batch executing multiple filtering models to obtain a target set.

[0007] The execution of the filtering model includes: for each filtering unit in the filtering model, sequentially traversing the filtering dimensions of all filtering units; executing the filtering statement of the filtering dimension, performing a set operation on the filtering results according to the relationship of the same parent node to obtain the target set, and participating in the relationship operation of the next level; executing the filtering dimensions of all filtering units to obtain the target set corresponding to each filtering unit; merging the target sets of all filtering units to form the target set corresponding to the filtering model.

[0008] As a preferred embodiment, the execution of the filtering model further includes: analyzing the filtering execution paths of multiple filtering models and extracting common nodes, wherein the common nodes include filtering dimensions or filtering units; replacing the common nodes of each filtering model with virtual nodes; executing the filtering statements of the common nodes and saving the execution results; when executing the filtering model, determining whether the filtering dimension or filtering unit is a virtual node, if so, substituting the pre-executed and saved execution results as the execution results of the virtual node, otherwise, executing the filtering statements.

[0009] As a preferred embodiment, before extracting multiple attribute values ​​from the original data, the method further includes cleaning and transforming the original data, such as obtaining birthday information from the ID card information after excluding invalid ID card information, and then converting the birthday information into age.

[0010] As a preferred embodiment, the execution filtering model determines whether multiple attribute values ​​are stored in the same record table. If so, the multiple attribute values ​​in the same record table are filtered once.

[0011] Secondly, the present invention provides a general rule filtering device, comprising: a meta-rule definition module for defining meta-rules for filtering, wherein the meta-rules include at least one attribute category among attribute name, attribute storage location, attribute value type, attribute filtering operation set, and attribute selectable value range; an attribute import module for extracting multiple attribute values ​​from the original data and importing them into the attribute categories of the meta-rules; an attribute storage module for storing the attribute values ​​in a database; a model building module for selecting at least one meta-rule and defining filtering logic for the meta-rule to form a filtering dimension; performing set operations on multiple filtering dimensions to form a filtering unit; performing multiple set operations on multiple filtering units to form a filtering model with a tree structure; and a filtering execution module for batch executing multiple filtering models to obtain a target set.

[0012] The filtering execution module includes: a filtering traversal unit, used to traverse the filtering dimensions of all filtering units sequentially for each filtering unit in the filtering model; a filtering dimension operation unit, used to execute the filtering statement of the filtering dimension, perform set operations on the filtering results according to the relationship of the same parent node, obtain the target set, and participate in the relationship operation of the next level; a filtering unit operation unit, used to execute the filtering dimensions of all filtering units to obtain the target set corresponding to each filtering unit; and a filtering model operation unit, used to merge the target sets of all filtering units to form the target set corresponding to the filtering model.

[0013] As a preferred embodiment, the filtering execution module further includes: a node substitution unit, used to analyze the filtering execution path of multiple filtering models, extract common nodes, the common nodes including filtering dimensions or filtering units; replace the common nodes of each filtering model with virtual nodes; execute the filtering statements of the common nodes and save the execution results; and a node judgment unit, used to determine whether the filtering dimension or filtering unit is a virtual node when executing the filtering model. If it is, the pre-executed and saved execution results are substituted as the execution results of the virtual nodes; otherwise, the filtering statements are executed.

[0014] As a preferred embodiment, the system also includes a cleaning and transformation module, which cleans and transforms the original data before extracting multiple attribute values ​​from the original data. For example, after excluding invalid ID card information, the system obtains birthday information from the ID card information and then converts the birthday information into age.

[0015] As a preferred embodiment, the filtering execution module further includes an attribute judgment unit, which is used to determine whether multiple attribute values ​​are stored in the same record table when the filtering model is executed. If so, the multiple attribute values ​​in the same record table are filtered once.

[0016] Thirdly, the present invention provides a computing device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing the method described above according to the obtained program.

[0017] Fourthly, the present invention provides a computer-readable storage medium including computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method described in any of the preceding claims.

[0018] Compared with the prior art, the beneficial effects of the present invention include: it can improve the execution efficiency of the general screening model algorithm required for refined operation, and alleviate the problem that the extraction of target users becomes slower due to the increasing number of models and increasingly refined screening dimensions caused by the continuous refinement of business operations. Attached Figure Description

[0019] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0020] Figure 1 A flowchart illustrating the general rule-based filtering method provided in this embodiment of the invention;

[0021] Figure 2 A flowchart illustrating the execution of the filtering model provided in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram illustrating the process of extracting and replacing common nodes according to an embodiment of the present invention;

[0023] Figure 4 A tree structure diagram of the screening model provided in the embodiments of the present invention;

[0024] Figure 5 Interface of the general rule filtering method provided in the embodiments of the present invention Figure 1 ;

[0025] Figure 6 Interface of the general rule filtering method provided in the embodiments of the present invention Figure 2 ;

[0026] Figure 7 A schematic diagram of the structure of the general rule filtering device provided in the embodiments of the present invention;

[0027] Figure 8 This is a schematic diagram of the structure of the filtering execution module provided in an embodiment of the present invention. Detailed Implementation

[0028] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0029] According to one embodiment of the present invention, Figure 1 As shown. This invention provides a general rule-based filtering method, comprising the following steps:

[0030] S100 defines the meta-rules for filtering. The meta-rules include at least one attribute category among the attribute name, attribute storage location, attribute value type, attribute filtering operation set, and attribute optional value range.

[0031] The attribute storage location is defined as the location where the attribute value is stored. In this embodiment, the storage location is in the database, so it is defined as a database table field.

[0032] Attribute value types can be categorized into character, numeric, and date types. Considering performance and readability requirements, date attributes are handled specially in terms of storage and display. Firstly, they are further subdivided into date types (accurate to the day) and date types (accurate to the second), facilitating precise or relatively rough filtering by operations personnel. Date attributes are stored in the database as timestamps, similar to 1565491864, a storage format that promotes fast indexing and retrieval. When displayed on the interface, date attributes are displayed as something like "2019-08-11 10:51:03" or "2019-08-11".

[0033] The type of an attribute value affects the set of filtering operations that the attribute supports. For example, character attributes do not support comparisons of greater than or less than.

[0034] The set of attribute filtering operations can be defined as one or more of the following: equal to, greater than, greater than or equal to, less than, less than or equal to, not equal to, contain, do not contain, empty, not empty, range, and custom. The meta-rules themselves are also stored in the database and provided for reference by other modules.

[0035] S200: Extract multiple attribute values ​​from the original data and import them into the attribute categories of the meta-rules. According to the definition of the meta-rules, transform the original data into filterable attribute values.

[0036] Before extracting multiple attribute values ​​from the raw data, the process also includes cleaning and transforming the raw data to generate filterable attribute values ​​and storing them in the table fields defined by the meta-rules. For example, after excluding invalid ID card information, the birthdate information is obtained from the ID card information, and then the birthdate information is converted into age.

[0037] S300 stores attribute values ​​in the database.

[0038] Based on a database implementation, not limited to SQL or NoSQL, attributes are stored in a standardized form and provided with fast retrieval capabilities. This embodiment uses MySQL as the attribute storage module. Multiple attributes can be stored in a single table, and different attributes can be stored in different tables.

[0039] S400, select at least one meta-rule and define filtering logic for the meta-rule to form a filtering dimension; perform a set operation on multiple filtering dimensions to form a filtering unit; perform multiple set operations on multiple filtering units to form a filtering model with a tree structure.

[0040] In this embodiment of the invention, a graphical interface is provided to define the model. Operators can select multiple meta-rules from the predefined meta-rules, define filtering logic for each meta-rule, and combine multiple meta-rules to form a multi-dimensional filtering model.

[0041] like Figure 5 and 6 The image shown is a screenshot of the interface for the general rule-based filtering method. Figure 1 The document shows two filtering dimensions: membership level and membership registration channel. These two dimensions are linked (AND). Each dimension can be used with similar... Figure 5 The interface shown allows you to edit each dimension. For example, in the membership level channel, the filter condition is "inclusive," and the filter value is "paid members" and "platinum card members." This means that if a member's membership level is either a paid member or a platinum card member, then this filter dimension is met.

[0042] It should be understood that each dimension of the screening model defines the screening logic for a certain attribute. That is, applying this rule can yield a definite result: whether an individual's attribute value meets or does not meet the requirement.

[0043] The multiple dimensions of the screening model should have intersection, union, and complement relationships, and should support multi-level set intersection, union, and complement relationships. For example... Figure 5 As shown, this represents the intersection of two filtering dimensions: members whose membership level includes either paid or platinum card membership, and whose registration channel is CJL, are eligible. This actually involves two relationships. The first layer is the membership level dimension, where paid and platinum card memberships are the union of their respective options. The second layer is the intersection of the two dimensions.

[0044] Furthermore, the screening model not only includes the definition of screening dimensions, but also needs to maintain the hierarchy and intersection, union, and complement relationships of the screening dimensions, ultimately forming a relation forest. In other words, the screening model is a forest structure of screening relationships, where the first-level nodes are initially related by union. Each first-level node is a tree structure, and intersection, union, and complement operations are performed on the relationships of its child nodes. For example... Figure 4 As shown.

[0045] Preferably, the filtering model defines an execution strategy, specifying whether it is a single execution or repeated execution. If it is repeated execution, the repetition period is defined. The repetition period can be defined using year, month, day, hour, minute, second, or weekday.

[0046] S500, execute multiple filtering models in batches to obtain the target set.

[0047] Based on the execution time strategy settings of the filtering model, the filtering rules are executed at a predetermined time to obtain the target group whose attribute values ​​meet the conditions. Its implementation logic is expressed by the following pseudocode: Search for all filtering model identifiers to be executed based on the current time, and for each filtering model identifier, look up the associated filtering dimension definition in the corresponding table of the database. Each model has a model name and a model identifier; the model identifier is a string that uniquely identifies a model. One model corresponds to several filtering dimension definitions, for example... Figure 5 This represents the intersection of two filtering dimensions. The definitions of the filtering dimensions are stored in a database table, and filtering dimensions belonging to the same model are identified by their model identifiers.

[0048] Because there are multi-level intersection, union, and complement relationships between the filtering dimensions, it is necessary to first construct a forest of relationships between the filtering dimensions in memory. For example... Figure 4 As shown, in computer memory, a relational tree with multiple filtering dimensions can be constructed using arrays or linked lists to form a relational forest.

[0049] When executing the filtering model, it determines whether multiple attribute values ​​are stored in the same record table. If so, it performs a single filtering operation on the multiple attribute values ​​in the same record table. It can find the meta-rule definition based on the filtering model, and then locate the table where the attributes are stored based on the meta-rule definition.

[0050] For example, a member information table stores age and gender information. When we want to query female members who are over 25 years old, logically we need to perform two filtering steps: first, find members who are over 25 years old, then find members who are female, and finally take the intersection of the two to get the target members. This is inefficient. It can be combined into a single traversal, determining whether each member meets the requirements of both age and gender attributes, and extracting the members that meet the conditions.

[0051] See Figure 2 The process of executing the above-mentioned screening model includes:

[0052] S5011, for each filter unit in the filter model, iterate through all the filter dimensions of all filter units in turn.

[0053] For each relation tree in the relation forest, traverse to the leaf nodes in turn. Each leaf node can be translated into a certain filtering operation on a certain field of a certain table in the database, such as querying all records in the age field of the member basic information table where the value is greater than 25.

[0054] S5012 executes the filtering statement for the filtering dimension node, performs set operations on the filtering results based on the relationship of the same parent node, obtains the target set, and participates in the relationship operation of the previous level.

[0055] The filtering dimensions of sibling nodes under the same parent node are categorized and combined according to the different tables being filtered. Filtering dimensions belonging to the same table and using the same filtering operator are merged into a single filtering statement, with the AND (intersection), OR (union), or NOT IN (complement) operation determined by the relationship between the parent nodes.

[0056] S5013, execute the filtering dimensions of all filtering units to obtain the target set corresponding to each filtering unit.

[0057] S5014 merges the target sets of all filtering units to form the target set corresponding to the filtering model.

[0058] See Figure 3 Before executing multiple filtering models in batches, the following steps are also included:

[0059] S5021 analyzes the filtering execution paths of multiple filtering models and extracts common nodes, which include filtering dimensions or filtering units. Common nodes include... Figure 4 As shown in the image.

[0060] S5022, replace the common nodes of each screening model with virtual nodes.

[0061] S5023, execute the filtering statement of the common node and save the execution result.

[0062] S5024, when executing the filtering model, determine whether the filtering dimension or filtering unit is a virtual node. If so, substitute the pre-executed and saved execution result as the execution result of the virtual node. If not, execute the filtering statement.

[0063] See Figure 7 The present invention provides a universal rule-based filtering device, comprising:

[0064] The meta-rule definition module 100 is used to define the meta-rules for filtering. The meta-rules include at least one attribute category among the attribute name, attribute storage location, attribute value type, attribute filtering operation set, and attribute optional value range.

[0065] The attribute import module 200 is used to extract the attribute values ​​of various attributes from the original data and import them into the attribute categories of the meta-rules.

[0066] The attribute storage module 300 is used to store attribute values ​​in the database.

[0067] The model building module 400 is used to select at least one meta-rule and define filtering logic for the meta-rule to form a filtering dimension; perform set operations on multiple filtering dimensions to form a filtering unit; and perform multiple set operations on multiple filtering units to form a filtering model with a tree structure.

[0068] The filtering execution module 500 is used to execute multiple filtering models in batches to obtain the target set.

[0069] See Figure 8 The aforementioned filtering execution module 500 includes:

[0070] Filter traversal unit 5011 is used to traverse all filter dimensions of all filter units in turn for each filter unit in the filter model.

[0071] The filtering dimension operation unit 5012 is used to execute the filtering statement of the filtering dimension node, perform set operation on the filtering results according to the relationship of the same parent node, obtain the target set, and participate in the relationship operation of the previous level.

[0072] The filtering unit operation unit 5013 is used to execute the filtering dimension nodes of all filtering units to obtain the target set corresponding to each filtering unit.

[0073] The filtering model operation unit 5014 is used to merge the target sets of all filtering units to form the target set corresponding to the filtering model.

[0074] In addition, the filtering execution module 500 also includes a node substitution unit and a node judgment unit. The node substitution unit is used to analyze the filtering execution path of multiple filtering models, extract common nodes, which include filtering dimensions or filtering units; replace the common nodes of each filtering model with virtual nodes; execute the filtering statements of the common nodes, and save the execution results.

[0075] The node judgment unit is used to determine whether the filtering dimension or filtering unit is a virtual node when executing the filtering model. If it is, the execution result that has been executed and saved in advance is substituted as the execution result of the virtual node. If not, the filtering statement is executed.

[0076] Furthermore, the filtering execution module 500 also includes an attribute judgment unit, which is used to determine whether multiple attribute values ​​are stored in the same record table when executing the filtering model. If so, the multiple attribute values ​​in the same record table are filtered once.

[0077] Preferably, the device further includes a cleaning and transformation module, which is used to clean and transform the original data before extracting multiple attribute values ​​from the original data. For example, after excluding invalid ID card information, the birthday information is obtained from the ID card information and then converted into age.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0079] The present invention also provides a computing device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing any of the methods described above according to the obtained program.

[0080] The present invention also provides a computer-readable storage medium including computer-readable instructions, which, when read and executed by a computer, cause the computer to perform any of the methods described above.

[0081] In summary, the beneficial effects of this invention include: improving the execution efficiency of general screening model algorithms required for refined operations, and alleviating the problem of increasingly slower target user extraction caused by the increasing number of models and increasingly refined screening dimensions resulting from the continuous refinement of business operations.

[0082] It should be understood that if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A method of general rule screening, characterized by, The method comprises the following steps: At least one of the attribute category in the attribute value type, the attribute filtering operation set and the attribute optional value range; Extracting a plurality of attribute values in the original data into the attribute category corresponding to the imported meta rule and storing them; Selecting at least one meta rule and defining filtering logic for the meta rule to form a filtering dimension; Performing a collection operation on a plurality of filtering dimensions to form a filtering unit; Performing a plurality of collection operations on a plurality of filtering units to form a filtering model with a tree structure; Batch executing a plurality of filtering models to obtain a target set; Wherein, the execution of the filtering model comprises: For each filtering unit in the filtering model, sequentially traverse the filtering dimensions of all filtering units; Execute the filtering statement of the filtering dimension, collect the filtering results according to the relationship of the same parent node, obtain the target set, and participate in the relationship operation of the upper level; Execute the filtering dimensions of all filtering units to obtain the target set corresponding to each filtering unit; Combine the target sets of all filtering units to form the target set corresponding to the filtering model; The execution of the filtering model further comprises: Analyze the filtering execution path of a plurality of filtering models, extract common nodes, and the common nodes include filtering dimensions or filtering units; Replace each filtering model common node with a virtual node; Execute the filtering statement of the common node and save the execution result; When executing the filtering model, determine whether the filtering dimension or the filtering unit is a virtual node, if yes, use the pre-executed and saved execution result as the execution result of the virtual node, if not, execute the filtering statement.

2. The general rule screening method according to claim 1, characterized by, Before the plurality of attribute values of the original data are extracted, the original data is cleaned and converted, such as obtaining the birthday information from the ID card information after excluding the illegal ID card information, and then converting the birthday information into age.

3. The general rule screening method according to claim 1, characterized by, The execution of the filtering model further comprises: Determine whether a plurality of attribute values are stored in the same record table, if yes, perform a filtering operation on the plurality of attribute values in the same record table.

4. A general rule screening apparatus characterized by comprising: Comprise: The meta rule definition module is used for defining the meta rule of filtering, and the meta rule comprises at least one of the attribute name, the attribute storage location, the attribute value type, the attribute filtering operation set and the attribute optional value range; The attribute import module is used for extracting a plurality of attribute values of the original data into the attribute category corresponding to the imported meta rule; The attribute storage module is used for storing the attribute values in the database; The model construction module is used for selecting at least one meta rule, defining filtering logic for the meta rule to form a filtering dimension, performing a collection operation on a plurality of filtering dimensions to form a filtering unit, and performing a plurality of collection operations on a plurality of filtering units to form a filtering model with a tree structure; The filtering execution module is used for batch executing a plurality of filtering models to obtain a target set; Wherein, the filtering execution module comprises: The filtering traversal unit is used for sequentially traversing the filtering dimensions of all filtering units for each filtering unit in the filtering model; Execute the filtering statement of the filtering dimension, collect the filtering results according to the relationship of the same parent node, obtain the target set, and participate in the relationship operation of the upper level; The screening dimension operation unit is configured to execute the screening statements of the screening dimensions, perform a set operation on the screening results according to the relations of the same parent node, obtain a target set, and participate in the relation operation of the upper level; The screening unit operation unit is configured to execute the screening dimensions of all the screening units, and obtain the target set corresponding to each screening unit; The screening model operation unit is configured to combine the target sets of all the screening units to form a target set corresponding to the screening model; The screening execution module further comprises: The node replacement unit is configured to analyze the screening execution paths of the multiple screening models, extract common nodes, the common nodes including screening dimensions or screening units, replace the common nodes of each screening model with virtual nodes, execute the screening statements of the common nodes, and save the execution results; The node judgment unit is configured to, when executing the screening model, judge whether the screening dimensions or the screening units are virtual nodes, if yes, replace the execution results saved in advance as the execution results of the virtual nodes, and if no, execute the screening statements.

5. The general rule screening device of claim 4, wherein, The cleaning conversion module is further configured to, before extracting the multiple attribute values of the original data, clean and convert the original data, for example, after excluding illegal ID card information, obtaining birthday information from the ID card information, and converting the birthday information into age.

6. The general rule screening device of claim 4, wherein, The screening execution module further comprises an attribute judgment unit configured to, when executing the screening model, judge whether the multiple attribute values are stored in the same record table, if yes, perform a screening on the multiple attribute values in the same record table.

7. A computing device, comprising: The memory is configured to store program instructions; The processor is configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 3 according to the obtained program execution right. The computer readable instructions, when read and executed by the computer, enable the computer to execute the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, ​

Images