A low-frequency index-based fast calculation method for text logical expression

Abstract

This invention discloses a fast calculation method for text logical expressions based on low-frequency indexing. First, the expression is traversed and grouped according to the operation conditions. Then, the scores of each OR operation group are calculated through dynamic weighting, and the OR operation group with the highest keyword frequency score is used as the index group. Next, keywords in all groups are globally deduplicated to generate a matching machine for multi-modal keyword matching. Finally, based on the original binary tree rules, OR nodes are merged into bitmap nodes, and the target text is matched with the matching machine. The matched keywords form a set, and the intersection of this set and the index is calculated. Whether the intersection is greater than 0 determines whether a node is matched. This invention reduces the computational load by using low-frequency indexing and improves computational performance through bitwise operations, ensuring high computational performance even with an increased number of keywords and expressions, thus meeting the timeliness requirements of business data.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, and in particular relates to a method for fast calculation of text logic expressions based on low-frequency indexes. Background Technology

[0002] Many enterprises and organizations in the industry need to extract target text using logical expressions based on keyword combinations as rules. In this business scenario, as the number of keywords increases and the logical expressions proliferate, the computational load increases exponentially. Once the number of expressions reaches a certain level, it may even fail to meet the performance and data timeliness requirements of the business system. Currently, the following expression computation solutions are widely used in the industry:

[0003] (1) Zero-index calculation: The zero-index expression calculation steps include: taking an expression rule, splitting the expression into multiple calculation logic units, then traversing all logic units (keywords) of an expression, matching and calculating them one by one with the target text, obtaining the calculation result of each logic unit, and then calculating the expression based on the results of each logic unit, finally obtaining the calculation result of the expression. Traversing all expression rules and calculating them one by one according to the above steps. This scheme requires two layers of loop traversal for expression calculation. All expressions and keywords must be calculated, which has a high time complexity. When the number of keywords and expressions increases, the performance will decrease exponentially and cannot meet the business requirements.

[0004] (2) Full index calculation: The full index expression calculation steps include: first, traversing all expression rules, splitting each expression into multiple calculation logic units, deduplicating and storing all the split keywords, and establishing an index with each expression;

[0005] The algorithm iterates through the keywords, performs fuzzy matching calculations with the target text one by one, filters out the expressions corresponding to the matched keywords through the index, calculates the expressions based on the results of the keywords in each logical unit, and finally obtains the calculation result of the expression. It iterates through all indexed expression rules to complete the logical operation.

[0006] This solution indexes keywords and expressions, which can reduce the time spent calculating expressions. However, each keyword is indexed along with its corresponding expression, resulting in redundant calculations for expressions that are not ultimately matched. As the number of keywords and expressions increases, the performance still cannot meet business needs. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a fast calculation method for text logical expressions based on low-frequency indexing. When the number of keywords and expressions increases, the computational load is reduced by using low-frequency indexing, and the computational performance is improved by bitwise operations, ensuring that high-efficiency computational performance can still be maintained under such circumstances, thus meeting the timeliness requirements of business data.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for fast calculation of text logical expressions based on low-frequency indexing, characterized by comprising the following steps:

[0009] Step S1, Expression Grouping and Low-Frequency Index Construction: Traverse all expressions and convert expressions with mixed OR and AND operations into AND expressions with multiple OR operation groups according to the operation conditions;

[0010] Step S2, Low-frequency index construction: The scores of each OR operation group are calculated by dynamic weighting, sorted according to the frequency of keywords in the expression, and the OR operation group with the highest score is used as the index group and indexed with the expression to ensure that an expression is only indexed with one of the groups.

[0011] Step S3: After globally deduplicating the keywords in all groups, a matching machine is generated for use in multi-modal keyword matching.

[0012] Step S4, OR operation bitmap calculation: Based on the original binary tree rules, merge OR nodes into bitmap nodes. Match the target text with the matching machine obtained in step S3, generate a bitmap from the set of matched keywords, and calculate the intersection of this bitmap with the index obtained in step S2 to obtain the intersection bitmap. Determine whether a node is matched by checking if the capacity of the intersection bitmap is greater than 0. During the process of merging OR nodes into bitmap nodes, the leaf nodes of the binary tree are keywords, and the root and branch nodes are operators. Adjacent consecutive OR operations are merged into a single bitmap node, and the entire binary tree then becomes composed of bitmap nodes and AND operations. Multiple OR nodes are merged into a single bitmap node, and one bitmap node corresponds to one OR operation group.

[0013] Preferably, in step S2, after the expression is grouped, the score of each group is calculated by calculating the dynamic hit rate, word frequency, and keyword sensitivity of each group.

[0014] Preferably, the dynamic hit rate and word frequency are obtained from a historical thesaurus; the keyword sensitivity is calculated using the TF-IDF algorithm. (The lower the dynamic hit rate, the lower the word frequency, and the higher the keyword sensitivity, the higher the dynamic weighted score for each group).

[0015] Beneficial effects: Compared with existing technologies, this invention creates low-frequency indexes and bitmaps for keywords and expressions. Under the same business scenario, even with an increase in the number of keywords and expressions, it can still maintain efficient expression calculation. Specifically, it has the following advantages:

[0016] 1. With the target text and hit rate remaining unchanged, as the number of expressions and keywords increases, the effective computational load remains unchanged through low-frequency indexing technology. Combined with in-index group or operation bitmap calculations, efficient computation is maintained, so that the performance hardly decreases with the increase in the number of rules, and efficient computational performance is still maintained.

[0017] 2. With the increase in hit rate, the performance degradation of the index group or operation bitmap calculation is controllable even when the number of keywords in the index group increases. It can still ensure efficient calculation and meet business needs. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the principle framework of the fast text logic expression calculation method based on low-frequency indexing described in this invention.

[0019] Figure 2 This is a schematic diagram of the low-frequency index creation process according to a specific embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the bitmap calculation process for OR operation according to a specific embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0022] like Figure 1 As shown, this invention is a method for expression calculation that maintains high efficiency even when the number of keywords and expressions increases. The framework is as follows: Figure 1 As shown:

[0023] First, expression grouping: Group the expressions according to the operation conditions, that is, convert expressions with mixed OR and AND operations into AND expressions with multiple OR operations grouped together, as shown in the following example:

[0024] A like 'a' OR (B like 'b' AND C like 'c') can be transformed into the following grouping:

[0025] (A like 'a' OR B like 'b') AND (A like 'a' OR C like 'c')

[0026] Low-frequency index group election: Based on all expressions, a score for each group is calculated through dynamic weighting. The group with the highest score is selected as the index group, and an index is created for that expression.

[0027] Or operation bitmap transformation: First, a dictionary (mapping between numbers and keywords) is built based on keywords. For keywords in the index group, calculations are performed. Based on the original binary rule tree, OR nodes are merged into bitmap nodes to reduce recursive calculations and traversal depth, thereby improving performance.

[0028] Example of generating bitmaps using rules

[0029] (TEXT like “A” OR TEXT like “B”) AND (TEXT like “C” OR TEXT like “D” OR TEXT like “E” )

[0030] After optimization, it is

[0031] Text like “(A | B)&(C | D | E)”; Both nodes (A | B) and (C | D | E) are bitmap objects (which can be created using bitmap technology).

[0032] Logical expression calculation: The expression is calculated based on the results of each logical unit to obtain the expression calculation result.

[0033] 2) Expression evaluation process

[0034] First, iterate through all expressions, group them according to the operation conditions, and obtain all groups as index groups. Calculate the score of each group through dynamic weighting. The lower the frequency of the keyword, the higher the score. Take the group with the highest score as the index group and create an index for the expression. Ensure that an expression is only indexed by one group.

[0035] After deduplicating all grouped keywords globally, a matching machine is generated for use in multi-modal keyword matching.

[0036] The calculation process filters out redundant expression calculations. After fuzzy matching of keywords and target text (using a multi-mode matching machine to perform full matching in one go without traversal calculation), the expression is obtained through low-frequency indexing and then the final expression calculation is performed.

[0037] 3) Low-frequency index creation process

[0038] The mapping from keywords to rule trees is achieved by selecting low-frequency groups to build indexes, reducing the number of keyword and expression indexes, filtering redundant expression calculations, lowering the frequency of expression calculations, and improving computational performance.

[0039] Low-frequency index term calculation method: Group the expression by operation conditions, and obtain the optimal index term by dynamic hit rate, term frequency, keyword sensitivity, etc.

[0040] Dynamic hit rate and word frequency can be obtained through statistical analysis, while keyword sensitivity can be calculated using algorithms such as TF-IDF. The higher the score, the better the dynamic hit rate, the lower the word frequency, and the higher the keyword sensitivity.

[0041] 4) OR operation bitmap calculation process

[0042] Based on the original binary rule tree, merge or convert nodes into bitmap nodes. Generate a bitmap by using the set of keywords that the target text hits. Perform bitmap intersection calculation with the nodes of the rule to obtain the intersection bitmap of the keyword calculation result and the expression node. Determine whether the node has been hit by checking if the capacity of the intersection bitmap is greater than 0. This can reduce the traversal depth and the number of calculations, thereby improving performance.

[0043] Implementation

[0044] Please refer to Figures 1-3 The specific steps of the logical expression calculation method based on low-frequency indexes in this invention are as follows:

[0045] S1. Obtain the rule expression, for example: A like 'a' OR (B like 'b' AND C like 'c')

[0046] S2. Construct the original binary tree based on the regular expression.

[0047] S3. Perform a paradigm shift, converting the mixed AND / OR operation into multiple AND operations of OR.

[0048] For example: (A like 'a' OR B like 'b') AND (A like 'a' OR C like 'c')

[0049] S4. Calculate the score for each group based on dynamic hit rate, word frequency (obtained through long-term accumulation of common and uncommon words via the internet or programs), and keyword sensitivity. Lower dynamic hit rate, lower word frequency, and higher keyword sensitivity result in higher scores. Select the group with the highest score and create an index for that expression.

[0050] S5. Build a dictionary based on keywords (mapping between numbers and keywords).

[0051] S6. Convert the original binary tree leaf node dictionary, and merge multiple OR nodes into a single bitmap node.

[0052] S7. After globally deduplicating all grouped keywords, a matching machine is generated for use in multi-modal keyword matching.

[0053] S8. Input the target text, perform multi-modal matching, generate a list of keyword results, and produce a data bitmap.

[0054] S9. Indexing to a binary tree of regular expressions using result keywords.

[0055] S11. Perform operations on the entire binary tree and return the results.

[0056] The above solution is implemented in Java, and the specific steps are as follows:

[0057] 1. Rule compilation, defining the rule binary tree object.

[0058] 2. Create an index collection to store rule indexes.

[0059] 3. Construct a keyword dictionary mapping to provide a basis for bitmap conversion.

[0060] 4. Construct an automatic multi-mode matching machine

[0061] 5. Input the target text, perform multi-pattern matching calculation, obtain the matching results, and construct the result bitmap object.

[0062] 6. Based on the multi-pattern matching results, retrieve the specific rule binary tree object through the rule index.

[0063] 7. Based on the multi-mode matching results, perform bitmap AND operations on each leaf node of the rule binary tree to obtain the calculation result of the entire binary tree.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fast calculation of text logical expressions based on low-frequency indexing, characterized in that... Includes the following steps: Step S1, Expression Grouping and Low-Frequency Index Construction: Traverse all expressions and convert expressions with mixed OR and AND operations into AND expressions with multiple OR operation groups according to the operation conditions; Step S2, Low-frequency index construction: Calculate the score of each OR operation group by dynamic weighting, sort the scores according to the frequency of keywords in the expression, and use the OR operation group with the highest score as the index group and create an index with the expression. Step S3: After globally deduplicating the keywords in all groups, a matching machine is generated for use in multi-modal keyword matching. Step S4, OR operation bitmap calculation: Based on the original binary tree rules, merge OR nodes into bitmap nodes, match the target text with the matching machine obtained in step S3, generate a bitmap from the set of hit keywords, calculate the intersection of this bitmap with the index obtained in step S2 to obtain the intersection bitmap, and determine whether the node is hit by whether the capacity of the intersection bitmap is greater than 0.

2. The method for fast calculation of text logical expressions based on low-frequency indexing according to claim 1, characterized in that: In step S2, after the expressions are grouped, the dynamic hit rate, word frequency, and keyword sensitivity of each group are calculated to obtain the score of each group.

3. The method for fast calculation of text logical expressions based on low-frequency indexing according to claim 2, characterized in that: The dynamic hit rate and word frequency are obtained from the historical word database; the keyword sensitivity is calculated using the TF-IDF algorithm.

Images