Data compression method, device and equipment of regular expression HNFA model

Abstract

The application relates to a data compression method, device and equipment of a regular expression HNFA model. The method comprises the following steps: obtaining HNFA models corresponding to a plurality of regular expressions, extracting an effective character set group of each model, clustering the models based on the similarity between the effective character sets, forming a plurality of clustering groups, constructing a unified character mapping table for each group, mapping characters in the group to continuous indexes, compressing original character activation module data into a dense matrix according to the mapping table, converting input characters through the mapping table during matching, and enabling an engine to correctly read a state activation vector from the compressed data. The method significantly reduces storage overhead and transmission delay while maintaining hardware compatibility, and improves regular matching efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data compression method, apparatus and device for a regular expression HNFA model. Background Technology

[0002] Regular expressions, as a powerful pattern description tool, are widely used in critical areas such as network security (e.g., deep packet inspection, intrusion prevention systems), text search, and data processing. Their core function is to determine whether a predefined complex character pattern appears in the target text. Currently, efficient regular expression matching engines are typically implemented based on finite state automata, that is, compiling regular expressions into a state machine model and performing matching by simulating the state transition process.

[0003] Existing general-purpose hardware regular expression matching engines mainly consist of two modules: a character activation module (M×N storage units, where M=256 is the size of the ASCII character set, and N is the maximum number of supported states), which contains the state activation vector for each character in each line; and a state transition module (N×N storage units, defining the state transition relationships). During matching, the next active state set is obtained by performing a bitwise AND operation between the character activation set and the state transition set, adapting to regular HNFA models with a state number ≤ N.

[0004] However, this architecture suffers from significant storage redundancy: the actual regular expression only uses a few dozen valid characters, yet it requires generating a complete 256×N character activation module data, instead of storing only the n×N data corresponding to the valid characters. In large-scale scenarios with tens of thousands of rules, this redundancy leads to a surge in global storage overhead and hardware costs. Furthermore, the high bandwidth requirements, latency, and power consumption during data loading limit deployment on resource-constrained devices. Therefore, a hardware-compatible HNFA data compression method is urgently needed to significantly reduce storage and transmission overhead. Summary of the Invention

[0005] Therefore, it is necessary to provide a data compression method, apparatus, and device for regular expression HNFA models that can solve the problems of high storage overhead and high loading latency caused by data redundancy in existing general regular expression matching engines.

[0006] A data compression method for a regular expression HNFA model, the method comprising:

[0007] Obtain the HNFA models corresponding to the multiple regular expressions to be matched, group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same valid character set, and construct the corresponding valid character set group from all valid character sets of each HNFA model.

[0008] Based on the proportion of the same valid character set among the various HNFA models, the HNFA models corresponding to the multiple regular expressions are clustered to generate at least one cluster group.

[0009] Based on each cluster group, a corresponding character mapping table is constructed. Each HNFA model compresses the original data of the character activation module into a dense matrix of a preset format according to the character mapping table corresponding to its respective cluster group.

[0010] When matching a regular expression, the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs is used to map the input characters, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0011] In one embodiment, the clustering based on the proportion of the same effective character sets among the various HNFA models includes:

[0012] Calculate the proportion of the same effective character set between two HNFA models, and cluster the two HNFA models whose proportions meet the preset value so that the HNFA models with similar character features are grouped into the same cluster group.

[0013] When performing clustering: the total number of all valid character sets in the same cluster group does not exceed a preset upper limit, and the valid character sets in the same cluster group do not overlap with each other. If there is overlap between multiple valid character sets, the overlapping part is independently divided into a new valid character set.

[0014] In one embodiment, the character mapping table is a lookup table of size 256;

[0015] The lookup table maps characters within the same valid character set in the corresponding cluster group to the same index number, and characters from different valid character sets to different index numbers, so that all valid characters are mapped to consecutive index numbers from 0 to n-1, and invalid characters are mapped to a preset invalid identifier, where n is the upper limit of the total number of valid character sets in the cluster group.

[0016] In one embodiment, the dense matrix of the preset format is an n-row × N-bit matrix, where N is the maximum number of states supported by the general regular expression matching engine;

[0017] The i-th row of the dense matrix stores the state activation vectors corresponding to all characters in the original character activation module that are mapped to index number i.

[0018] In one embodiment, when mapping input characters:

[0019] The character mapping table is queried using the ASCII code of the input character as the address by a general regular expression matching engine.

[0020] If the query result is a valid index number, then the state activation vector of the corresponding row is read from the dense matrix;

[0021] If the query result is the invalid identifier, then the state activation vector of all zeros is obtained directly.

[0022] In one embodiment, the state activation vector stored in the i-th row of the dense matrix is ​​completely consistent with the state activation vectors corresponding to all characters mapped to index number i in the original character activation module.

[0023] In one embodiment, after the general regular expression matching engine reads the state activation vector:

[0024] By combining the state transition module of the general regular expression matching engine, the set of transitionable states is obtained through OR operation and the next active state set is obtained through AND operation to complete the regular expression matching process.

[0025] This application also provides a data compression device for a regular expression HNFA model, the device comprising:

[0026] The effective character set extraction module is used to obtain the HNFA models corresponding to the multiple regular expressions to be matched, and to group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same effective character set, and to construct the corresponding effective character set group from all the effective character sets of each HNFA model.

[0027] The clustering module is used to cluster the HNFA models corresponding to the multiple regular expressions according to the proportion of the same valid character sets among the HNFA models, and generate at least one cluster group.

[0028] The character mapping table construction and data compression module is used to construct the corresponding character mapping table according to each cluster group, and each HNFA model compresses the original data of the character activation module into a dense matrix in a preset format according to the character mapping table corresponding to its respective cluster group.

[0029] The matching processing module is used to map the input characters by using the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs when matching a regular expression, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0030] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the data compression method of the above-described regular expression HNFA model.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the data compression method of the above-described regular expression HNFA model.

[0032] The aforementioned data compression method, apparatus, and device for regular expression HNFA models acquire multiple HNFA models corresponding to regular expressions, extract the effective character set groups for each model, cluster the models based on the similarity between the effective character sets, forming several cluster groups, and construct a unified character mapping table for each group, mapping characters within the group to consecutive indices. Based on this mapping table, the original character activation module data is compressed into a dense matrix. During matching, the input characters are transformed through the mapping table, enabling the engine to correctly read the state activation vector from the compressed data. This method significantly reduces storage overhead and transmission latency while maintaining hardware compatibility, thus improving regular expression matching efficiency. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a data compression method for a regular expression HNFA model in one embodiment;

[0034] Figure 2 This is a schematic diagram of a general hardware regular expression matching engine structure in one embodiment;

[0035] Figure 3 This is a schematic diagram of the method based on a general hardware regular expression matching engine structure in one embodiment;

[0036] Figure 4 This is a structural block diagram of a data compression device for a regular expression HNFA model in one embodiment;

[0037] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] In this embodiment, as Figure 1As shown, a data compression method for regular expression HNFA models is provided, that is, a data compression method for HNFA models corresponding to regular expressions, where the model is the data compression object, and the method includes the following steps:

[0040] Step S100: Obtain the HNFA models corresponding to the multiple regular expressions to be matched, group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same valid character set, and construct the corresponding valid character set group from all valid character sets of each HNFA model.

[0041] Step S110: Based on the proportion of the same valid character set among the HNFA models, cluster the HNFA models corresponding to multiple regular expressions to generate at least one cluster group.

[0042] Step S120: Construct a corresponding character mapping table based on each cluster group. Each HNFA model compresses the original data of the character activation module into a dense matrix in a preset format based on the character mapping table corresponding to its cluster group.

[0043] Step S130: When matching a regular expression, the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs is used to map the input characters, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0044] Among numerous state machine models, the Homogeneous Non-deterministic Finite Automaton (HNFA) has attracted considerable attention due to its unique "homogeneous" property. This property means that all transition edges leading to the same state must correspond to the same set of input characters. Based on this, HNFA can regularly decompose the complex process of calculating the next active state into the intersection operation of two independent sets (the set of states activated by the current input character, and the set of states reachable from the current active state through transition edges). This regular parallelism makes the HNFA model extremely suitable for high-throughput matching implementation on hardware (such as FPGAs and ASICs).

[0045] Existing technologies have already designed general-purpose hardware regularization matching engines based on the HNFA model, with a typical structure as follows: Figure 2As shown, the engine mainly consists of two modules: a character activation module and a state transition model. The character activation module is typically implemented using an M × N storage unit (such as BRAM), where M is the character set size (256 for standard ASCII) and N is the maximum number of states supported by the engine (e.g., 32). Each row of this module corresponds to an input character, and its stored N-bit vector represents all states that the character can directly activate. The state transition module is typically implemented using an N × N storage unit, where each column defines all potential states that a given state, active in the previous cycle, can transition to in the current cycle.

[0046] During matching, for each input character, the engine reads the corresponding line from the character activation module to obtain the state set activated by that character. Simultaneously, it reads the column corresponding to the active state set of the previous cycle from the state transition module and performs an OR operation to obtain the transitionable state set. Finally, an AND operation is performed on the two results to obtain the next active state set for the current cycle. Any regular expression with no more than N states can have its HNFA model data configured to run in this engine.

[0047] However, this general-purpose engine architecture has a significant efficiency problem: severe waste of storage resources. Specifically, the character activation module allocates one row of storage space for the entire 256-character ASCII code table. But in practical applications, a specific regular expression often only uses a few dozen characters to activate the state. However, to adapt to the fixed hardware structure, even if the vast majority of characters do not activate any state, the HNFA data corresponding to the regular expression must still generate a complete 256-row × 32-bit character activation module data, instead of storing only the n × 32-bit data corresponding to the small number of characters (n) actually used.

[0048] At this point, when the system needs to process tens of thousands of regular expression rules (for example, the rule base of a modern intrusion detection system may contain tens of thousands of rules), the redundant storage of HNFA data for each rule will increase dramatically, leading to: First, the storage overhead of the global memory is huge, increasing the system hardware cost. Second, during the matching process, when the massive HNFA data needs to be loaded from the global memory to the matching engine, the transmission bandwidth requirement is high, resulting in long loading delays and high power consumption. This limits the feasibility of deploying large-scale rule bases on resource-constrained embedded or edge computing devices. Therefore, this application proposes a compression method for HNFA model data, the structure of which is as follows: Figure 3 As shown, it can significantly reduce its storage and transmission overhead while maintaining its regular hardware compatibility.

[0049] In step S100, as shown in Table 1, there are HNFA models for different regular expressions and their corresponding valid character sets.

[0050] Table 1. HNFA models for different regular expressions and their corresponding valid character sets.

[0051]

[0052] Specifically, taking the regular expression “(ab|cd)e” in Table 1 as an example, the character “a” can only activate state S0 to obtain the valid character set {a}; similarly, the characters “b”, “c”, “d”, and “e” can activate states S1, S2, S3, and S4 respectively to obtain the valid character sets {b}, {c}, {d}, and {e}, which ultimately form the valid character set group [{a}, {b}, {c}, {d}, {e}].

[0053] Furthermore, for the HNFA corresponding to the regular expression "[ab][cd][ae]", the character "a" can activate states S0 and S2 simultaneously, while "b" and "e" activate states S0 and S2 respectively. Therefore, "a", "b", and "e" each constitute the valid character sets {a}, {b}, and {e}, while "c" and "d" only activate state S1, resulting in the valid character set {c,d}. Finally, the character set group [{a},{b},{c,d},{e} is formed.

[0054] Similarly, the regular expression "123456789[az]" yields the character set [{1},{2},{3},{4},{5},{6},{7},{8},{9},{az}] based on the principle that characters with the same activation state are grouped into the same set.

[0055] In step S110, clustering based on the proportion of identical effective character sets among the HNFA models includes: calculating the proportion of identical effective character sets between two HNFA models, clustering two HNFA models whose proportions meet a preset value, so that HNFA models with similar character usage features are grouped into the same cluster group. During clustering: the total number of all effective character sets within the same cluster group does not exceed a preset upper limit, and the effective character sets within each cluster group do not overlap. If multiple effective character sets overlap, the overlapping portion is independently divided into a new effective character set.

[0056] In this embodiment, the clustering process divides all HNFA models into several clustering groups (24 in this embodiment) based on the proportion of the same effective character sets among different HNFA models. The total number of effective character sets in each clustering group does not exceed a preset upper limit n, where n is a positive integer less than 256 (n=64 in this embodiment). It also ensures that the character sets in each group do not overlap. If there is overlap between multiple effective character sets, the overlapping part is independently divided into a new effective character set.

[0057] Specifically, taking the three regular expressions in Table 1 as examples, group “(ab|cd)e” and “[ab][cd][ae]” together, merge the valid character sets, and independently divide the overlapping parts into a new valid character set, resulting in the grouped valid character set group [{a},{b},{c},{d},{e}]. If “123456789[az]” is then added to the group, the grouped valid character set is expanded to:

[0058] [{a},{b},{c},{d},{e},{fz},{1},{2},{3},{4},{5},{6},{7},{8},{9}]

[0059] Because the effective character set of "123456789[az]" differs greatly from the character sets of the other two regular expression groups, the number of character sets in the character set group obtained by clustering the three increases dramatically. This would be avoided as much as possible when there are tens of thousands of regular expressions in reality, but in this case there are only 3 regular expressions. At the same time, the number of character sets in the character set group obtained by clustering is 15, which is lower than the upper limit of 64 set in this example, so it can be supported by the current matching engine.

[0060] In step S120, the character mapping table is a lookup table of size 256. The lookup table maps characters within the same valid character set to the same index number for corresponding cluster groups, and maps characters from different valid character sets to different index numbers, so that all valid characters are mapped to consecutive index numbers from 0 to n-1, and invalid characters are mapped to a preset invalid identifier, where n is the upper limit of the total number of valid character sets in the cluster group.

[0061] In this embodiment, the character mapping table is a lookup table of size 256, which maps characters in the valid character set within a group to consecutive index numbers [0, n-1], preferably n is 64. Characters outside the group are mapped to a specific invalid identifier. An n-row × N-bit compression matrix, i.e., a dense matrix, is created for each HNFA model (N is the maximum number of states supported by the engine, preferably N=32). The i-th row of the compression matrix stores the state activation vectors corresponding to all characters mapped to index number i in the original character activation module, and the state activation vectors stored in the i-th row of the compression matrix are completely consistent with the state activation vectors corresponding to all characters mapped to index number i in the original character activation module.

[0062] Specifically, taking the clustering grouping composed of three regular expressions in Table 1 as an example, the mapping table maps "a" to index number 0, corresponding to... Figure 3 The first line of the character activation module. "b" maps to index number 1, corresponding to the second line of the module. Specifically, each character in the character set {fz} maps to index number 5, corresponding to the sixth line of the module. Based on this mapping, the character activation module data for the HNFA model of the three regular expressions can be compressed from 256×32bit to 64×32bit, reducing storage and transmission overhead to 1 / 4 of the original.

[0063] In step S130, when mapping the input characters: the ASCII code of the input characters is used as the address to query the character mapping table using a general regular expression matching engine. If the query result is a valid index number, the state activation vector of the corresponding row is read from the dense matrix. If the query result is an invalid identifier, the state activation vector of all zeros is obtained directly.

[0064] Furthermore, after the general regular expression matching engine reads the state activation vector, it combines the state transition module of the general regular expression matching engine to obtain the set of transitionable states through OR operation and obtain the next active state set through AND operation, so as to complete the regular expression matching process.

[0065] Specifically, when the input text needs to match a certain regular expression, the engine loads the corresponding compressed HNFA data and specifies a pre-stored character mapping table. After the mapping module converts the input characters into indices, the engine can complete the regular expression matching based on the compressed data. Since the storage overhead of the 24 mapping tables is small, they can be built into a fixed module without repeated loading, thereby further improving processing efficiency.

[0066] The data compression method for the HNFA regular expression model described above analyzes the similarity of character-state activation patterns in different HNFA models, clusters the models, and constructs a unified compact character mapping table for each group. This compresses the originally sparse and large-scale character activation matrix into a significantly smaller dense matrix. While maintaining compatibility with the hardware architecture of general regular expression matching engines, this method effectively reduces storage resource requirements and data loading latency, and improves matching processing efficiency.

[0067] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0068] In one embodiment, such as Figure 4 As shown, a data compression device for a regular expression HNFA model is provided, comprising: an effective character set extraction module 200, a clustering module 210, a character mapping table construction and data compression module 220, and a matching processing module 230, wherein:

[0069] The effective character set extraction module 200 is used to obtain the HNFA models corresponding to the multiple regular expressions to be matched, and to group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same effective character set, and to construct the corresponding effective character set group from all the effective character sets of each HNFA model.

[0070] Clustering module 210 is used to cluster the HNFA models corresponding to the multiple regular expressions according to the proportion of the same valid character sets among the HNFA models, and generate at least one cluster group.

[0071] The character mapping table construction and data compression module 220 is used to construct a corresponding character mapping table according to each cluster group, and each HNFA model compresses the original data of the character activation module into a dense matrix in a preset format according to the character mapping table corresponding to its cluster group.

[0072] The matching processing module 230 is used to map the input characters using the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs when matching the regular expression, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0073] Specific limitations regarding the data compression device for the HNFA regular expression model can be found in the limitations of the data compression method for the HNFA regular expression model above, and will not be repeated here. Each module in the aforementioned data compression device for the HNFA regular expression model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0074] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a data compression method based on a regular expression HNFA model. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0075] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0076] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0077] Obtain the HNFA models corresponding to the multiple regular expressions to be matched, group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same valid character set, and construct the corresponding valid character set group from all valid character sets of each HNFA model.

[0078] Based on the proportion of the same valid character set among the various HNFA models, the HNFA models corresponding to the multiple regular expressions are clustered to generate at least one cluster group.

[0079] Based on each cluster group, a corresponding character mapping table is constructed. Each HNFA model compresses the original data of the character activation module into a dense matrix of a preset format according to the character mapping table corresponding to its respective cluster group.

[0080] When matching a regular expression, the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs is used to map the input characters, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0081] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0082] Obtain the HNFA models corresponding to the multiple regular expressions to be matched, group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same valid character set, and construct the corresponding valid character set group from all valid character sets of each HNFA model.

[0083] Based on the proportion of the same valid character set among the various HNFA models, the HNFA models corresponding to the multiple regular expressions are clustered to generate at least one cluster group.

[0084] Based on each cluster group, a corresponding character mapping table is constructed. Each HNFA model compresses the original data of the character activation module into a dense matrix of a preset format according to the character mapping table corresponding to its respective cluster group.

[0085] When matching a regular expression, the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs is used to map the input characters, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

[0086] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data compression method for a regular expression HNFA model, characterized in that, The method includes: Obtain the HNFA models corresponding to the multiple regular expressions to be matched, group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same valid character set, and construct the corresponding valid character set group from all valid character sets of each HNFA model. Based on the proportion of the same valid character set among the various HNFA models, the HNFA models corresponding to the multiple regular expressions are clustered to generate at least one cluster group. Based on each cluster group, a corresponding character mapping table is constructed. Each HNFA model compresses the original data of the character activation module into a dense matrix in a preset format according to the character mapping table corresponding to its respective cluster group. When matching a regular expression, the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs is used to map the input characters, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

2. The data compression method for the regular expression HNFA model according to claim 1, characterized in that, The clustering based on the proportion of the same effective character set among the various HNFA models includes: Calculate the proportion of the same effective character set between two HNFA models, and cluster the two HNFA models whose proportions meet the preset value so that characters using similar features are grouped into the same cluster group; When performing clustering: the total number of all valid character sets in the same cluster group does not exceed a preset upper limit, and the valid character sets in the same cluster group do not overlap with each other. If there is overlap between multiple valid character sets, the overlapping part is independently divided into a new valid character set.

3. The data compression method for the regular expression HNFA model according to claim 2, characterized in that, The character mapping table is a lookup table of size 256; The lookup table maps characters within the same valid character set in the corresponding cluster group to the same index number, and characters from different valid character sets to different index numbers, so that all valid characters are mapped to consecutive index numbers from 0 to n-1, and invalid characters are mapped to a preset invalid identifier, where n is the upper limit of the total number of valid character sets in the cluster group.

4. The data compression method for the regular expression HNFA model according to claim 3, characterized in that, The dense matrix in the preset format is an n-row × N-bit matrix, where N is the maximum number of states supported by the general regular expression matching engine; The i-th row of the dense matrix stores the state activation vectors corresponding to all characters in the original character activation module that are mapped to index number i.

5. The data compression method for the regular expression HNFA model according to claim 3, characterized in that, When mapping input characters: The character mapping table is queried using the ASCII code of the input character as the address by a general regular expression matching engine. If the query result is a valid index number, then the state activation vector of the corresponding row is read from the dense matrix; If the query result is the invalid identifier, then the state activation vector of all zeros is obtained directly.

6. The data compression method for the regular expression HNFA model according to claim 4, characterized in that, The state activation vector stored in the i-th row of the dense matrix is ​​completely consistent with the state activation vectors corresponding to all characters mapped to index number i in the original character activation module.

7. The data compression method for the regular expression HNFA model according to claim 5, characterized in that, After the general regular expression matching engine reads the state activation vector: By combining the state transition module of the general regular expression matching engine, the set of transitionable states is obtained through OR operation and the next active state set is obtained through AND operation to complete the regular expression matching process.

8. A data compression device for a regular expression HNFA model, characterized in that, The device includes: The effective character set extraction module is used to obtain the HNFA models corresponding to the multiple regular expressions to be matched, and to group the input characters that can activate the same set of states in all transition edges of each HNFA model into the same effective character set, and to construct the corresponding effective character set group from all the effective character sets of each HNFA model. The clustering module is used to cluster the HNFA models corresponding to the multiple regular expressions according to the proportion of the same valid character sets among the HNFA models, and generate at least one cluster group. The character mapping table construction and data compression module is used to construct the corresponding character mapping table according to each cluster group, and each HNFA model compresses the original data of the character activation module into a dense matrix in a preset format according to the character mapping table corresponding to its respective cluster group. The matching processing module is used to map the input characters by using the character mapping table of the cluster group to which the HNFA model corresponding to the regular expression belongs when matching a regular expression, so that the general regular expression matching engine can read the correct state activation vector from the compressed character activation data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images