An illegal character automatic cleaning system

Abstract

The present application relates to computer data processing and network security technical field, specifically to a kind of illegal character automatic cleaning system, comprising: rule compiling module: monitoring rule changes, rule set is fused with topological mapping with semantics, constructs deterministic finite automaton and is mapped into state transition table;State switching module: constructs double-buffer context, realizes hot update by atomic pointer operation lockless switching;Speculative execution module: utilize single-instruction multiple-data stream parallel loading, vectorized pre-scanning is carried out based on state transition table, identifies wandering path, falls into safe state and releases, falls into trap state and triggers external check;Adaptive feedback module: the frequency of trap state triggering is counted, generates rule hypersensitivity report, dynamically adjusts reading fragment size;The present application solves the contradiction between rule flexibility and execution efficiency, realizes high-performance cleaning based on speculative execution.

Description

Technical Field

[0001] This invention relates to the fields of computer data processing and network security technology, specifically to an automatic illegal character cleaning system. Background Technology

[0002] In the current field of large-scale data processing and information security, real-time cleaning and filtering of illegal characters or sensitive information in network data streams is a core task. Network data streams can specifically be web page input streams, API call request streams, or instant messaging message streams. To ensure business compliance and system security, data processing systems typically need to load massive amounts of discrete cleaning rules, including complex sets of regular expressions and enumerated string lists. These rules are often frequently updated as business needs change, and in actual execution, they require scanning and matching high-speed data packets byte by byte, a process involving significant computational resource consumption and memory access operations. Traditional cleaning path designs often rely on interpreted rule matching engines. As the number of rules increases, the time complexity of the matching algorithm often grows linearly or even exponentially, making it difficult to maintain stable throughput in scenarios with low latency requirements.

[0003] While existing technologies offer solutions based on traditional finite automata or string matching algorithms, these methods typically require global locking mechanisms to ensure context consistency when faced with dynamic changes in the rule set, leading to service jitter or momentary pauses. Furthermore, existing matching engines struggle to fully utilize the vectorized instruction sets and pipelined features of modern processors, with frequent branch prediction failures and cross-cache line accesses severely limiting processing performance. Additionally, traditional methods lack adaptive feedback mechanisms based on runtime status to address real-time fluctuations in the density of dirty data in the data stream, making it impossible to dynamically adjust data reading granularity or verification strategies. This can easily trigger memory overflows when dirty data surges or cause idle and wasted computing resources when data is clean.

[0004] Therefore, how to provide an automatic illegal character cleaning system that can balance the flexibility of rule hot updates and execution efficiency, and has adaptive adjustment capabilities, is a problem that urgently needs to be solved by those skilled in the art.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide an automatic illegal character cleaning system that avoids the linear increase in processor computing load caused by the expansion of the security rule set and the service blocking caused by dynamic rule updates. Furthermore, it is easier to achieve high-performance data cleaning and closed-loop system control based on speculative execution mechanisms and adaptive feedback. Specifically, the technical solution of this invention is as follows:

[0007] The rule compilation module is used to listen for rule change events in the configuration center, obtain a discrete set of cleaning rules, perform semantic fusion and topological mapping on the set of cleaning rules, construct a deterministic finite automaton in memory, and map the deterministic finite automaton to a state transition table with continuous memory addresses to achieve flattening of the automaton graph structure.

[0008] The state switching module is used to build a double-buffered execution context. After the state transition table is built, the current execution context is switched to the new state transition table without lock through atomic pointer operations, so as to realize the hot update of the cleaning logic.

[0009] The speculative execution module is used to load the data stream to be cleaned in parallel using a single instruction multiple data stream instruction set, perform vectorized pre-scanning of the data stream based on the state transition table, identify the walking path of the data stream in the deterministic finite automaton, and directly allow the data stream when the walking path falls into a safe state, thereby constructing a computational firewall to intercept invalid external calls; when the walking path falls into a trap state, an external verification request is triggered to determine whether the data stream is dirty data.

[0010] The adaptive feedback module is used to statistically analyze the trigger frequency and accurate verification results of the trap state in real time, generate a rule-based sensitivity report, and dynamically adjust the read fragment size of the data stream based on the rule-based sensitivity report to complete the closed-loop control of the cleaning system.

[0011] Preferably, the method for semantic fusion and topological mapping of the cleaning rule set includes:

[0012] Each regular expression rule and enumerated rule in the cleansing rule set is parsed to generate the corresponding abstract syntax tree node;

[0013] Merge all abstract syntax tree nodes into a unified global syntax tree;

[0014] Perform node splitting and state merging on the global syntax tree to generate a nondeterministic finite automaton;

[0015] A subset construction algorithm is used to convert a nondeterministic finite automaton into a deterministic finite automaton.

[0016] The deterministic finite automata are minimized to remove redundant state nodes, and semantic fusion and topological mapping are completed.

[0017] Preferably, the method for mapping a deterministic finite automaton to a state transition table with contiguous memory addresses includes:

[0018] Traverse all state nodes of the minimized deterministic finite automaton and assign a unique memory index offset to each state node.

[0019] Extract the jump target index of each state node under different input characters, and construct a two-dimensional jump matrix;

[0020] The two-dimensional jump matrix is ​​flattened into a one-dimensional linear array, and the linear array is aligned to the processor's cache line size to generate a state transition table.

[0021] Preferably, the method for vectorized pre-scanning of the data stream based on the state transition table includes:

[0022] Extract fixed-length byte segments from the data stream and load them into a vector register;

[0023] The character category index corresponding to each byte in a byte segment is computed in parallel using vectorized instructions. Specifically, this includes: using a high-low bit segmentation mapping method to decompose the original byte vector into high- and low-bit index vectors, and using vectorized table lookup instructions to map them into character category indexes in parallel.

[0024] Based on the current state index and the character category index, a direct address lookup is performed in the state transition table to obtain the next state index;

[0025] The table lookup operation is performed repeatedly until the byte segment traversal is completed or the process enters a termination state.

[0026] Paths that have not entered the termination state are marked as safe, and paths that have entered the termination state and are marked as potentially illegal characters are marked as traps.

[0027] Preferably, the method for triggering an external verification request when the traversal path falls into a trap state includes:

[0028] Lock the data segment that has fallen into the trap state and pause the vectorized pre-scan of the current thread;

[0029] Construct a standard verification request containing data fragments and send it to a predefined external standard verification interface;

[0030] Receive the Boolean check value returned by the external standard check interface;

[0031] If the Boolean check value is true, the data segment is determined to be dirty data, and a cleaning and removal operation is performed.

[0032] If the Boolean check value is false, the data segment is determined to be false alarm data, the data segment is retained, and the false alarm event is recorded.

[0033] Preferred methods for generating rule-based allergy reports include:

[0034] Within a preset time window, the total number of all data segments that fall into the trap state is accumulated and marked as the total amount of suspected dirty data;

[0035] The total number of false alarms with all Boolean check values ​​of false is recorded as the total number of false alarms.

[0036] Calculate the ratio of the total number of false alarms to the total number of suspected dirty data, and use it as the sensitivity coefficient of the current rule set;

[0037] The ratio of the total amount of suspected dirty data to the total number of bytes of the data stream scanned within the preset time window is used as the spatial distribution density of suspected dirty data.

[0038] By correlating the allergy coefficient with the distribution density of suspected dirty data, a rule-based allergy report is generated.

[0039] Preferred methods for dynamically adjusting the read fragment size of the data stream based on rule-based allergy reports include:

[0040] Get the preset step size of the current data stream read cursor;

[0041] Extract the distribution density of suspected dirty data from the rule-based allergy report;

[0042] A preset density threshold range includes a high density threshold and a low density threshold, wherein the high density threshold is greater than the low density threshold;

[0043] If the distribution density of suspected dirty data is greater than the high density threshold, the preset step size will be reduced by a preset decay ratio to reduce the concurrent pressure of external verification requests.

[0044] If the distribution density of suspected dirty data is less than the low density threshold, the preset step size is increased by a preset growth ratio to improve the throughput of vectorized pre-scan.

[0045] If the distribution density of suspected dirty data is greater than or equal to the low density threshold and less than or equal to the high density threshold, then the preset step size remains unchanged.

[0046] Preferably, the method for the state switching module to perform lock-free switching includes:

[0047] Pre-allocate a primary buffer and a secondary buffer in memory;

[0048] Store the current state transition table in the main buffer and write the newly constructed state transition table into the secondary buffer;

[0049] Set the global atomic pointer to point to the main buffer;

[0050] After the secondary buffer is written, an atomic comparison and swap operation is performed to change the address pointed to by the global atomic pointer from the primary buffer to the secondary buffer;

[0051] After confirming that all existing read threads have exited the main buffer, release the memory space of the main buffer.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. This invention uses a rule compilation module to perform semantic fusion and topological mapping on discrete sets of cleaning rules, constructing a memory-state minimized deterministic finite automaton, and flattening the graph structure into a state transition table with continuous memory addresses; achieving the effect of reducing the complex regular expression matching problem to a constant-level table lookup problem; compared with the problem of the computational load of the interpreted engine in the prior art increasing linearly with the number of rules, this invention completely eliminates the negative impact of the increase in the number of rules on performance, and resolves the contradiction between rule flexibility and execution efficiency.

[0054] 2. This invention utilizes a speculative execution module to load data in parallel using single-instruction multiple-data stream technology, and employs a high-low bit segmentation mapping method and a vector scalar bridging mechanism for vectorized pre-scanning. This achieves the effect of replacing conditional branch jumps with direct address calculation, making the scan throughput close to the physical limit of memory bandwidth. Compared with the complex regular expression matching method performed byte by byte in the prior art, this invention constructs a computational firewall that blocks the vast majority of invalid external calls, effectively avoiding the performance cost caused by branch prediction failure and reducing input / output latency.

[0055] 3. This invention constructs a double-buffered execution context through a state switching module, utilizes the processor's atomic pointer operations to achieve lock-free switching between the old and new state transition tables, and combines a periodic memory reclamation mechanism to ensure the safe release of resources; it achieves the effect of completing the hot update of the cleaning logic within milliseconds without blocking worker threads; compared with the frequent global lock contention during rule updates in the prior art, this invention effectively eliminates the performance jitter caused by context switching and solves the service blocking problem caused by dynamic rule changes in high-concurrency scenarios.

[0056] 4. This invention uses an adaptive feedback module to statistically analyze the trap state trigger frequency and accurate verification results in real time, generating a rule-based allergy report containing allergy coefficients and distribution density. Based on this, it dynamically adjusts the size of the data stream read shards, achieving automatic traffic shaping and load balancing based on data distribution characteristics. Compared to the fixed-step processing method in existing technologies, this invention can automatically shrink shards to prevent memory overflow when dirty data is concentrated, and expand shards to improve throughput when data is clean, thus solving the system stability risk caused by uneven data distribution. Attached Figure Description

[0057] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0058] Figure 1 This is a system structure diagram of the present invention;

[0059] Figure 2 This is a flowchart illustrating how the adaptive feedback module dynamically adjusts the size of the data stream read fragments. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0061] Example 1:

[0062] Please see Figure 1 An automatic illegal character cleaning system includes:

[0063] The rule compilation module is used to listen for rule change events in the configuration center, obtain a discrete set of cleaning rules, perform semantic fusion and topological mapping on the set of cleaning rules, construct a deterministic finite automaton in memory, and map the deterministic finite automaton to a state transition table with continuous memory addresses to achieve flattening of the automaton graph structure.

[0064] The state switching module is used to build a double-buffered execution context. After the state transition table is built, the current execution context is switched to the new state transition table without lock through atomic pointer operations, so as to realize the hot update of the cleaning logic.

[0065] The speculative execution module loads the data stream to be cleaned in parallel using a single-instruction, multi-data-stream instruction set. It performs a vectorized pre-scan of the data stream based on a state transition table, identifying the data stream's path within a deterministic finite automaton. When the path falls into a safe state, it allows the data to pass directly, thus constructing a computational firewall to intercept invalid external calls. When the path falls into a trap state, it triggers an external verification request to determine whether the data stream is dirty data. The data stream refers to the raw byte sequence transmitted in the network protocol stack; in this embodiment, it specifically represents form submission data from a website or the JSON request body of an API interface. The discrete cleansing rule set refers to a list of sensitive keywords and regular expression patterns preset for specific security vulnerabilities, such as SQL injection and cross-site scripting attacks.

[0066] The adaptive feedback module is used to statistically analyze the trigger frequency and accurate verification results of the trap state in real time, generate a rule-based sensitivity report, and dynamically adjust the read fragment size of the data stream based on the rule-based sensitivity report to complete the closed-loop control of the cleaning system.

[0067] This embodiment details the overall architecture and operating mechanism of the illegal character automatic cleaning system. The system aims to resolve the fundamental contradiction between rule flexibility and execution efficiency in existing technologies by converting interpreted rules into machine code-level automata to achieve high-performance cleaning based on speculative execution. The rule compilation module acts as the system's just-in-time compiler, which aims to transform human-readable configurations into a topology structure that is efficiently executed by the machine. This module monitors rule change events in the configuration center in real time. Once it obtains the discrete set of cleaning rules, namely the list of regular expressions and the list of enumerated strings, it does not directly store these rules but instead starts the compilation process.

[0068] This module performs semantic fusion on the set of cleaning rules, treating multiple rules as a whole logic and performing topological mapping to transform the logic into a graph structure, thus constructing a memory-state deterministic finite automaton. Based on this, the module flattens the graph structure of the deterministic finite automaton, mapping it to a state transition table of a contiguous memory address. This table refers to a linearized representation of a two-dimensional data structure, where rows represent the current state, columns represent input characters, and cells store the index of the next state.

[0069] The state switching module is used to implement seamless hot updates of the cleaning logic. The system maintains two buffer pointers. When the rule compilation module generates a new state transition table in the background, it writes it into the secondary buffer. After the construction is completed, the context pointer of the current working thread is switched to the new state transition table without lock through the atomic pointer operation of the processor.

[0070] The speculative execution module, acting as the system's execution engine, utilizes single-instruction multiple-data-stream technology to load the data stream to be cleaned in parallel, for example, loading 64 bytes at a time. It performs vectorized pre-scanning of the data stream based on a state transition table. The system does not perform complex regular expression matching on each byte; instead, it only identifies the data stream's path within a deterministic finite automaton. If the path falls into a safe state, the system directly allows the data to pass without further processing. If the path falls into a trap state, indicating a termination node where a potentially illegal character has been successfully matched, the system determines that the data is high-risk and triggers an external verification request to obtain accurate verification results. Illegal characters refer to specific characters or character sequences in the data stream that do not meet preset security and compliance requirements, such as single quotes "'" or "--" in SQL injection filtering scenarios. Dirty data refers to illegal character fragments that have been further confirmed by the external verification interface as attack payloads or violation information. For example, if the sequence "admin'--" appears in the data stream, and the automaton identifies it as an illegal character and it falls into a trap state, if the external verification determines that it has injection risk, it will be marked as dirty data and removed.

[0071] The adaptive feedback module is used for closed-loop control of the system. It performs real-time statistics on the trigger frequency and accurate verification results of trap states, calculates the false alarm rate and hit rate of rules, and generates a rule sensitivity report. Based on this report, the module dynamically adjusts the size of the data stream read fragments. For example, when dirty data is dense, it automatically shrinks the fragments to reduce memory pressure, and when the data is clean, it expands the fragments to improve throughput.

[0072] This embodiment uses the rule compilation module to... The complex matching problem is reduced to dimensionality The table lookup problem, among which For data volume, The rule count is reduced, completely eliminating the negative impact of increasing rule count on performance; the state switching module uses atomic operations to eliminate global lock contention, achieving millisecond-level hot rule activation; the speculative execution module constructs a computational firewall through single instruction multiple data stream technology and deterministic finite automaton pre-scanning, blocking the vast majority of invalid external calls and reducing input / output latency; the adaptive feedback module ensures load balancing of the system under different data distributions, preventing the risk of memory overflow caused by a surge in dirty data.

[0073] Example 2:

[0074] Methods for semantic fusion and topological mapping of cleaning rule sets include:

[0075] Each regular expression rule and enumerated rule in the cleansing rule set is parsed to generate the corresponding abstract syntax tree node;

[0076] Merge all abstract syntax tree nodes into a unified global syntax tree;

[0077] Perform node splitting and state merging on the global syntax tree to generate a nondeterministic finite automaton;

[0078] A subset construction algorithm is used to convert a nondeterministic finite automaton into a deterministic finite automaton.

[0079] The deterministic finite automata are minimized to remove redundant state nodes, and semantic fusion and topological mapping are completed.

[0080] This embodiment details the specific logic of semantic fusion and topological mapping of the cleansing rule set; the system executes the abstract syntax tree construction step, parses each regular rule and enumeration rule in the cleansing rule set, and uses the lexical analyzer to generate the corresponding abstract syntax tree node; a virtual root node is introduced, and all independent abstract syntax tree nodes are connected by OR logic and merged into a unified global syntax tree;

[0081] Based on this, the global syntax tree is traversed. For join, selection, and closure operations, Thompson's method is applied to perform node splitting and state merging. Specifically, for join operations, the terminating state of the predecessor node is processed by... Transform the initial state connected to the successor node; for selection operations, construct new start and end nodes, and branch them. Transform the connection between subexpressions; for closure operations, introduce a callback from the terminated state to the starting state. Transformation and direct crossing of subexpressions Transformation to fully express regular expression logic;

[0082] This process generates a nondeterministic finite automaton that allows null character jumps and multiple path branches; a subset construction algorithm is used to eliminate the nondeterminism in the nondeterministic finite automaton.

[0083] The system calculates the closure of the initial state of the nondeterministic finite automaton and uses it as the initial state of the deterministic finite automaton. By tracking the state transitions under different inputs, the deterministic finite automaton is generated. At this point, for any given input character, the automaton has only one jump path.

[0084] The Hopcroft algorithm is applied to minimize the deterministic finite automaton. The system merges all equivalent state nodes, that is, nodes that produce the same output behavior under the same input, and eliminates unreachable redundant state nodes. The final semantic fusion and topological mapping are completed, generating a graph structure with the fewest nodes.

[0085] This embodiment achieves the physical fusion of multiple rules by constructing a global abstract syntax tree and transforming it into a minimal deterministic finite automaton. This design ensures that the final generated automaton structure is logically equivalent regardless of whether the user configures a single rule or thousands of rules, and the number of nodes is compressed to the theoretical minimum. This allows subsequent table lookup operations to complete the parallel matching of all rules with only one memory access, greatly improving the processor's instruction cache hit rate and effectively solving the performance degradation problem under large-scale rule sets.

[0086] Example 3:

[0087] The rule compilation module is configured to perform the following operations to map a deterministic finite automaton to a state transition table with contiguous memory addresses: traverse all state nodes of the minimized deterministic finite automaton and assign a unique memory index offset to each state node;

[0088] Extract the jump target index of each state node under different input characters, and construct a two-dimensional jump matrix;

[0089] The two-dimensional jump matrix is ​​flattened into a one-dimensional linear array, and the linear array is aligned to the processor's cache line size to generate a state transition table.

[0090] This embodiment details a method for mapping a deterministic finite automaton to a state transition table with contiguous memory addresses. This method aims to optimize memory access patterns. It involves performing an index allocation step, traversing all state nodes of the minimized deterministic finite automaton, and assigning a unique memory index offset, starting from zero, to each state node in a breadth-first search order. ;

[0091] In the matrix construction step, to further compress the state table size and adapt to the vectorized lookup mechanism of Example 4, the system performs input alphabet reduction; it traverses all state nodes, identifies the set of input characters with the same transition behavior in all states, and merges them into equivalent character classes; the system generates a 256->ClassID mapping table, where 256->ClassID represents a mapping relationship, that is, mapping the full 8-bit bytes to the equivalent character class identifier ClassID in a smaller space; this step realizes the reduction of the input alphabet, which is the key to subsequently flattening the graph structure into a compact state transition table; for input characters , Its lower four bits are defined as the lookup index. Its high four bits are defined as the lookup index. ,in, The input character's ASCII or raw byte value; This is the bitwise AND operator, used to extract bit information; This is the right shift operator, used for bit shift operations; The system performs a high-order lookup index for the input byte. To ensure the generated mapping table can adapt to the slice lookup logic in the vectorized instructions, the system performs mapping table decomposition at this stage. The 256-item mapping table is decomposed into two 16-byte lookup tables using a backtracking search algorithm. and To satisfy the following bitwise operation constraints:

[0092]

[0093] in, For the input character's ASCII or raw byte value, This is a function that maps raw bytes to character classes. This is the bitwise OR operator used for merging results; the formula defines how to convert raw bytes to equivalent character classes using high-low bit fragmentation indexes. Fast parallel mapping;

[0094] Specifically, the backtracking search algorithm is defined as a depth-first recursive process with a definite termination condition: the algorithm maintains two lookup tables to be filled. and For each equivalent character class Each binary bit ( The following logic is executed:

[0095] like The If the bit is 0, then a mandatory constraint is applied. and The All bits are 0;

[0096] like The If the bit is 1, then a branch point is established;

[0097] Save current State snapshot;

[0098] Prioritize trying to The Set the bit to 1 and recursively call the next search function;

[0099] If the recursive call fails, a backtracking operation is performed. The The bits are restored to the snapshot state, that is, reset to 0;

[0100] Then try to The Set the bit to 1 and call it recursively again;

[0101] If both attempts fail, the current branch is returned as unsolvable.

[0102] If, during the backtracking process, it is found that the ID allocation of the current equivalent character class causes a bit constraint conflict, that is, the bit setting required by the current character is different from the previously determined bit setting... or If values ​​are mutually exclusive, the system will execute a conflict resolution strategy: locate the specific character causing the bit logic conflict, remove it from the current equivalent character class, and assign it a new, globally unique character class identifier. That is, execution operate;

[0103] The specific logic of this operation is as follows: The system first calculates the current conflicting character in... and The corresponding index positions in the table. If the bit values ​​of the table entries pointed to by these two indexes are already locked by other character constraints, and the locked values ​​do not match the bit characteristics required by the current character, then a conflict is determined. At this time, the system uses the remaining unallocated space... Search the space to find a new one ;filter The condition is: the lower 4 bits of the index derived from the new ID. With the high 4-bit index The entry it points to and It is not set with a hard bit constraint that is mutually exclusive with the current character.

[0104] Once a non-conflict condition is found... If the target value of the character is not found in the current conflicting character, it will be assigned to it, thus ensuring that the new assignment does not violate existing bit constraints. This operation changes the target value of the character in the mapping table, allowing the algorithm to find new target values ​​for the character that do not conflict with existing bit constraints when re-executing the decomposition process. and combination;

[0105] The system re-executes the decomposition process based on the updated character classification mapping table; to eliminate the risk of infinite loops, a maximum retry threshold is set for this conflict resolution loop. For example, 50 times; if the number of re-decompositions exceeds The system will terminate compression optimization and automatically downgrade to using a standard full mapping table to ensure that the algorithm will inevitably stop and output a usable state table within a finite number of steps; extract the jump target index of each state node under each equivalent character class; and construct a... A two-dimensional jump matrix, in which The total number of states. The number of equivalent character classes to be determined in the final calculation;

[0106] The linearization and alignment steps are performed to flatten the two-dimensional jump matrix into a one-dimensional linear array in row-major order. During this process, to ensure the atomicity and efficiency of state reads, the system performs row span alignment calculations on each row of the matrix, i.e., all jump information for each state. This involves padding the end of each row with invalid bytes, and setting the processor's cache line size to [value missing]. Typically 64 bytes, to ensure that memory allocation completely covers each row of the state transition matrix and to enforce cache row alignment to eliminate cross-row access overhead, a rounding up operator is introduced here. Step length The calculation formula is:

[0107]

[0108] This formula ensures that the offset of the starting address of each row in the two-dimensional matrix relative to the base address is constant. Integer multiples of;

[0109] in, Indicates the byte width of the status index type. This is the processor's cache line size, typically 64 bytes; this rounding-up logic ensures that the starting address of each line falls within the specified range. Offset to an integer multiple of the specified value;

[0110] This design ensures that in any state... All jump instructions are strictly contained within one or several consecutive cache lines, avoiding latency penalties caused by cross-cache line accesses. This ensures that memory read operations do not generate additional bus transactions due to crossing cache line boundaries, maximizing memory bandwidth utilization at the hardware level.

[0111] Example 4:

[0112] The speculative execution module is configured to perform a vectorized pre-scan of the data stream based on the state transition table:

[0113] Extract fixed-length byte segments from the data stream and load them into a vector register;

[0114] The character category index corresponding to each byte in a byte segment is computed in parallel using vectorized instructions. Specifically, this includes: using a high-low bit segmentation mapping method to decompose the original byte vector into high- and low-bit index vectors, and using vectorized table lookup instructions to map them into character category indexes in parallel.

[0115] Based on the current state index and the character category index, a direct address lookup is performed in the state transition table to obtain the next state index;

[0116] The table lookup operation is performed repeatedly until the byte segment traversal is completed or the process enters a termination state.

[0117] Paths that have not entered the termination state are marked as safe, and paths that have entered the termination state and are marked as potentially illegal characters are marked as traps.

[0118] This embodiment details a method for vectorized pre-scanning of a data stream based on a state transition table. This method utilizes the processor's vector register for parallel processing. The data loading step involves extracting a fixed-length byte segment, such as 64 bytes, from the data stream to be cleaned and loading it into the vector register in one go using vector loading instructions. Parallel index calculation is then performed, using vectorized instructions to calculate the character category index corresponding to each byte in the byte segment in parallel. Specifically, this includes using a high-low bit segmentation mapping method to decompose the original byte vector into high and low bit index vectors, and using vectorized table lookup instructions to map them in parallel to character category indexes, thus mapping the original bytes to input column indices that can be recognized by a deterministic finite automaton.

[0119] To address the issue that a single SIMD out-of-order lookup instruction typically only supports a finite-length lookup table and cannot directly cover 256 ASCII characters, this embodiment employs a high-low bit segmentation mapping method; the specific vectorized operation logic is as follows:

[0120] Load the pre-computed low-order table With high-level table To the vector register; based on the raw byte vector loaded from the data stream to be cleaned. Calculate the low-order index vector With high-order index vector ; Execute parallel lookup functions, such as the VPSHUFB instruction under the x86 architecture or the TBL instruction under the ARM architecture, which are general-purpose vector byte rearrangement instructions used to achieve parallel mapping from raw bytes to character classes through vectorized table lookup; that is, using the bytes in the vector as indices to retrieve the corresponding values ​​in the lookup table and outputting them to the target vector, resulting in:

[0121] and

[0122] Execute bit merge instruction This allows the conversion from raw bytes to character category indexes to be completed in approximately 5 instruction cycles;

[0123] The entire process is completed in the register file without accessing the L1 cache. The original ASCII / UTF-8 byte sequence is directly converted into the corresponding character category ID sequence in the vector register, avoiding byte-by-byte branch judgment.

[0124] In response to the situation where the mapping table decomposition fails in Example 3 and the full mapping table is used instead, this example also presets a compatible calculation logic: if the mapping table is detected to be in an uncompressed full table format, the system will skip the above high and low bit sharding logic and instead use the vector collection instruction or the full permutation instruction to directly index the full mapping table based on the original bytes.

[0125] If the hardware instruction set does not support this type of wide bit-width lookup, it will automatically switch to scalar table lookup mode to complete the character classification at this stage, ensuring that the technical solution is feasible under boundary conditions.

[0126] To perform direct address lookup and retain the advantages of SIMD classification while maintaining compatibility with the state dependencies of DFA, the system executes a vector-scalar bridging operation: the calculated values ​​in the registers are then used to perform the lookup. The vector is explicitly stored in a 64-byte aligned array `ClassBuffer` on the thread stack; the system enters a scalar loop context and iterates through the array; based on the currently held state index and the character class index read from the array, a direct address lookup is performed in the flattened state transition table, calculated as follows:

[0127]

[0128] in, The memory address for the next state; This is the base address of the state transition table; Index of the current state; The alignment step size for a single state line; This is the column offset corresponding to the character category, i.e. This is used to determine the memory offset of a specific character category within the state line; based on this, the above table lookup operation is performed repeatedly until the byte fragment traversal ends or the state machine enters the terminating state; in response to the end of traversal and not entering the terminating state, the path is marked as a safe state; in response to entering the terminating state and the state being marked as a potentially illegal character, the path is marked as a trap state.

[0129] This embodiment achieves single instruction multiple data processing through vectorized instructions. Although the state transition is essentially serial, the data loading and character classification are parallelized. More importantly, by replacing conditional branch jumps with direct address calculation, the cost of branch prediction failure is eliminated, making the scan throughput close to the physical limit of memory bandwidth, and exhibiting extremely high computational density when processing massive data streams.

[0130] Example 5:

[0131] The speculative execution module is configured to perform the following operation when the traversal path falls into a trap state:

[0132] Lock the data segment that has fallen into the trap state and pause the vectorized pre-scan of the current thread;

[0133] Construct a standard verification request containing data fragments and send it to a predefined external standard verification interface;

[0134] Receive the Boolean check value returned by the external standard check interface;

[0135] If the Boolean check value is true, the data segment is determined to be dirty data, and a cleaning and removal operation is performed.

[0136] If the Boolean check value is false, the data segment is determined to be false alarm data, the data segment is retained, and the false alarm event is recorded.

[0137] This embodiment details the method for triggering an external verification request when the traversal path falls into a trap state. This process constitutes the system's slow, precise path. When the automaton engine detects a trap state, it immediately locks the data segment currently trapped. Specifically, the locking operation involves increasing the reference count of this memory segment in the circular buffer to prevent the data generation thread from overwriting unverified data. The current thread's vectorized pre-scan task is paused, the data segment is extracted, and a standard verification request is constructed. Serialized into a structure containing metadata:

[0138]

[0139] in, This serves as a globally unique tracking identifier for this cleaning task. This is the Base64 encoded data segment to be verified. This is the byte offset of the segment in the original data stream. To trigger this trap state, the state machine context metadata is sent to a pre-defined external standard verification interface.

[0140] To prevent deadlock of the scanning thread due to delayed external interface responses, the system sets a timeout threshold. For example, 200ms, the system maintains a fixed length of A circular sampling buffer is used to record the most recent data in real time. Response time of the second external verification request Whenever a new verification response is received, the buffer is updated and the statistical indicators are recalculated. The specific threshold is set based on statistical analysis of the data in the buffer, and the specific calculation formula is as follows:

[0141]

[0142] in, This represents the 99th percentile response latency of the interface under peak load. Indicates a fixed length of The circular sampling buffer, The standard deviation of network transmission delay is used to ensure the system's robustness under network fluctuations; if in If no verification result response is received within the time limit, a degradation strategy is triggered. The system returns true (high security mode) or false (high availability mode) according to the preset security level. If the final Boolean verification value is true, the system determines that the data segment is definitely dirty data and performs cleaning and removal operations. If the Boolean verification value is false, the system determines that the data segment is false alarm data, retains the data segment, and records a false alarm event.

[0143] This embodiment addresses the uncertainty risks brought about by external coupling through a clear timeout and degradation mechanism, ensuring the real-time constraints of the cleaning system.

[0144] Example 6:

[0145] The adaptive feedback module is configured to generate rule-based allergy reports, and the specific operations include:

[0146] Within a preset time window, the total number of all data segments that fall into the trap state is accumulated and marked as the total amount of suspected dirty data;

[0147] The total number of false alarms with all Boolean check values ​​of false is recorded as the total number of false alarms.

[0148] Calculate the ratio of the total number of false alarms to the total number of suspected dirty data, and use it as the sensitivity coefficient of the current rule set;

[0149] The ratio of the total amount of suspected dirty data to the total number of bytes of the data stream scanned within the preset time window is used as the spatial distribution density of suspected dirty data.

[0150] By correlating the allergy coefficient with the distribution density of suspected dirty data, a rule-based allergy report is generated.

[0151] This embodiment details the method for generating rule-based allergy reports. To quantify the quality of the rules, a calculation model for relevant indicators is defined; a time window is set. The system monitors relevant parameters in real time within the window; it accumulates the total number of data segments that fall into the trap state to obtain the total amount of suspected dirty data. Symbols are used here. To distinguish it from the number of rules defined in Example 1 The total number of false alarms is obtained by summing the number of all events that return false results through external validation. ;

[0152] Calculate the allergy coefficient based on the above parameters. Before performing calculations, the system will first verify the total amount of suspected dirty data. Is it zero? If it is zero, then the allergy coefficient will be directly set. Set to 0 to reflect the absence of allergy risk and avoid floating-point division-by-zero anomalies; if If the value is greater than zero, the following calculation formula will be applied:

[0153]

[0154] in, : This represents the sensitivity or false alarm rate of the current rule set. The higher the value, the lower the accuracy of the rule, meaning that the rule tends to incorrectly identify normal data as trap states rather than being strict. Total number of false alarms Total amount of suspected dirty data;

[0155] Simultaneously, the ratio of the total amount of suspected dirty data to the total number of bytes in the data stream scanned within the preset time window is calculated as the spatial distribution density of suspected dirty data; the formula is as follows:

[0156]

[0157] in, This indicates the density of suspected dirty data per unit volume. This indicates the estimated total number of bytes of data stream actually scanned and processed by the execution module within a preset time window;

[0158] The system will and Perform correlation to generate a rule sensitivity report; this report reflects not only the current level of data corruption but also the accuracy of the rule itself.

[0159] This embodiment introduces allergy coefficient and distribution density, and the system quantifies the rule quality and data characteristics digitally. This provides a precise mathematical basis for subsequent adaptive adjustments, avoids performance fluctuations caused by blindly adjusting parameters, and enables the system to quantitatively perceive the characteristics of the current operating environment.

[0160] Example 7:

[0161] Please see the appendix Figure 2 The adaptive feedback module is configured to dynamically adjust the read fragment size of the data stream based on the rule-based allergy report. Specific operations include:

[0162] Get the preset step size of the current data stream read cursor;

[0163] Extract the distribution density of suspected dirty data from the rule-based allergy report;

[0164] A preset density threshold range, including a high density threshold and a low density threshold, wherein the high density threshold is greater than the low density threshold;

[0165] If the distribution density of suspected dirty data is greater than the high density threshold, the preset step size will be reduced by a preset decay ratio to reduce the concurrent pressure of external verification requests.

[0166] If the distribution density of suspected dirty data is less than the low density threshold, the preset step size is increased by a preset growth ratio to improve the throughput of vectorized pre-scan.

[0167] If the distribution density of suspected dirty data is greater than or equal to the low density threshold and less than or equal to the high density threshold, then the preset step size remains unchanged.

[0168] This embodiment details a method for dynamically adjusting the read fragment size of a data stream based on rule-based allergy reports. This method optimizes system throughput through a negative feedback control loop; the read fragment step size of the current data stream read cursor is set to... And preset density threshold range ,in, For high density threshold, Low density threshold;

[0169] High density threshold The value retrieval logic is based on the maximum processing capacity of the external validation interface. With the system's theoretical maximum throughput The ratio is calculated using the following formula:

[0170]

[0171] This ensures that it has a spatial distribution density similar to that of suspected dirty data. Same dimensions, where The safety margin factor is set to a value of [value to be filled in]. to This range is based on empirical values ​​derived from high-load stress tests of the system, aiming to balance system throughput with the risk of external interface queue overflow. By setting this threshold, the system unifies the abstract dirty data density with the specific physical processing capacity of the external interface, ensuring the physical meaning of flow control.

[0172] In this computational model, the system pre-defines each data fragment that falls into a trap state and triggers external verification as a unit of suspected dirty data; each fragment is equivalent to generating one discrete external interface query request; based on this weighted mapping relationship, the distribution density of suspected dirty data, measured in units per second, is determined. With interface processing capacity measured in queries per second The numerical comparisons demonstrate a clear consistency in physical meaning, thus ensuring the accuracy of the subsequent adaptive adjustment logic.

[0173] Low density threshold Set as To mitigate system oscillations at critical points, a hysteresis comparison range is constructed, ranging from 10% to 20%; high-load degradation judgment is implemented, responding to... This indicates that the current data segment contains a large amount of suspected dirty data, leading to a surge in external verification requests and posing a risk of memory backlog to the system.

[0174] According to the preset attenuation ratio Reduce the size of the read fragment, where, The attenuation ratio is preset within a certain range. Between, the preferred value is to ;Right now This is designed to reduce the concurrent pressure from external verification requests; the pressure reduction mechanism is based on a fixed-overhead throttling model:

[0175] Each data shard requires a fixed amount of context switching and security point checks to process. The overhead value is a preset system parameter, obtained by measuring the average scan time during the system initialization phase. Total processing time:

[0176]

[0177] When reading the fragment size When significantly reduced, useful computation time Reduce, and Remaining unchanged, this invention utilizes reduced read fragmentation. Cyclic overhead resulting in unit data amortization This increased proportion, by leveraging the system overhead that is typically considered a side effect, leads to a higher effective data throughput per unit time. This decrease, in turn, reduces the rate at which external requests are generated. This allows for soft flow shaping without introducing explicit sleep commands;

[0178] In this embodiment, the attenuation ratio The range of values ​​is preset to be The preferred value is to The specific method for determining this parameter is based on the release rate of the system memory buffer, which is relatively small. This value can more quickly curb memory backlog; it accelerates low-load judgment and responds in response to... This indicates that the current data segment is very clean, and the automaton almost never triggers a trap state; at this point, the preset growth rate will continue. Increase the read fragment size, where, The growth rate was set to [value]. to The values ​​between these ranges are used to achieve a gentle performance ramp-up, namely:

[0179]

[0180] In this embodiment, the growth ratio Set as to The smaller of the two values, i.e., using a mild multiplicative growth strategy, compared to the decay ratio. The multiplication formula shrinks dramatically. This asymmetric design prevents the read fragment size from oscillating wildly when dirty data fluctuates, aiming to reduce the number of database interactions and improve the throughput of vectorized pre-scan.

[0181] To adapt to the vectorized pre-scanning mechanism of Example 4, the calculated new read fragment size Alignment operations for the vector register width, either round-up or round-down, need to be performed to ensure... Always an integer multiple of the vector width, thus ensuring that SIMD instructions meet memory alignment requirements when loading data and avoiding performance penalties caused by unaligned access across cache lines;

[0182] To prevent divergence or deadlock in the negative feedback control loop, this embodiment introduces a boundary saturation constraint mechanism; the system presets a lower limit for the read fragment size. With the upper limit of read fragment size ;in, Set the processing width to a single vector instruction to prevent excessive loop overhead caused by reading too small a slice size; Set to 50% of the processor's L1D-Cache size to prevent cache thrashing caused by excessively large read fragment sizes; the calculated new read fragment size Saturation treatment is required:

[0183]

[0184] Only after alignment and saturation processing Only then will it be applied as the new read fragment size to the data stream read cursor; a steady-state hold strategy will be executed in response to... Then maintain the read fragment size. constant;

[0185] This embodiment implements an adaptive cruise mechanism for the cleaning system. When cleaning massive amounts of historical data, the dirty data in the data is often sparsely distributed. This mechanism ensures that the system can quickly traverse large clean areas, while automatically slowing down when encountering areas with dense dirty data. This maximizes the overall cleaning efficiency with limited memory resources and ensures the system's elastic scalability.

[0186] Example 8:

[0187] The state switching module is configured to perform lock-free switching, specifically including: pre-allocating a main buffer and a secondary buffer in memory;

[0188] Store the current state transition table in the main buffer and write the newly constructed state transition table into the secondary buffer;

[0189] Set the global atomic pointer to point to the main buffer;

[0190] After the secondary buffer is written, an atomic comparison and swap operation is performed to change the address pointed to by the global atomic pointer from the primary buffer to the secondary buffer;

[0191] After confirming that all existing read threads have exited the main buffer, release the memory space of the main buffer.

[0192] This embodiment details the method for the state switching module to perform lock-free switching, which utilizes the memory ordering characteristics of modern processors; it performs a memory pre-allocation step, in which two independent contiguous memory spaces are pre-allocated in the heap memory during the system initialization phase, and are marked as the main buffer and the secondary buffer respectively;

[0193] The background build process stores the current state transition table for the service being provided in the main buffer. When a rule change triggers compilation, the newly built state transition table is written to the secondary buffer.

[0194] Perform the atomic swap step and set a global atom pointer. Initially points to the main buffer;

[0195] After the data in the secondary buffer has been written and verified for integrity, an atomic compare-and-swap operation at the processor instruction level is executed. This operation will complete within a single instruction cycle. The address pointed to by the buffer is changed from the main buffer to the secondary buffer;

[0196] The system performs resource reclamation steps; specifically, to accurately confirm that all old read threads have exited the main buffer, the system introduces a periodic memory reclamation mechanism; the system maintains a global period counter. and each worker thread Local cycle counter ;

[0197] To ensure the validity of the count, each worker thread reads the current global cycle counter before starting to process a new data slice, i.e., when it is at a lock-free access safe point. And update to its local cycle counter. ,Right now This declares that the thread has entered the latest logical cycle; when a state transition occurs, the main thread atomically increments the value. The resource reclamation thread is responsible for polling the local cycle counters of each worker thread. Perform buffer release;

[0198] To prevent the global cycle count from failing to update due to some threads being in an idle waiting state, thus causing a memory reclamation deadlock, the system introduces a silent state management mechanism; each worker thread must perform an atomic operation to mark its own state as silent before entering a sleep waiting state after processing the current slice and when there is no new data to read.

[0199] Silent state refers to a worker thread being in an inactive, blocked waiting state, or confirming at a preset safe point that it will not access the old state transition table memory again within the current cycle; the resource reclamation thread polls and checks all active threads, only retrieving threads if the following conditions are met:

[0200]

[0201] in, This represents the set of all active worker threads in the system. For threads The current local cycle counter, It is a global cycle counter. Represents thread The running status, This represents a preset silent safety state;

[0202] That is, when the local cycle counters of all non-silent threads have caught up with the global cycle counter, or when the threads are clearly in a silent safe state that do not access shared memory, it is determined that all threads have migrated to the new state table context, and only then is the memory space of the original main buffer safely released.

[0203] This embodiment avoids the ABA problem and dangling pointer risk common in lock-free programming through the Epoch mechanism, achieving truly safe hot updates.

[0204] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An automatic illegal character cleaning system, characterized in that, include: The rule compilation module is used to listen for rule change events in the configuration center, obtain a discrete set of cleaning rules, perform semantic fusion and topological mapping on the set of cleaning rules, construct a deterministic finite automaton in memory, and map the deterministic finite automaton to a state transition table with continuous memory addresses to achieve flattening of the automaton graph structure. The state switching module is used to build a double-buffered execution context. After the state transition table is built, the current execution context is switched to the new state transition table without lock through atomic pointer operations, so as to realize the hot update of the cleaning logic. The speculative execution module is used to load the data stream to be cleaned in parallel using a single instruction multiple data stream instruction set, perform vectorized pre-scanning of the data stream based on the state transition table, identify the walk path of the data stream in the deterministic finite automaton, and directly allow the data stream when the walk path falls into a safe state, thereby constructing a computational firewall to intercept invalid external calls; when the walk path falls into a trap state, an external verification request is triggered to determine whether the data stream is dirty data. The adaptive feedback module is used to statistically analyze the trigger frequency and accurate verification results of the trap state in real time, generate a rule sensitivity report, and dynamically adjust the reading fragment size of the data stream based on the rule sensitivity report to complete the closed-loop control of the illegal character automatic cleaning system. Methods for semantic fusion and topological mapping of cleaning rule sets include: Each regular expression rule and enumerated rule in the cleansing rule set is parsed to generate the corresponding abstract syntax tree node; Merge all abstract syntax tree nodes into a unified global syntax tree; Perform node splitting and state merging on the global syntax tree to generate a nondeterministic finite automaton; A subset construction algorithm is used to convert a nondeterministic finite automaton into a deterministic finite automaton. The deterministic finite automata is minimized to eliminate redundant state nodes, and semantic fusion and topological mapping are completed. The methods for the state switching module to perform lock-free switching include: Pre-allocate a primary buffer and a secondary buffer in memory; Store the current state transition table in the main buffer and write the newly constructed state transition table into the secondary buffer; Set the global atomic pointer to point to the main buffer; After the secondary buffer is written, an atomic comparison and swap operation is performed to change the address pointed to by the global atomic pointer from the primary buffer to the secondary buffer; After confirming that all existing read threads have exited the main buffer, release the memory space of the main buffer.

2. The automatic illegal character cleaning system according to claim 1, characterized in that, The rule compilation module is configured to perform the following operations to map a deterministic finite automaton into a state transition table with contiguous memory addresses: Traverse all state nodes of the minimized deterministic finite automaton and assign a unique memory index offset to each state node; Extract the jump target index of each state node under different input characters, and construct a two-dimensional jump matrix; The two-dimensional jump matrix is ​​flattened into a one-dimensional linear array, and the linear array is aligned to the processor's cache line size to generate a state transition table.

3. The automatic illegal character cleaning system according to claim 1, characterized in that, The speculative execution module is configured to perform a vectorized pre-scan of the data stream based on the state transition table: Extract fixed-length byte segments from the data stream and load them into a vector register; The character category index corresponding to each byte in a byte segment is computed in parallel using vectorized instructions. Specifically, this includes: using a high-low bit segmentation mapping method to decompose the original byte vector into high- and low-bit index vectors, and using vectorized table lookup instructions to map them into character category indexes in parallel. Based on the current state index and the character category index, a direct address lookup is performed in the state transition table to obtain the next state index; The table lookup operation is performed repeatedly until the byte segment traversal is completed or the process enters a termination state. Paths that have not entered the termination state are marked as safe, and paths that have entered the termination state and are marked as potentially illegal characters are marked as traps.

4. The automatic illegal character cleaning system according to claim 1, characterized in that, The speculative execution module is configured to perform the following operation when the traversal path falls into a trap state: Lock the data segment that has fallen into the trap state and pause the vectorized pre-scan of the current thread; Construct a standard verification request containing data fragments and send it to a predefined external standard verification interface; Receive the Boolean check value returned by the external standard check interface; If the Boolean check value is true, the data segment is determined to be dirty data, and a cleaning and removal operation is performed. If the Boolean check value is false, the data segment is determined to be false alarm data, the data segment is retained, and the false alarm event is recorded.

5. The automatic illegal character cleaning system according to claim 4, characterized in that, The adaptive feedback module is configured to generate a rule-based allergy report, and the specific operations include: Within a preset time window, the total number of all data segments that fall into the trap state is accumulated and marked as the total amount of suspected dirty data; The total number of false alarms with all Boolean check values ​​of false is recorded as the total number of false alarms. Calculate the ratio of the total number of false alarms to the total number of suspected dirty data, and use it as the sensitivity coefficient of the current rule set; The ratio of the total amount of suspected dirty data to the total number of bytes of the data stream scanned within the preset time window is used as the spatial distribution density of suspected dirty data. By correlating the allergy coefficient with the distribution density of suspected dirty data, a rule-based allergy report is generated.

6. The automatic illegal character cleaning system according to claim 5, characterized in that, The adaptive feedback module is configured to dynamically adjust the read fragment size of the data stream based on the rule-based allergy report. Specific operations include: Get the preset step size of the current data stream read cursor; Extract the distribution density of suspected dirty data from the rule-based allergy report; A preset density threshold range includes a high density threshold and a low density threshold, wherein the high density threshold is greater than the low density threshold; If the distribution density of suspected dirty data is greater than the high density threshold, the preset step size will be reduced by a preset decay ratio to reduce the concurrent pressure of external verification requests. If the distribution density of suspected dirty data is less than the low density threshold, the preset step size is increased by a preset growth ratio to improve the throughput of vectorized pre-scan. If the distribution density of suspected dirty data is greater than or equal to the low density threshold and less than or equal to the high density threshold, then the preset step size remains unchanged.

Images