Data filter execution system based on chain of responsibility mode and implementation method thereof

Abstract

The invention discloses a data filter execution system based on a chain of responsibility mode and an implementation method thereof. The system comprises a filter chain manager, a filter execution engine, a context management system, an exception handling and recovery center and a monitoring and auditing module. The filter chain manager is responsible for dynamically configuring the filter and managing the dependency relationship of the filter; the execution engine executes the filters in sequence and supports front check and state monitoring; the context management system is used for creating and maintaining a FilterContext object for packaging the key information; the exception handling and recovery center provides multi-level exception capture and data consistency guarantee; and the monitoring and auditing module is responsible for collecting performance data and recording an execution track. Through dynamic configuration, efficient execution, intelligent management and comprehensive monitoring, the reliability, consistency and expansibility of data processing are remarkably improved.

Description

Technical Field

[0001] This application relates to the fields of computer software technology and data processing technology, and in particular to a data filter execution system based on the chain of responsibility model and its implementation method. Background Technology

[0002] In enterprise-level data processing systems, data typically undergoes multi-level processing and transformation. Traditional data processing methods primarily employ a serial processing model, which presents numerous technical problems, limiting the flexibility and reliability of data processing. Specific problems include:

[0003] Fixed data processing logic: Traditional data processing systems typically fix the processing logic in the code, lacking a flexible extension mechanism and making it difficult to adapt to rapidly changing business needs.

[0004] Insufficient exception handling capabilities: In the data processing process, traditional systems often lack a sound exception handling mechanism. Once an exception occurs, it can easily lead to data loss or inconsistency, affecting the stability of the system and the reliability of the data.

[0005] Lack of fault recovery and rollback mechanisms: When a fault occurs during data processing, traditional systems lack effective fault recovery and rollback mechanisms, making it difficult to guarantee the integrity and consistency of data processing.

[0006] Complex dependencies between filters: In data processing scenarios with multiple filters, the dependencies between filters are complex, making them difficult to maintain and debug, which increases the complexity and maintenance cost of the system.

[0007] Dynamic configuration and runtime adjustment are not supported: Traditional data processing systems typically do not support dynamic configuration and runtime adjustment of filter chains, making it difficult to meet the changing needs of real-time business requirements. Summary of the Invention

[0008] Based on this, this application provides a data filter execution system and its implementation method based on the chain of responsibility model. By configuring filter chain management, efficient execution engine, intelligent context management, comprehensive exception handling and recovery mechanism, and intelligent monitoring and auditing technology, it solves the technical problems of poor data processing flexibility, imperfect exception handling, and weak fault recovery capability in the prior art.

[0009] Firstly, a data filter execution method based on the chain of responsibility pattern is provided, the method comprising:

[0010] A filter chain manager is used to load filter configuration information from a configuration database, construct a directed acyclic graph of filter dependencies, and determine the execution order of filters using a topology sorting algorithm; wherein the filter configuration information includes at least filter type, priority, dependency relationship, and execution parameters;

[0011] The filter execution engine executes filters sequentially according to the execution order determined by the filter chain manager. Before each filter is executed, a pre-state check is performed. During execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output results is verified, the context state is updated, and the result is passed to the next filter. Filters access and modify data through the FilterContext object.

[0012] A context management system is used to create and maintain FilterContext context objects; wherein, the FilterContext context object encapsulates input data, query parameters, user information, and data source identification information;

[0013] The exception handling and recovery center is used to continuously monitor the execution status of the filter chain. It adopts a multi-level exception capture mechanism to capture exceptions at the filter level and determine the handling strategy according to the exception type.

[0014] The monitoring and auditing module is used to collect performance data of filter execution in real time, calculate performance trends using a sliding window algorithm, issue warnings when performance indicators exceed preset thresholds, and record the execution trajectory of the filter chain; the performance data includes at least execution time, throughput, memory usage, and CPU usage.

[0015] Optionally, the filter chain manager includes:

[0016] A directed acyclic graph of filter dependencies is constructed using graph theory algorithms, and the execution order of the filters is determined by a topological sorting algorithm.

[0017] The system dynamically adds, removes, and modifies filter configurations at runtime, maintains the version history and change records of the filter chain configuration, and automatically detects and resolves configuration conflicts between filters.

[0018] Optionally, the filter execution engine includes:

[0019] Before each filter is executed, obtain the FilterContext context object from the context management system;

[0020] Perform pre-processing state checks on each filter, including data integrity verification, business rule checks, and permission verification;

[0021] Execute the specific processing logic of the filter and update the processing result to the FilterContext context object;

[0022] Create checkpoints at key nodes to record the current execution status;

[0023] After execution, verify the correctness of the output, update the context state, and pass the FilterContext object to the next filter.

[0024] Optionally, the context management system includes:

[0025] Assign a unique context ID to each request and create an initial state snapshot. Create context snapshots at critical nodes to support rollback and recovery.

[0026] Automatically manage the creation, use, and destruction of context objects, and employ weak references and timely cleanup mechanisms to prevent memory leaks;

[0027] Before each filter is executed, the FilterContext object is passed to the filter execution engine;

[0028] After each filter completes its execution, the updated FilterContext object is received and passed to the next filter.

[0029] Optionally, the exception handling and recovery center includes:

[0030] Continuously monitor the execution status of the filter chain and employ a multi-level exception handling mechanism;

[0031] At the filter level, business anomalies, parameter anomalies, and resource anomalies are captured, and the handling strategy is determined based on the anomaly type.

[0032] At the chain level, the execution status of the entire filter chain is monitored, and a global recovery process is initiated when a chain interruption, timeout, or dependency failure is detected.

[0033] At the system level, when thread interruption, insufficient memory, or network anomalies are detected, a decision is made on whether to continue executing subsequent filters based on a pre-configured exception handling strategy. A compensation operation stack is maintained, and the compensation actions of each filter are recorded. During rollback, the compensation operations are executed in reverse order to ensure data consistency.

[0034] Optionally, the monitoring and auditing module includes:

[0035] Provides a visual monitoring interface to display the execution status, performance statistics, and anomaly analysis information of the filter chain;

[0036] Record the execution trajectory of the filter chain, including input data, execution time, output results, and exception information;

[0037] Provides a graphical monitoring interface and report analysis.

[0038] Optionally, the data filter execution system further includes a data backup manager for creating data snapshots before and after the filter chain execution, supporting fine-grained data recovery; wherein, the data backup manager specifically includes:

[0039] Before the filter chain is executed, the original data is deeply copied. After each filter is executed, the backup data changes. An efficient compression algorithm is used to reduce the storage space of the backup data. At the same time, an asynchronous mechanism is used to reduce the impact of backup operations on the main process.

[0040] Optionally, the system further includes a state verifier for performing state checks and data integrity verification before and after filter execution; wherein, the state verifier specifically includes:

[0041] Before the filter is executed, a pre-processing state check is performed, including data integrity verification, business rule check, and resource availability check.

[0042] Post-processing validation is performed after the filter is executed, including output data format validation and data volume consistency check.

[0043] It provides chained verification to verify the data consistency of the entire filter chain and issues early warnings of anomalies based on historical data and pattern recognition.

[0044] Optionally, the system further includes a checkpoint manager, used to set checkpoints at key nodes of the filter chain to achieve fine-grained saving and restoring of execution state; wherein, the checkpoint manager specifically includes:

[0045] Create execution checkpoints at key nodes to record the current execution status, intermediate results, and filter status;

[0046] An asynchronous mechanism is used to persist checkpoint data to the storage system, while simultaneously saving checkpoint information to a memory cache;

[0047] When recovering from a checkpoint, verify the integrity and correctness of the checkpoint data to ensure that the data has not been tampered with;

[0048] It supports resuming execution from any checkpoint, avoiding redundant processing, and automatically detects abnormal interruptions in the filter chain execution to trigger the recovery process.

[0049] Secondly, a method for implementing a data filter execution system based on the chain of responsibility model, implemented in any of the data filter execution systems described in the first aspect above, the method comprising:

[0050] Filter chain initialization and configuration: Load filter configuration information from the configuration database, construct a directed acyclic graph of filter dependencies, and determine the execution order of filters using a topology sorting algorithm; wherein, the filter configuration information includes at least filter type, priority, dependency relationship, and execution parameters;

[0051] Filter chain execution and context management: Filters are executed sequentially according to a defined execution order, with a pre-execution state check performed before each filter. During execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output results is verified, the context state is updated, and the result is passed to the next filter. Filters access and modify data through the FilterContext object. The FilterContext object is created and maintained, encapsulating input data, query parameters, user information, and data source identification information.

[0052] Exception handling and fault recovery: Continuously monitor the execution status of the filter chain, adopt a multi-level exception capture mechanism to capture exceptions at the filter level, and determine the handling strategy according to the exception type;

[0053] Monitoring, auditing, and performance optimization: Real-time collection of filter execution performance data, calculation of performance trends using a sliding window algorithm, issuance of alerts when performance metrics exceed preset thresholds, and recording of the filter chain execution trajectory; performance data includes at least execution time, throughput, memory usage, and CPU usage.

[0054] The beneficial effects of the technical solutions provided in this application include at least the following:

[0055] 1. High reliability: Through a comprehensive anomaly handling and fault recovery mechanism, the system reliability is improved by more than 90%;

[0056] 2. Strong consistency: Through transaction management and data backup, ensure the atomicity and consistency of data processing;

[0057] 3. High scalability: Supports dynamic configuration and runtime adjustment of filters, improving scalability by 80%;

[0058] 4. High performance: Processing performance is improved by 60% through parallel execution and performance optimization;

[0059] 5. Easy to maintain: Provides comprehensive monitoring and auditing functions, greatly improving the maintainability of the system;

[0060] 6. Fault tolerance: Supports interrupt point recovery and multi-level rollback, significantly improving the system's fault tolerance. Attached Figure Description

[0061] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0062] Figure 1 A block diagram of a data filter execution system based on the chain of responsibility pattern is provided for embodiments of this application;

[0063] Figure 2 This document presents a step diagram illustrating the implementation method of a data filter execution system based on the chain of responsibility pattern, as provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0066] This application can be widely used in enterprise data processing platforms, ETL data processing, business process engines, API gateways, message processing middleware, and other scenarios, and is particularly suitable for enterprise-level systems that require complex data processing and high reliability. Through this application, enterprises can build flexible and reliable data processing pipelines, improving the efficiency and quality of data processing.

[0067] Please refer to Figure 1 The diagram illustrates a block diagram of a data filter execution system based on the chain of responsibility pattern provided in an embodiment of this application. The system specifically includes:

[0068] The filter chain manager is used to load filter configuration information from the configuration database, construct a directed acyclic graph of filter dependencies, and determine the execution order of filters through a topology sorting algorithm. The filter configuration information includes at least filter type, priority, dependencies, and execution parameters.

[0069] The filter execution engine executes filters sequentially according to the execution order determined by the filter chain manager. Before each filter is executed, a pre-state check is performed. During execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output results is verified, the context state is updated, and the result is passed to the next filter. Filters access and modify data through the FilterContext object.

[0070] The context management system is used to create and maintain FilterContext context objects; the FilterContext context object encapsulates input data, query parameters, user information, and data source identification information.

[0071] The exception handling and recovery center is used to continuously monitor the execution status of the filter chain. It adopts a multi-level exception capture mechanism to capture exceptions at the filter level and determine the handling strategy according to the exception type.

[0072] The monitoring and auditing module is used to collect performance data of filter execution in real time, calculate performance trends using a sliding window algorithm, issue warnings when performance indicators exceed preset thresholds, and record the execution trajectory of the filter chain; the performance data includes at least execution time, throughput, memory usage, and CPU usage.

[0073] The filter chain manager includes: constructing a directed acyclic graph of filter dependencies using graph theory algorithms, and determining the execution order of filters using a topological sorting algorithm; dynamically adding, removing, and modifying filter configurations at runtime, maintaining the version history and change records of filter chain configurations, and automatically detecting and resolving configuration conflicts between filters.

[0074] The filter execution engine includes: obtaining a FilterContext object from the context management system before each filter is executed; performing pre-state checks on each filter, including data integrity verification, business rule checks, and permission verification; executing the specific processing logic of the filter and updating the processing results in the FilterContext object; creating checkpoints at key nodes to record the current execution state; and verifying the correctness of the output results, updating the context state, and passing the FilterContext object to the next filter after execution is complete.

[0075] The context management system includes: assigning a unique context ID to each request and creating an initial state snapshot; creating context snapshots at critical nodes and supporting rollback and recovery; automatically managing the creation, use, and destruction of context objects, and employing weak references and timely cleanup mechanisms to prevent memory leaks; passing the FilterContext object to the filter execution engine before each filter is executed; and receiving the updated FilterContext object after each filter is executed and passing it to the next filter.

[0076] The exception handling and recovery center includes: continuously monitoring the execution status of the filter chain and employing a multi-layered exception capture mechanism; at the filter level, capturing business exceptions, parameter exceptions, and resource exceptions, and determining the handling strategy based on the exception type; at the chain level, monitoring the execution status of the entire filter chain, and initiating a global recovery process when a link interruption, timeout, or dependency failure is detected; at the system level, capturing thread interruptions, insufficient memory, and network exceptions, and determining whether to continue executing subsequent filters based on pre-configured exception handling strategies, maintaining a compensation operation stack, recording the compensation actions of each filter, and executing compensation operations in reverse order during rollback to ensure data consistency.

[0077] The monitoring and auditing module includes: providing a visual monitoring interface to display the execution status, performance statistics, and anomaly analysis information of the filter chain; recording the execution trajectory of the filter chain, including input data, execution time, output results, and anomaly information; and providing a graphical monitoring interface and report analysis.

[0078] The data filter execution system also includes a data backup manager, which creates data snapshots before and after the filter chain is executed, supporting fine-grained data recovery; specifically, the data backup manager includes:

[0079] Before the filter chain is executed, the original data is deeply copied. After each filter is executed, the backup data changes. An efficient compression algorithm is used to reduce the storage space of the backup data. At the same time, an asynchronous mechanism is used to reduce the impact of backup operations on the main process.

[0080] The system also includes a state validator, used to perform state checks and data integrity verification before and after filter execution; specifically, the state validator includes:

[0081] Before the filter is executed, a pre-processing state check is performed, including data integrity verification, business rule check, and resource availability check.

[0082] Post-processing validation is performed after the filter is executed, including output data format validation and data volume consistency check.

[0083] It provides chained verification to verify the data consistency of the entire filter chain and issues early warnings of anomalies based on historical data and pattern recognition.

[0084] The system also includes a checkpoint manager, used to set checkpoints at critical nodes in the filter chain, enabling fine-grained saving and restoring of execution state; specifically, the checkpoint manager includes:

[0085] Create execution checkpoints at key nodes to record the current execution status, intermediate results, and filter status;

[0086] An asynchronous mechanism is used to persist checkpoint data to the storage system, while simultaneously saving checkpoint information to a memory cache;

[0087] When recovering from a checkpoint, verify the integrity and correctness of the checkpoint data to ensure that the data has not been tampered with;

[0088] It supports resuming execution from any checkpoint, avoiding redundant processing, and automatically detects abnormal interruptions in the filter chain execution to trigger the recovery process.

[0089] In summary, this application provides a data filter execution system based on the chain of responsibility model, comprising:

[0090] 1. Filter Chain Manager: Responsible for dynamic configuration and dependency management of filters.

[0091] 2. Filter execution engine: Provides an efficient chain of responsibility execution mechanism.

[0092] 3. Context Management System: Maintains context information during data processing.

[0093] 4. Anomaly Handling and Recovery Center: Provides comprehensive anomaly capture and fault recovery capabilities.

[0094] 5. Monitoring and Auditing Module: Enables full-process monitoring and auditing of filter execution.

[0095] The following are specific technical features of the embodiments of this application:

[0096] Feature 1: Intelligent Filter Chain Management Technology

[0097] The system employs a dynamically configured filter chain management mechanism:

[0098] 1. Dependency Analysis: Construct a directed acyclic graph (DAG) to analyze the dependencies between filters.

[0099] 2. Priority sorting: The execution order of filters is automatically sorted based on a weighted algorithm.

[0100] 3. Dynamic registration: Supports dynamically adding, removing, and modifying filter configurations at runtime.

[0101] 4. Version Management: Maintain the version history and change records of the filter chain configuration.

[0102] 5. Conflict Detection: Automatically detects and resolves configuration conflicts between filters.

[0103] Feature 2: High-performance chain of responsibility execution engine

[0104] Achieving efficient data processing based on the chain of responsibility model:

[0105] 1. Bidirectional traversal: Supports forward traversal of the pre-filter and reverse traversal of the post-filter.

[0106] 2. Parallel execution: Parallel execution is supported for filters with no dependencies.

[0107] 3. Short-circuit mechanism: When the filter returns a termination flag, subsequent execution is stopped immediately.

[0108] 4. Context passing: Data passing between filters is achieved through the FilterContext object.

[0109] 5. Performance optimization: Use object pooling and caching mechanisms to reduce memory allocation overhead.

[0110] Feature 3: Intelligent Context Management System

[0111] Achieve full lifecycle context management:

[0112] 1. Context encapsulation: Encapsulate query parameters, data source information, user information, etc. into a unified context.

[0113] 2. State Snapshot: Creates context snapshots at critical nodes, supporting rollback and recovery.

[0114] 3. Data transfer: Supports secure data transfer and sharing between filters.

[0115] 4. Lifecycle Management: Automatically manages the creation, use, and destruction of context objects.

[0116] 5. Memory optimization: Weak references and timely cleanup mechanisms are used to prevent memory leaks.

[0117] Feature 4: Comprehensive Anomaly Handling and Recovery Mechanism

[0118] Implement multi-level exception handling and fault recovery strategies:

[0119] 1. Layered exception capture and handling: The system implements refined exception capture at each stage of filter execution.

[0120] Filter-level exceptions: Capture business exceptions, runtime exceptions, and resource exceptions during the execution of a single filter.

[0121] Chain-level exceptions: Capture configuration exceptions, dependency exceptions, and timeout exceptions during the overall execution of the filter chain.

[0122] System-level exceptions: Capture system-level failures such as thread interruption, insufficient memory, and network errors.

[0123] Exception handling mechanism: The system uses try-catch blocks to comprehensively capture exceptions during filter execution. First, it checks if the filter execution result contains error information; if an error is found, a dedicated error handling method is invoked. For business exceptions, the system records detailed error log information and adds warning messages to the context, then continues execution of the next filter, ensuring the continuity of the business process. For runtime exceptions, the system immediately triggers a rollback mechanism, stopping the current execution flow and throwing a filter chain exception to ensure data consistency. For other types of exceptions, the system decides whether to continue executing subsequent filters based on pre-configured exception handling strategies, achieving flexible exception handling control.

[0124] 2. Global transaction management and rollback strategy: Design a filter-based rollback mechanism based on transaction semantics.

[0125] Transaction boundary definition: Defines transaction boundaries for the filter chain, supporting nested transactions.

[0126] Compensation operation: Define a compensation operation for each filter, and perform the reverse operation during rollback.

[0127] Two-phase commit: a pre-commit check is performed before the critical filters are executed.

[0128] Atomicity guarantee: Ensures that the execution result of the filter chain has atomic characteristics.

[0129] Consistency check: Verify the consistency and integrity of the data after the rollback is complete.

[0130] The core implementation mechanism of the transaction manager is as follows: The system design uses the `TransactionManager` class, which internally maintains a stack-based compensation operation queue to record the reverse compensation operations of each filter. When executing a filter, the corresponding compensation operation object is first retrieved from the current filter and pushed onto the compensation stack for storage. Then, the specific processing logic of the filter is executed in a try-catch structure. If an exception occurs during execution, the rollback method is immediately called to restore the data. The rollback mechanism adopts the last-in-first-out (LIFO) characteristic of the stack structure, popping compensation operations sequentially and executing them to ensure that the rollback is performed in the reverse order of execution. The execution of each compensation operation also adopts an exception handling mechanism, so that even if one compensation operation fails, it will not affect the execution of other compensation operations, and detailed information about the compensation failure is recorded in the log.

[0131] 3. Data backup and rapid recovery, implementing a multi-layered data protection and recovery mechanism:

[0132] Input data backup: Perform a deep copy of the original data before the filter chain is executed.

[0133] Incremental backup: Backup data changes after each filter is executed.

[0134] Versioned backup: Maintains multiple versions of data and supports rollback to any version.

[0135] Compressed storage: Employs efficient compression algorithms to reduce the storage space required for backup data.

[0136] Asynchronous backup: Using an asynchronous mechanism reduces the impact of backup operations on the main process.

[0137] Data Backup Manager Implementation Mechanism: The system is designed as a DataBackupManager, which uses a ConcurrentHashMap thread-safe collection to maintain backup history, with context identifiers as keys and the corresponding list of backup snapshots as values. When creating a data snapshot, the system constructs a DataSnapshot object containing key information such as the current timestamp, filter name, deep copy of the data, and data checksum. The system uses the computeIfAbsent method to ensure thread-safe addition of snapshot records to the backup history; if the corresponding context does not exist, a new snapshot list is automatically created. During data recovery, the system retrieves the corresponding snapshot list based on the context identifier, then iterates through the snapshot records in reverse chronological order to find a backup point matching the specified filter name. After finding the target backup, the system verifies the snapshot checksum to ensure data integrity; if verification is successful, a copy of the backup data is returned. If no valid backup is found or verification fails, the system throws a "Backup Not Found" exception, providing detailed error information for problem localization.

[0138] 4. Status checks and data integrity verification: Establish a comprehensive status monitoring and verification system.

[0139] Pre-processing status check: Verify the data status and system environment before the filter is executed.

[0140] Execution process monitoring: Real-time monitoring of filter execution status, resource consumption, and performance metrics.

[0141] Post-processing result verification: Check the data integrity and business constraints after the filter is executed.

[0142] Chained verification: Verifies the data consistency of the entire filter chain.

[0143] Anomaly warning: Issues early warnings of anomalies based on historical data and pattern recognition.

[0144] State Validator Implementation Mechanism: The system designs a StateValidator class, providing two core methods: precondition validation and postcondition validation. The precondition validation method receives the filter context and filter instance as parameters, creating a ValidationResult object to collect validation information. The validation process includes three levels: First, a data integrity check is performed, validating the integrity and validity of the data in the context. If the check fails, an error message indicating a data integrity check failure is added to the result object. Second, business rule validation is performed, checking the compliance of the context data according to the specific filter's business rules. If validation fails, a business rule validation failure error is recorded. Finally, a resource availability check is performed, confirming the availability of system resources and external dependencies required for filter execution. If unavailable, corresponding error messages are added. The postcondition validation method receives the context and filter execution results, validating the results: First, it checks whether the output data format conforms to the expected specifications; if the format is invalid, an invalid output data format error is recorded. Then, a data consistency check is performed, comparing the consistency between the input and output data. If inconsistencies are found, a warning message is added. After validation, a ValidationResult object containing all validation results is returned.

[0145] 5. Interrupt point recovery mechanism to achieve fine-grained saving and restoration of execution state:

[0146] Checkpoint strategy: Set checkpoints at critical nodes in the filter chain.

[0147] State serialization: Serializes and saves the execution context, intermediate results, and filter states.

[0148] Resume interrupted downloads: Supports resuming execution from any checkpoint, avoiding redundant processing.

[0149] Fault detection: Automatically detects abnormal interruptions in the execution of the filter chain and triggers the recovery process.

[0150] Resume verification: Verify the correctness of the status and the continuity of the data after resumption of execution.

[0151] Checkpoint Manager Implementation Mechanism: The system designs a CheckpointManager class, using a ConcurrentHashMap thread-safe collection to maintain checkpoint information, with the context identifier as the key to correspond to the execution checkpoint object. When saving a checkpoint, the system creates an ExecutionCheckpoint object, containing key information such as the current system timestamp, filter index position, a deep copy of the context object, and the state hash value. To avoid the checkpoint saving operation affecting the performance of the main execution flow, the system uses a CompletableFuture asynchronous execution mechanism, persisting the checkpoint data to the storage system in a background thread, while simultaneously saving the checkpoint information to a memory cache for fast access. When recovering from a checkpoint, the system first looks up the corresponding checkpoint in the memory cache based on the context identifier, checks whether the filter index meets the recovery requirements, and then verifies the integrity of the checkpoint data to ensure that the data has not been tampered with. If the checkpoint in memory passes verification, a deep copy of the context object is directly returned; if the in-memory checkpoint is unavailable or verification fails, the system automatically loads the checkpoint data from persistent storage for recovery, ensuring that the system has robust fault recovery capabilities.

[0152] Feature 5: Intelligent Monitoring and Auditing Technology

[0153] Achieve comprehensive execution monitoring and auditing:

[0154] 1. Performance monitoring: Real-time monitoring of filter execution time, throughput, and resource consumption.

[0155] 2. Execution Link Tracing: Records the complete filter execution link and data change trajectory.

[0156] 3. Anomaly Statistics: Statistics and analysis of the frequency, type, and scope of impact of anomalies.

[0157] 4. Audit Log: Records detailed operation logs to support compliance audits.

[0158] 5. Visualization: Provides a graphical monitoring interface and report analysis.

[0159] Please refer to Figure 2 This document illustrates the steps of an implementation method for a data filter execution system based on the chain of responsibility pattern, provided in an embodiment of this application. The method may include:

[0160] Step 1: Filter Chain Initialization and Configuration

[0161] When the system starts, the FilterChainManager loads filter configuration information from the configuration database, including filter type, priority, dependencies, and execution parameters. The system uses graph theory algorithms to construct a directed acyclic graph (DAG) of filter dependencies and a topological sorting algorithm to determine the execution order of filters, ensuring the correctness of dependencies. Then, the filters are sorted according to their priority configuration: `Collections.sort(filters, Comparator.comparing(Filter::getPriority))`, constructing both pre-filter and post-filter lists. The system creates filter instances using the Spring IoC container, injects necessary dependencies such as data sources, caching services, and configuration services, and performs initialization checks on the filters to verify the validity of configuration parameters and interface compatibility. Finally, filter change listeners are registered to support dynamic configuration updates at runtime.

[0162] Step 2: Filter Chain Execution and Context Management

[0163] When a data processing request arrives, the system first creates a `FilterContext` object, encapsulating key information such as input data, query parameters, user information, and data source identifier. The `ContextManager` assigns a unique context ID to each request and creates an initial state snapshot. During the pre-filter execution phase, the system executes pre-filters sequentially in a pre-sorted order: `for (ProcessorFilter filter: preFilters) { FilterResult result = filter.preProcess(context); if (result.isTerminated()) { return result;} context.merge(result.getContext());}`. Before each filter is executed, pre-state checks are performed, including data integrity verification, business rule checks, and permission verification. During filter execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output result is verified, the context state is updated, and the filter is passed to the next filter.

[0164] Step 3: Anomaly Handling and Fault Recovery

[0165] The ExceptionMonitor continuously monitors the execution status of the filter chain, employing a multi-layered exception handling mechanism. At the filter level, it captures business exceptions, parameter exceptions, resource exceptions, etc., and determines the handling strategy based on the exception type: for recoverable exceptions, it records warning messages and continues execution; for fatal exceptions, it immediately stops execution and triggers a rollback process. The chain-level exception handling mechanism monitors the execution status of the entire filter chain. When exceptions such as chain interruption, timeout, or dependency failure are detected, a global recovery process is initiated. The TransactionManager maintains a compensation operation stack, recording the compensation actions of each filter. During rollback, the compensation operations are executed in reverse order to ensure data consistency. The DataBackupManager creates data snapshots before and after each filter execution, supporting fine-grained data recovery. When an exception occurs, the system selects an appropriate recovery strategy based on the fault level: partial rollback, full rollback, or recovery from a checkpoint.

[0166] Step 4: Monitoring, Auditing, and Performance Optimization

[0167] PerformanceMonitor collects real-time performance data of filter execution, including key metrics such as execution time, throughput, memory usage, and CPU utilization. The system uses a sliding window algorithm to calculate performance trends and issues alerts when performance metrics exceed preset thresholds. AuditLogger records the execution trajectory of the filter chain in detail, including input data, execution time, output results, and exception information, supporting post-event troubleshooting and compliance auditing. The system provides a visual monitoring interface that displays the execution status, performance statistics, and anomaly analysis of the filter chain, helping operations personnel to promptly identify and resolve issues. Based on historical execution data, the system automatically optimizes the execution order and resource allocation of filters to improve overall performance. For frequently executed filter chains, a caching mechanism is used to reduce redundant calculations; for resource-intensive filters, asynchronous execution and resource pool management are employed to ensure efficient and stable system operation.

[0168] In summary, the innovative points of this application are:

[0169] 1. Multi-level exception handling architecture: An innovative three-level exception handling system of filters, links, and system was designed.

[0170] 2. Transactional filter execution: This is the first time that transactional semantics have been introduced into the execution of the filter chain to ensure data consistency.

[0171] 3. Intelligent fault recovery mechanism: Automatic fault recovery technology based on checkpoints and compensation operations.

[0172] 4. Dynamic configuration management: Supports runtime dynamic configuration and hot updates of filter chains.

[0173] 5. End-to-end monitoring and auditing: Provides complete link tracing and auditing capabilities from input to output.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data filter execution system based on the chain of responsibility model, characterized in that, The system includes: A filter chain manager is used to load filter configuration information from a configuration database, construct a directed acyclic graph of filter dependencies, and determine the execution order of filters using a topology sorting algorithm; wherein the filter configuration information includes at least filter type, priority, dependency relationship, and execution parameters; The filter execution engine executes filters sequentially according to the execution order determined by the filter chain manager. Before each filter is executed, a pre-state check is performed. During execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output results is verified, the context state is updated, and the result is passed to the next filter. Filters access and modify data through the FilterContext object. A context management system is used to create and maintain FilterContext context objects; wherein, the FilterContext context object encapsulates input data, query parameters, user information, and data source identification information; The exception handling and recovery center is used to continuously monitor the execution status of the filter chain. It adopts a multi-level exception capture mechanism to capture exceptions at the filter level and determine the handling strategy according to the exception type. The monitoring and auditing module is used to collect performance data of filter execution in real time, calculate performance trends using a sliding window algorithm, issue warnings when performance indicators exceed preset thresholds, and record the execution trajectory of the filter chain; the performance data includes at least execution time, throughput, memory usage, and CPU usage.

2. The data filter execution system according to claim 1, characterized in that, The filter chain manager includes: A directed acyclic graph of filter dependencies is constructed using graph theory algorithms, and the execution order of the filters is determined by a topological sorting algorithm. The system dynamically adds, removes, and modifies filter configurations at runtime, maintains the version history and change records of the filter chain configuration, and automatically detects and resolves configuration conflicts between filters.

3. The data filter execution system according to claim 1, characterized in that, The filter execution engine includes: Before each filter is executed, obtain the FilterContext context object from the context management system; Perform pre-processing state checks on each filter, including data integrity verification, business rule checks, and permission verification; Execute the specific processing logic of the filter and update the processing result to the FilterContext context object; Create checkpoints at key nodes to record the current execution status; After execution, verify the correctness of the output, update the context state, and pass the FilterContext object to the next filter.

4. The data filter execution system according to claim 1, characterized in that, The context management system includes: Assign a unique context ID to each request and create an initial state snapshot. Create context snapshots at critical nodes to support rollback and recovery. Automatically manage the creation, use, and destruction of context objects, and employ weak references and timely cleanup mechanisms to prevent memory leaks; Before each filter is executed, the FilterContext object is passed to the filter execution engine; After each filter completes its execution, the updated FilterContext object is received and passed to the next filter.

5. The data filter execution system according to claim 1, characterized in that, The anomaly handling and recovery center includes: Continuously monitor the execution status of the filter chain and employ a multi-level exception handling mechanism; At the filter level, business anomalies, parameter anomalies, and resource anomalies are captured, and the handling strategy is determined based on the anomaly type. At the chain level, the execution status of the entire filter chain is monitored, and a global recovery process is initiated when a chain interruption, timeout, or dependency failure is detected. At the system level, when thread interruption, insufficient memory, or network anomalies are detected, a decision is made on whether to continue executing subsequent filters based on a pre-configured exception handling strategy. A compensation operation stack is maintained, and the compensation actions of each filter are recorded. During rollback, the compensation operations are executed in reverse order to ensure data consistency.

6. The data filter execution system according to claim 1, characterized in that, The monitoring and auditing module includes: Provides a visual monitoring interface to display the execution status, performance statistics, and anomaly analysis information of the filter chain; Record the execution trajectory of the filter chain, including input data, execution time, output results, and exception information; Provides a graphical monitoring interface and report analysis.

7. The data filter execution system according to claim 1, characterized in that, The data filter execution system also includes a data backup manager, used to create data snapshots before and after the filter chain execution, supporting fine-grained data recovery; specifically, the data backup manager includes: Before the filter chain is executed, the original data is deeply copied. After each filter is executed, the backup data changes. An efficient compression algorithm is used to reduce the storage space of the backup data. At the same time, an asynchronous mechanism is used to reduce the impact of backup operations on the main process.

8. The data filter execution system according to claim 1, characterized in that, The system also includes a state verifier for performing state checks and data integrity verification before and after filter execution; specifically, the state verifier includes: Before the filter is executed, a pre-processing state check is performed, including data integrity verification, business rule check, and resource availability check. Post-processing validation is performed after the filter is executed, including output data format validation and data volume consistency check. It provides chained verification to verify the data consistency of the entire filter chain and issues early warnings of anomalies based on historical data and pattern recognition.

9. The data filter execution system according to claim 1, characterized in that, The system also includes a checkpoint manager, used to set checkpoints at key nodes of the filter chain to achieve fine-grained saving and restoring of execution state; specifically, the checkpoint manager includes: Create execution checkpoints at key nodes to record the current execution status, intermediate results, and filter status; An asynchronous mechanism is used to persist checkpoint data to the storage system, while simultaneously saving checkpoint information to a memory cache; When recovering from a checkpoint, verify the integrity and correctness of the checkpoint data to ensure that the data has not been tampered with; It supports resuming execution from any checkpoint, avoiding redundant processing, and automatically detects abnormal interruptions in the filter chain execution to trigger the recovery process.

10. A method for implementing a data filter execution system based on the chain of responsibility pattern, implemented in the data filter execution system as described in any one of claims 1-9, characterized in that, The implementation method includes: Filter chain initialization and configuration: Load filter configuration information from the configuration database, construct a directed acyclic graph of filter dependencies, and determine the execution order of filters using a topology sorting algorithm; wherein, the filter configuration information includes at least filter type, priority, dependency relationship, and execution parameters; Filter chain execution and context management: Filters are executed sequentially according to a defined execution order, with a pre-execution state check performed before each filter. During execution, the execution status is monitored in real time, performance metrics are recorded, and checkpoints are created at key nodes. After execution, the correctness of the output results is verified, the context state is updated, and the result is passed to the next filter. Filters access and modify data through the FilterContext object. The FilterContext object is created and maintained, encapsulating input data, query parameters, user information, and data source identification information. Exception handling and fault recovery: Continuously monitor the execution status of the filter chain, adopt a multi-level exception capture mechanism to capture exceptions at the filter level, and determine the handling strategy according to the exception type; Monitoring, auditing, and performance optimization: Real-time collection of filter execution performance data, calculation of performance trends using a sliding window algorithm, issuance of alerts when performance metrics exceed preset thresholds, and recording of the filter chain execution trajectory; performance data includes at least execution time, throughput, memory usage, and CPU usage.

Images