Hot plug data processing method and device, equipment and storage medium
By binding filter instances to the data processing system and adopting a double-buffering strategy, dynamic replacement of filters is achieved, solving the problem of inflexible filter replacement in existing technologies and ensuring the continuity of data processing and the maintainability of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-24
AI Technical Summary
In existing technologies, filters are bound at system startup and cannot be flexibly replaced, which restricts the adjustment of business rules. Furthermore, modifying filtering rules requires stopping the service, affecting business continuity. In addition, the data collection module and the filtering module are highly coupled, making it difficult to adapt to diverse business iteration needs.
By binding the first filter instance to the target data processing chain and using a double buffering strategy for initialization when a new filter instance registration request is received, the data stream is switched from the current filter instance to the new filter instance at a preset switching point, ensuring the continuity and flexibility of data processing.
It enables dynamic replacement of filter instances without interrupting the data processing chain, reducing upgrade costs and business interruption risks, and improving system maintainability and scalability.
Smart Images
Figure CN121722631A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a hot-swappable data processing method, apparatus, device and storage medium. Background Technology
[0002] In large-scale data processing systems, input data usually needs to undergo multiple filtering and processing processes to meet the business needs of different scenarios. The data filtering module is a core component for ensuring data quality and adapting to business logic.
[0003] In existing technologies, filters are bound at system startup, and the filtering logic is fixed and cannot be flexibly replaced at runtime, which severely restricts the adjustment of business rules. At the same time, modifying filtering rules requires stopping the service and redeploying, which not only has high upgrade costs but also interrupts the data processing flow and affects business continuity. In addition, the data collection module and the filtering module are tightly bound together, with a high degree of coupling, which greatly increases the difficulty of system expansion and makes it difficult to adapt to diverse business iteration needs.
[0004] While dependency injection technology can reduce module coupling to some extent, how to achieve seamless replacement (hot-plugging) of filters during system operation, while ensuring uninterrupted data collection and consistent data transmission, has become a key problem that urgently needs to be solved in the current technology field. Summary of the Invention
[0005] This application provides a hot-swappable data processing method, apparatus, device, and storage medium, which can realize the dynamic replacement of filter instances at runtime without interrupting the data flow of the target data processing chain, while ensuring the continuity of data processing and reducing the risk of switching.
[0006] In a first aspect, this application provides a hot-swappable data processing method, comprising: binding a first filter instance to a target data processing chain, so that the default filter instance performs data processing on the data stream in the target data processing chain; when a new filter instance registration request is received, registering the new filter instance and initializing the new filter instance using a double buffering strategy; and at a preset switching point, switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance.
[0007] In one possible implementation, after switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance, the method further includes: recording a switching log, wherein the switching log is used to support rollback operations; if an abnormality is detected in the operation of the new filter instance, the data stream in the target data processing chain is switched back to the first filter based on the switching log.
[0008] In one possible implementation, binding the first filter instance to the target data processing chain so that the default filter instance processes the data stream in the target data processing chain specifically includes: in response to system startup, obtaining the initialization parameters of the default filter through a dependency injection container, and instantiating and configuring the default filter according to the initialization parameters to obtain the first filter instance; binding the first filter instance to a preset filter node of the target data processing chain so that the business data flowing through the target data processing chain is processed by the first filter instance.
[0009] In one possible implementation, obtaining the initialization parameters of the default filter through a dependency injection container, and instantiating and configuring the default filter according to the initialization parameters to obtain a first filter instance, specifically includes: obtaining a preset system configuration file through a dependency injection container, and extracting the initialization parameters of the default filter from the system configuration file, wherein the initialization parameters include at least the full path of the default filter implementation class, the target data processing chain identifier, the unified interface specification, and the filter binding priority; loading the default filter implementation class based on the full path of the default filter implementation class, performing interface compatibility verification on the default filter implementation class, determining whether the default filter implementation class conforms to the unified interface specification definition, and if the verification passes, creating a default filter instance based on the dependency injection container, and injecting the filter binding priority and the target data processing chain identifier from the initialization parameters into the default filter instance to obtain the first filter instance.
[0010] In one possible implementation, registering a new filter instance upon receiving a new filter instance registration request specifically includes: receiving a new filter instance registration request, wherein the new filter instance registration request is automatically generated by obtaining the real-time performance monitoring results of the first filter instance, and if the performance monitoring results meet the filter instance replacement conditions; the performance monitoring results include one or more of the following: single data processing time, data filtering error rate, and resource utilization rate of the current filter; performing integrity and validity checks on the request parameters in the new filter instance registration request; if the request parameters pass the checks, obtaining the first initialization parameters of the new filter from the request parameters through a dependency injection container, and instantiating and configuring the new filter according to the first initialization parameters to obtain a new filter instance; synchronizing the first basic information of the new filter instance to the filter registry, and assigning a unique identifier to the new filter instance.
[0011] In one possible implementation, the initialization process of the new filter instance using a double-buffering strategy specifically includes: when an initialization instruction for the new filter instance is received, based on the double-buffering strategy, a buffer initialization area independent of the currently running first filter instance is created within the system, and the new filter instance is loaded and configured within the buffer initialization area; a warm-up test is performed on the new filter instance based on test data, and the running indicators of the new filter instance are collected; based on the comparison result between the running indicators and preset running indicator conditions, it is determined whether the new filter instance has been successfully initialized. If successful, the status of the new filter instance is updated to ready; if it fails, the resources of the buffer initialization area are released and an error is reported.
[0012] In one possible implementation, the step of switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point specifically includes: monitoring the first running state of the target data processing chain; determining whether the first filter instance has met the preset switching conditions based on the first running state; when the switching conditions are met, pausing the transmission of new data in the target data processing chain to the currently running first filter instance, and switching the data stream of the target data processing chain from the first filter instance to the new filter instance at the preset switching point; after the switching is completed, verifying the second running state of the new filter instance; if the verification passes, confirming the switching is successful; if the verification fails, triggering a rollback mechanism to switch the data stream back to the first filter instance.
[0013] Secondly, this application provides a hot-swappable data processing apparatus, comprising: a data binding module, a new filter instance processing module, and a filter instance switching module; wherein, the data binding module is used to bind a first filter instance to a target data processing chain, so that the default filter instance performs data processing on the data stream in the target data processing chain; the new filter instance processing module is used to register a new filter instance when a new filter instance registration request is received, and to initialize the new filter instance using a double buffering strategy; the filter instance switching module is used to switch the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point.
[0014] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.
[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.
[0016] This application provides a hot-swappable data processing method, apparatus, device, and storage medium, which have the following advantages compared with the prior art:
[0017] This method binds a first filter instance to a target data processing chain, enabling the default filter instance to process the data stream in the target data processing chain. Upon receiving a new filter instance registration request, the new filter instance is registered and initialized using a double-buffering strategy. At a preset switching point, the data stream in the target data processing chain is switched from the currently running first filter instance to the initialized new filter instance. Compared to existing technologies, this application's technical solution, by binding the first filter instance to the target data processing chain, ensures that the system can quickly enter a stable data processing state after startup, meeting the basic requirements. First, it meets basic business requirements. Second, when receiving a new filter instance registration request, it initializes the new instance using a double-buffering strategy, avoiding interference with the current data stream processing during the initialization process and ensuring the continuity of data processing. Finally, it completes the switching of the data stream from the first filter instance to the new filter instance at a preset switching point. This not only enables dynamic updates of the filtering logic without stopping the service and redeploying, reducing upgrade costs and business interruption risks, but also ensures data transmission consistency through a clear switching mechanism. At the same time, due to the binding method between the filter instance and the data processing chain and the switching logic, it effectively reduces the coupling between the data collection module and the filtering module, improving the maintainability and scalability of the system. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0021] Figure 1 This is a flowchart illustrating one embodiment of a hot-swappable data processing method provided in this application;
[0022] Figure 2 This is a schematic diagram of the structure of one embodiment of a hot-swappable data processing device provided in this application;
[0023] Figure 3 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0026] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0030] Example 1, see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of a hot-swappable data processing method provided in this application, as shown below. Figure 1 As shown, the method includes steps 101-103 and is applicable to large-scale data processing systems. The large-scale data processing system includes a dependency injection container, a filter implementation class, a filter registry, and a management module. Steps 101-103 are detailed below:
[0031] Step 101: Bind the first filter instance to the target data processing chain so that the default filter instance processes the data stream in the target data processing chain.
[0032] In one embodiment, in response to system startup, the initialization parameters of the default filter are obtained through a dependency injection container, and the default filter is instantiated and configured according to the initialization parameters to obtain a first filter instance.
[0033] Specifically, a dependency injection container is used to obtain a preset system configuration file, and the initialization parameters of the default filter are extracted from the system configuration file. The initialization parameters include at least the full path of the default filter implementation class, the target data processing chain identifier, the unified interface specification, and the filter binding priority. The default filter implementation class is loaded based on the full path of the default filter implementation class, and the interface compatibility of the default filter implementation class is checked to determine whether the default filter implementation class conforms to the definition of the unified interface specification. If the check passes, a default filter instance is created based on the dependency injection container, and the filter binding priority and the target data processing chain identifier in the initialization parameters are injected into the default filter instance to obtain the first filter instance.
[0034] Specifically, the dependency injection container is responsible for managing the creation, destruction, and replacement of filter instances; the filter implementation class is a specific filter implementation for different business logics and can be replaced at any time.
[0035] Specifically, when performing interface compatibility verification on the default filter implementation class, the method list of the default filter implementation class is obtained through reflection mechanism, and the method list is compared with the method signature of the unified interface specification. If there are missing methods or signature mismatches, the verification is deemed to have failed.
[0036] Specifically, when performing interface compatibility verification on the default filter implementation class to determine whether the default filter implementation class conforms to the unified interface specification definition, the method also includes generating an interface incompatibility exception log based on the dependency injection container if the verification fails, triggering a system degradation strategy, and loading a preset backup default filter implementation class.
[0037] Preferably, the system degradation strategy includes reading the full path of a preset backup default filter implementation class from the configuration file based on the dependency injection container, loading the backup default filter implementation class based on the full path of the backup default filter implementation class, performing interface compatibility verification on the backup default filter implementation class, determining whether the backup default filter implementation class conforms to the unified interface specification definition, and if the verification passes, creating a backup default filter instance based on the dependency injection container, and injecting the filter binding priority and the target data processing chain identifier from the initialization parameters into the backup default filter instance to obtain a first filter instance; if the backup filter still fails verification, triggering a system alarm and terminating the data processing chain initialization, awaiting manual intervention.
[0038] Specifically, after successful validation, the dependency injection container creates an instance of the default filter via reflection. Subsequently, the dependency injection container injects the filter binding priority and target data processing chain identifier from the initialization parameters into the instance object. Specifically, the priority is set by calling the instance's setPriority(int priority) method, and the target data processing chain is associated by the setTargetChain Id(String chain Id) method. The injection of these parameters makes the filter clear about its position in the business process: the priority determines its execution order in the combination of multiple filters, and the target data processing chain identifier ensures that it only processes the data flow of the specified chain.
[0039] Specifically, after instantiation and parameter injection, the default filter becomes the first filter instance, which is the first filter instance to take effect after the system starts. It has complete filtering logic and business configuration and can be directly connected to the target data processing chain to start working.
[0040] In one embodiment, the first filter instance is bound to a preset filter node of the target data processing chain so that the business data flowing through the target data processing chain is processed by the first filter instance.
[0041] Specifically, the data processing chain corresponding to the target data processing chain identifier is obtained, and the data processing chain is used as the target data processing chain; the first filter instance is bound to a preset filter node of the target data processing chain so that the business data flowing through the target data processing chain is processed by the first filter instance; after the binding is completed, the basic information of the first filter instance is synchronized to the filter registry, wherein the basic information includes at least the instance unique identifier, implementation class version, bound target data processing chain identifier, current status of the filter instance, and interface specification version.
[0042] Specifically, in large-scale data processing systems, such as high-concurrency hotel distribution systems, there are typically multiple independent business data processing chains. Each chain corresponds to a specific business scenario, such as a hotel order processing chain, an inventory data synchronization chain, and a pricing strategy calculation chain. Each data processing chain is distinguished by a unique identifier. After obtaining the target data processing chain identifier, the dependency injection container queries the mapping relationship corresponding to the identifier through the system's built-in data processing chain management module to locate the running instance of the corresponding data processing chain. This process ensures that the first filter instance can accurately access the target business process, avoiding business data processing anomalies caused by incorrect chain matching. After the location is completed, the chain is determined as the target data processing chain and awaits the binding of the filter instance.
[0043] Specifically, the preset filter node is a fixed node in the target data processing chain after the original data is received and before the data is stored / forwarded. The dependency injection container connects the default filter instance to this node through interface adaptation, ensuring that the data flow in the target data processing chain is filtered before entering the subsequent business process.
[0044] Specifically, after determining the target data processing chain, the dependency injection container calls the interface of the preset filter node of that chain to bind the first filter instance to the preset filter node. After binding, when business data flows through the target data processing chain, the doFilter() method of the first filter instance in the filter node will be automatically triggered to filter, clean, or transform the data according to the logic of the first filter instance.
[0045] In one embodiment, after binding the first filter instance to a preset filter node of the target data processing chain, the basic information of the first filter instance is synchronized to the filter registry based on the dependency injection container; at the same time, an initialization completion notification is sent to the system's management module to complete the loading and binding process of the default filter instance.
[0046] Specifically, the filter registry is a distributed registry that supports multi-node data synchronization. After the basic information of the first filter instance is synchronized, the filter registry generates a unique index for the first filter instance, which is used for subsequent filter queries and replacements.
[0047] Step 102: When a new filter instance registration request is received, register the new filter instance and initialize the new filter instance using a double buffering strategy.
[0048] In one embodiment, when a new filter instance registration request is received, the registration of the new filter instance is carried out by receiving the new filter instance registration request. The new filter instance registration request is automatically generated by obtaining the real-time performance monitoring results of the first filter instance. If the performance monitoring results meet the filter instance replacement conditions, the performance monitoring results include one or more of the following: single data processing time, data filtering error rate, and resource utilization rate of the current filter. The completeness and validity of the request parameters in the new filter instance registration request are verified. If the request parameters pass the verification, the first initialization parameters of the new filter are obtained from the request parameters through a dependency injection container. Based on the first initialization parameters, the new filter is instantiated and configured to obtain the new filter instance. The first basic information of the new filter instance is synchronized to the filter registry, and a unique identifier is assigned to the new filter instance.
[0049] Specifically, during system operation, the management module continuously collects the operational performance data of the first filter instance, including: processing time for a single data item, data filtering error rate, and resource utilization rate. The processing time for a single data item reflects processing efficiency, such as the average processing time for order data filtering. The data filtering error rate reflects processing accuracy, such as the percentage of orders that are misjudged or missed. The resource utilization rate reflects system load, such as the CPU and memory resources used by the filter instance. These data are compared in real-time with preset filter replacement conditions, such as processing time > 50ms, error rate > 3%, and memory utilization > 80%. If any indicator meets or exceeds the threshold, the management module automatically generates a new filter instance registration request.
[0050] Example description: In a hotel's high-concurrency distribution system, if the average processing time of the first filter for order data reaches 60ms, exceeding the threshold of 50ms, the system determines that it cannot meet the real-time requirements and automatically triggers a registration request. The request includes the expected functions and basic configuration requirements of the new filter. After the dependency injection container receives the request, it enters the subsequent verification and instantiation process, realizing the on-demand triggering of filter replacement and avoiding the lag of manual intervention.
[0051] Specifically, since the registration request for a new filter instance needs to carry multiple key parameters, the integrity check refers to checking whether required parameters are included, such as the full path of the implementation class and the version identifier. If any are missing, the check will fail and a message indicating incomplete parameters will be returned. The validity check refers to verifying whether the parameter format and value conform to the system specifications. For example, the version identifier must follow the semantic format of Vx.yz, the applicable business scenario tag must belong to the preset scenario library, and the filtering threshold in the initialization dependency configuration must be a positive integer. If the parameter verification passes, the dependency injection container confirms that the configuration of the new filter meets the system access requirements and enters the instantiation stage. If the verification fails, the container reports the reason for the failure to the management module through the event bus, terminates the registration process, and ensures that only compliant filters can enter the system.
[0052] In one embodiment, when initializing the new filter instance using a double-buffering strategy, upon receiving the initialization instruction for the new filter instance, a buffer initialization area independent of the currently running first filter instance is created within the system based on the double-buffering strategy. Within this buffer initialization area, the new filter instance is loaded and configured. A warm-up test is performed on the new filter instance based on test data, and its operational metrics are collected. Based on the comparison results between the operational metrics and preset operational metric conditions, it is determined whether the new filter instance has been successfully initialized. If successful, the status of the new filter instance is updated to ready; if it fails, the resources of the buffer initialization area are released and an error is reported.
[0053] Specifically, when the dependency injection container receives the initialization instruction for a new filter instance, it immediately allocates an independent buffer initialization area in the system memory based on a double-buffering strategy. This buffer initialization area is completely isolated from the running area of the first filter instance, and has its own dedicated thread pool, memory space, and resource quota. It will not occupy the running resources of the first filter, nor will it interfere with the normal data flow of the target data processing chain. For example, when the first filter instance is processing high-concurrency hotel order data, the buffer initialization area will allocate CPU cores and memory separately, specifically for loading and configuring the new filter, ensuring that the continuity and real-time performance of order data processing are not affected. This double-buffering isolation design is the core advantage of the double-buffering strategy. It provides an independent initialization environment for the new filter instance and avoids problems such as resource contention and data loss that may occur in the traditional run-while-initializing mode, laying the foundation for smooth switching.
[0054] Specifically, within the buffer initialization area, the dependency injection container performs the loading and configuration operations of the new filter instance based on the first initialization parameters determined during the previous registration phase, such as the full path of the implementation class, filtering thresholds, and applicable scenarios. First, the implementation class of the new filter is loaded through the class loader to ensure that the class file is complete and compatible with the system environment. Then, the instance's initialization method is called to inject the filtering rules, business parameters, and other configurations into the instance. For example, core configurations such as a single data processing timeout threshold of 30ms and an order amount filtering condition of ≥500 yuan are injected into the low-latency order filter. The entire loading and configuration process is completed independently within the buffer initialization area without interacting with the first filter instance or accessing the real business data stream. After configuration, the new filter instance has complete filtering logic but has not yet processed real business data and is in a state of pending testing.
[0055] Specifically, the dependency injection container extracts test data that is highly consistent with real business scenarios from the system's historical business database or test data pool, such as hotel order data and inventory synchronization data. The data volume, data format, and concurrency intensity are close to the actual business peak. For example, it extracts 1,000 mixed data records containing normal orders, abnormal orders, and large-amount orders. Then, the container calls the doFilter() method of the new filter instance to let the test data flow through the new filter and execute the complete filtering process, i.e., warm-up test.
[0056] Specifically, during the preheating test, the management module will collect the core operating metrics of the new filter instance in real time, including the processing time of a single data item, the data filtering accuracy, the number of exceptions thrown, and the resource utilization rate. For example, it will record the filtering time of each test order, statistically analyze the matching degree between the filtering results and the preset standards, and monitor the peak memory usage of the instance. These metrics will serve as the core basis for determining whether the initialization is successful.
[0057] Specifically, after the preheating test is completed, the system will compare the collected operating metrics with the preset operating metric conditions one by one. The preset conditions are usually formulated based on business needs and system performance requirements, such as single data processing time ≤ 50ms, filtering accuracy ≥ 99.9%, exception throwing count = 0, memory usage ≤ 50%, etc. If all metrics meet the preset conditions, it means that the performance and function of the new filter instance meet the business requirements, and the initialization is determined to be successful. The dependency injection container will update the status of the new filter instance to ready and synchronize it to the filter registry, waiting for subsequent data flow switching. If any metric does not meet the preset conditions, the initialization is determined to be unsuccessful. The dependency injection container will immediately release all resources in the buffer initialization area to avoid resource waste and report error details to the management module through the event bus, such as initialization failure, single data processing time exceeding the limit, while maintaining the normal operation of the first filter instance to ensure that the business is not affected.
[0058] Step 103: At a preset switching point, switch the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance.
[0059] In one embodiment, the first operating state of the target data processing chain is monitored, and based on the first operating state, it is determined whether the first filter instance has reached a preset switching condition.
[0060] Specifically, the dependency injection container continuously collects the first running status of the target data processing chain through the management module to determine whether a preset switching point has been reached. If the preset switching point has been reached, it determines whether the first filter instance has met the preset switching conditions. Specifically, when determining whether the preset switching point has been reached, if the preset switching point is a single data processing task completion node, it monitors whether the first filter instance has completed filtering the current data; if so, it determines whether the first filter instance has met the preset switching conditions. If the preset switching point is a fixed time node, it waits for the system time to reach a preset time; if so, it determines whether the first filter instance has met the preset switching conditions. If the preset switching point is a low-traffic period node, it monitors whether the real-time data traffic of the target data processing chain is below a preset threshold; if so, it determines whether the first filter instance has met the preset switching conditions.
[0061] In one embodiment, when the switching condition is met, the sending of new data in the target data processing chain to the currently running first filter instance is paused, and at a preset switching point, the data stream of the target data processing chain is switched from the first filter instance to the new filter instance.
[0062] Specifically, once the switching conditions are met, the dependency injection container sends a "switching preparation notification" to the event bus, triggering the data collection module to pause sending new data to the first filter instance while waiting for the first filter instance to complete processing the remaining data of the current batch. After the first filter instance finishes processing the current batch of data and releases temporary resources, the dependency injection container routes the data flow of the target data processing chain to the new filter instance. This design avoids the loss of data in transit caused by a one-size-fits-all switch, ensuring that the first filter instance can fully process the current load.
[0063] Specifically, after the first filter instance finishes processing all remaining data and releases temporary cache and other resources, the system performs a data flow route switch at a preset switching point. The switching operation is completed through the routing management module of the target data processing chain. The core is to redirect the receiving end of the data flow from the interface of the first filter instance to the unified interface of the new filter instance. For example, the order data flow that originally flowed to com.hotel.filter.DefaultFilter.doFilter() will be directly redirected to com.hotel.filter.NewLowLatencyFilter.doFilter() after the switch. The entire routing adjustment process is completed in milliseconds, with almost no noticeable business interruption.
[0064] In one embodiment, after the switch is completed, the second running state of the new filter instance is verified. If the verification passes, the switch is confirmed to be successful; if the verification fails, a rollback mechanism is triggered to switch the data stream back to the first filter instance.
[0065] Specifically, after the data stream switch is completed, the data collection module sends hotel distribution data to the new filter instance through the unified filter interface. The dependency injection container monitors the second running status of the new filter instance in real time and records the switch process log, which includes the switch time, the identifiers of the old and new filter instances, the switch point type, and the duration of the data processing interruption.
[0066] Specifically, the dependency injection container monitors the second running status of the new filter instance in real time, including: whether the new filter instance successfully receives and processes the first new data, whether the data filtering result conforms to the business rules, the response speed of the new filter instance, and whether there are any abnormal errors or resource usage exceeding limits.
[0067] Specifically, if the verification passes, the system confirms the switch was successful, updates the status of the new filter instance to "effective," and synchronizes it to the filter registry. Subsequent business data will continue to be processed by the new filter instance.
[0068] Specifically, if the verification fails, such as the new filter instance failing to receive data, the filtering result being incorrect, or the response timing out, the rollback mechanism is immediately triggered: the system suspends sending data to the new filter instance, reroutes the data stream back to the first filter instance at the current switching point or the next security node, and simultaneously resumes pushing new data to the first filter instance; after the rollback is completed, the system updates the registry status, restores the first filter instance to active status, and sends a switching failure + rollback completion notification to the management module so that operations and maintenance personnel can troubleshoot problems with the new filter instance. The entire rollback process does not affect the continuous processing of business data, ensuring system stability.
[0069] In one embodiment, after switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance, a switching log is also recorded, wherein the switching log is used to support rollback operations; if an abnormal operation of the new filter instance is detected, the data stream in the target data processing chain is switched back to the first filter based on the switching log.
[0070] Specifically, after the data stream successfully switches from the first filter instance to the new filter instance, the system automatically triggers the recording operation of the switch log. The switch log does not simply record the switch event, but contains a set of structured data with key dimensions, including: switch timestamp, unique identifiers of the old and new filter instances, target data processing chain identifier, switch trigger conditions, switch time, and system resource status at the time of switch.
[0071] Specifically, the logs are stored in the system's audit log library or distributed log system, possessing the characteristics of being immutable and quickly queried. Their core function is to provide complete context for rollback operations. When a new filter instance malfunctions, the system can quickly locate the state before the switch through the logs, such as the original first effective filter instance and the data flow routing configuration before the switch, ensuring that the rollback process is accurate and efficient without the need to reconfigure basic parameters.
[0072] Specifically, after the data stream switch is completed, the system's monitoring module will monitor the running status of the new filter instance in real time and at high frequency. The monitoring dimensions are consistent with the preheating test indicators in the initialization phase and cover both business and performance dimensions. The specific monitoring dimensions include: the accuracy of filtering results and the success rate of data reception at the business level; the processing time of a single piece of data and the resource utilization rate at the performance level; and the number of exceptions thrown at the system level.
[0073] Specifically, monitoring is not only carried out for a short period of time after the switch, but throughout the entire operation cycle of the new filter instance. Once any monitoring indicator meets the anomaly judgment rules, such as three consecutive data reception failures, processing time exceeding the threshold for 10 consecutive seconds, or a filtering error rate greater than 3%, the system will determine that the new filter instance is operating abnormally and trigger the subsequent rollback process.
[0074] Specifically, when an abnormal operation of a new filter instance is detected, the system immediately invokes the rollback mechanism, relying on previously recorded switch logs to perform the operation. First, the system quickly queries the corresponding switch log using the unique identifier of the new filter instance or the identifier of the target data processing chain, extracting key configurations from the logs before the switch, including the unique identifier of the original effective first filter instance, the data flow routing rules before the switch, and the running status of the first filter instance. Then, following the same process as the forward switch, a reverse data flow switch is performed: first, sending new data to the new filter instance is paused, waiting for it to process the remaining data in transit; then, based on the routing configuration in the logs, the data flow of the target data processing chain is switched back to the first filter instance, restoring the effective state of the first filter instance, while pausing the operation of the new filter instance.
[0075] Specifically, the entire rollback process requires no manual intervention, and the structured data based on the switching logs ensures that the system state before and after the rollback is completely consistent with that before the switch, preventing issues such as data loss and incorrect routing configurations. This effectively reduces the impact of abnormal new filter instances on business operations and ensures the stability of large-scale data processing systems.
[0076] In one embodiment, the hot-swappable data processing method provided in this application can be applied not only to large-scale data processing systems, but also to big data processing platforms, financial risk control systems, Internet of Things (IoT) data acquisition, online advertising and recommendation systems, cloud computing and maintenance service architectures, network security and intrusion detection, and medical and health data processing.
[0077] Specifically, in the field of big data processing platforms, it can dynamically replace data filtering logic without interrupting services in scenarios such as log analysis, behavioral data collection, and real-time stream processing (e.g., Apache Flink, Kafka Streams), enabling flexible adjustments to business rules. In the field of financial risk control systems, it can adjust filters in real time to address new risk models or compliance requirements in high-frequency trading, real-time risk monitoring, and anti-fraud systems, ensuring the continuity and security of transaction data processing. In the field of Internet of Things (IoT) data collection, it can dynamically replace data filters in smart devices and sensor networks to adapt to different device protocols or data quality requirements, ensuring the accuracy and timeliness of collected data. In the field of online advertising and recommendation systems, it can adjust filtering logic in real time during ad delivery or content recommendation. To respond to changes in user behavior or adjustments in market strategies, it improves system response speed and recommendation accuracy; in the field of cloud computing and maintenance service architecture, it enables hot-swapping of filtering components within services through dependency injection in microservice-based distributed systems, reducing coupling between services and supporting rapid iteration and canary releases; in the field of network security and intrusion detection, it enables dynamic replacement of filters in real-time traffic analysis and intrusion detection systems to cope with new attack patterns or security policy changes, ensuring system protection capabilities; in the field of medical and health data processing, it enables flexible adjustment of data filtering rules in medical device data acquisition, electronic medical record processing, and real-time monitoring systems to meet different diagnosis and treatment scenarios and regulatory requirements.
[0078] Example 2, see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of a hot-swappable data processing apparatus provided in this application. Corresponding to the above-described hot-swappable data processing method, this application also provides a hot-swappable data processing apparatus. This hot-swappable data processing apparatus includes modules for executing the above-described hot-swappable data processing method, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, the hot-swappable data processing apparatus includes a data binding module 201, a new filter instance processing module 202, and a filter instance switching module 203.
[0079] The data binding module 201 is used to bind the first filter instance to the target data processing chain, so that the default filter instance can perform data processing on the data stream in the target data processing chain.
[0080] The new filter instance processing module 202 is used to register a new filter instance when a new filter instance registration request is received, and to perform initialization processing on the new filter instance using a double buffering strategy.
[0081] The filter instance switching module 203 is used to switch the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point.
[0082] In one embodiment, the filter instance switching module 203, after switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance, further includes: recording a switching log, wherein the switching log is used to support rollback operations; if an abnormal operation of the new filter instance is detected, the data stream in the target data processing chain is switched back to the first filter based on the switching log.
[0083] In one embodiment, the data binding module 201 is used to bind a first filter instance to a target data processing chain so that the default filter instance processes the data stream in the target data processing chain. Specifically, this includes: in response to system startup, obtaining the initialization parameters of the default filter through a dependency injection container, and instantiating and configuring the default filter according to the initialization parameters to obtain a first filter instance; binding the first filter instance to a preset filter node of the target data processing chain so that the business data flowing through the target data processing chain is processed by the first filter instance.
[0084] In one embodiment, the data binding module 201 is used to obtain the initialization parameters of a default filter through a dependency injection container, and to instantiate and configure the default filter according to the initialization parameters to obtain a first filter instance. Specifically, this includes: obtaining a preset system configuration file through a dependency injection container, and extracting the initialization parameters of the default filter from the system configuration file, wherein the initialization parameters include at least the full path of the default filter implementation class, the target data processing chain identifier, the unified interface specification, and the filter binding priority; loading the default filter implementation class based on the full path of the default filter implementation class, performing interface compatibility verification on the default filter implementation class, determining whether the default filter implementation class conforms to the unified interface specification definition, and if the verification passes, creating a default filter instance based on the dependency injection container, and injecting the filter binding priority and the target data processing chain identifier from the initialization parameters into the default filter instance to obtain the first filter instance.
[0085] In one embodiment, the new filter instance processing module 202 is used to register a new filter instance when a new filter instance registration request is received. Specifically, it includes: receiving a new filter instance registration request, wherein the new filter instance registration request is automatically generated by obtaining the real-time performance monitoring results of the first filter instance, and if the performance monitoring results meet the filter instance replacement conditions. The performance monitoring results include one or more of the following: single data processing time, data filtering error rate, and resource utilization rate of the current filter; performing integrity and validity verification on the request parameters in the new filter instance registration request; if the request parameters pass the verification, obtaining the first initialization parameters of the new filter from the request parameters through a dependency injection container, and instantiating and configuring the new filter according to the first initialization parameters to obtain a new filter instance; synchronizing the first basic information of the new filter instance to the filter registry, and assigning a unique identifier to the new filter instance.
[0086] In one embodiment, the new filter instance processing module 202 is used to initialize the new filter instance using a double-buffering strategy. Specifically, it includes: when receiving the initialization instruction for the new filter instance, creating a buffer initialization area independent of the currently running first filter instance in the system based on the double-buffering strategy, and loading and configuring the new filter instance within the buffer initialization area; performing a warm-up test on the new filter instance based on test data and collecting the running indicators of the new filter instance; determining whether the new filter instance has been successfully initialized based on the comparison result between the running indicators and preset running indicator conditions; if successful, updating the status of the new filter instance to ready; if unsuccessful, releasing the resources of the buffer initialization area and reporting an error.
[0087] In one embodiment, the filter instance switching module 203 is used to switch the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point. Specifically, it includes: monitoring the first running state of the target data processing chain; determining whether the first filter instance has reached a preset switching condition based on the first running state; when the switching condition is reached, pausing the sending of new data in the target data processing chain to the currently running first filter instance, and switching the data stream of the target data processing chain from the first filter instance to the new filter instance at the preset switching point; after the switching is completed, verifying the second running state of the new filter instance; if the verification passes, confirming the switching is successful; if the verification fails, triggering a rollback mechanism to switch the data stream back to the first filter instance.
[0088] The hot-swappable data processing apparatus described above can implement the hot-swappable data processing method of the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here.
[0089] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a computer device provided in this application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.
[0090] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the hot-plug data processing method provided in any of the foregoing method embodiments.
[0091] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.
[0092] Therefore, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the hot-plug data processing method provided in any of the foregoing method embodiments.
[0093] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.
[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.
[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0096] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0099] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.
[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A hot-swappable data processing method, characterized in that, include: Bind the first filter instance to the target data processing chain so that the default filter instance processes the data stream in the target data processing chain; When a new filter instance registration request is received, the new filter instance is registered, and a double buffering strategy is used to initialize the new filter instance. At a preset switching point, the data stream in the target data processing chain is switched from the currently running first filter instance to the initialized new filter instance.
2. The method as described in claim 1 above, characterized in that, After switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance, the method further includes: Record a switching log, wherein the switching log is used to support rollback operations; If an abnormal operation of the new filter instance is detected, the data stream in the target data processing chain will be switched back to the first filter based on the switching log.
3. The method as described in claim 1 above, characterized in that, The step of binding the first filter instance to the target data processing chain, so that the default filter instance processes the data stream in the target data processing chain, specifically includes: In response to system startup, the initialization parameters of the default filter are obtained through a dependency injection container, and the default filter is instantiated and configured according to the initialization parameters to obtain a first filter instance; The first filter instance is bound to a preset filter node of the target data processing chain so that the business data flowing through the target data processing chain is processed by the first filter instance.
4. The method as described in claim 3 above, characterized in that, The process of obtaining the initialization parameters of the default filter through a dependency injection container, and instantiating and configuring the default filter according to the initialization parameters to obtain the first filter instance, specifically includes: A preset system configuration file is obtained through a dependency injection container, and the initialization parameters of the default filter are extracted from the system configuration file. The initialization parameters include at least the full path of the default filter implementation class, the target data processing chain identifier, the unified interface specification, and the filter binding priority. The default filter implementation class is loaded based on its full path. An interface compatibility check is performed on the default filter implementation class to determine whether it conforms to the unified interface specification definition. If the check passes, a default filter instance is created based on the dependency injection container, and the filter binding priority and the target data processing chain identifier in the initialization parameters are injected into the default filter instance to obtain the first filter instance.
5. The method as described in claim 1, characterized in that, The step of registering a new filter instance upon receiving a new filter instance registration request specifically includes: Receive a new filter instance registration request, wherein the new filter instance registration request is automatically generated by obtaining the real-time running performance monitoring results of the first filter instance. If the running performance monitoring results meet the filter instance replacement conditions, the running performance monitoring results include one or more of the following: single data processing time of the current filter, data filtering error rate, and resource utilization rate. The request parameters in the registration request for the new filter instance are validated for completeness and validity. If the request parameters pass the validation, the first initialization parameters of the new filter are obtained from the request parameters through a dependency injection container. Based on the first initialization parameters, the new filter is instantiated and configured to obtain a new filter instance. The first basic information of the new filter instance is synchronized to the filter registry, and a unique identifier is assigned to the new filter instance.
6. The method as described in claim 1, characterized in that, The initialization process for the new filter instance using a double-buffering strategy specifically includes: When the initialization instruction for the new filter instance is received, based on the double buffering strategy, a buffer initialization area independent of the currently running first filter instance is created in the system, and the new filter instance is loaded and configured within the buffer initialization area. The new filter instance was preheated based on the test data, and its operating metrics were collected. Based on the comparison results between the operating indicators and the preset operating indicator conditions, it is determined whether the new filter instance has been successfully initialized. If successful, the status of the new filter instance is updated to ready. If it fails, the resources of the buffer initialization area are released and an error is reported.
7. The method as described in claim 1, characterized in that, The step of switching the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point specifically includes: Monitor the first operating state of the target data processing chain, and determine whether the first filter instance has reached the preset switching condition based on the first operating state; When the switching condition is met, the sending of new data in the target data processing chain to the currently running first filter instance is paused, and at a preset switching point, the data stream of the target data processing chain is switched from the first filter instance to the new filter instance; After the switch is completed, the second running state of the new filter instance is verified. If the verification passes, the switch is confirmed to be successful; if the verification fails, the rollback mechanism is triggered to switch the data stream back to the first filter instance.
8. A hot-swappable data processing device, characterized in that, include: Data binding module, new filter instance processing module, and filter instance switching module; The data binding module is used to bind the first filter instance to the target data processing chain, so that the default filter instance can perform data processing on the data stream in the target data processing chain. The new filter instance processing module is used to register a new filter instance when a new filter instance registration request is received, and to initialize the new filter instance using a double buffering strategy. The filter instance switching module is used to switch the data stream in the target data processing chain from the currently running first filter instance to the initialized new filter instance at a preset switching point.
9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.