Timestamp-based data change intelligent deduplication and message simplification method and system

Abstract

The invention provides a timestamp-based data change intelligent deduplication and message simplification method and system, and the method comprises the steps: capturing a change event of a database in real time, and generating a change record containing a timestamp; based on a preset timestamp comparison algorithm, performing timestamp analysis and comparison on the multiple change records of the same service entity; the change records are grouped according to the service keys, the latest change records are stored and dynamically updated through a hash table, the latest timestamp data under the same service key are reserved, and intelligent duplicate removal of redundant data is achieved; when the data volume exceeds a preset threshold value, a data fragmentation strategy is triggered, batch data is uniformly segmented into a plurality of subtasks, multi-thread parallel processing of fragmented data is achieved by dynamically allocating thread resources, and processing results are summarized; and carrying out merging processing on the deduplicated changed data, constructing a standardized message body, selecting immediate sending or delayed queue distribution according to a message priority, and controlling a network transmission load through a message batching mechanism.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for intelligent data change deduplication and message simplification based on timestamps. Background Art

[0002] In large-scale distributed systems, data changes occur frequently, especially in database monitoring and message queue processing scenarios, where the same data may change multiple times in a short period of time. Traditional data processing methods often treat each change as an independent event, causing the system to process a large amount of duplicate or redundant data, which not only increases the system load but also reduces processing efficiency. Especially in database change capture (CDC) and message queue systems, when batch data changes at the same time, how to effectively identify the latest valid data and avoid processing intermediate state data has become an urgent problem to be solved. Existing technologies usually use simple timestamp comparison or full data processing methods, lacking intelligent deduplication mechanisms for high concurrency and multiple change scenarios, and are unable to effectively reduce system resource consumption and improve data processing efficiency.

[0003] Patent application document CN119668900A discloses a message data processing method, device, storage medium and electronic device, which obtains a certain number of target business message data in a specified message queue. These data contain information such as operation type, message identifier, message timestamp, grouping attribute information and deduplication unique identifier. According to the operation type, they are classified into deduplication class and merge class. Deduplication class messages are divided into groups according to grouping attributes, and deduplication is performed according to the deduplication unique identifier, retaining the message with the earliest timestamp. Merge class messages are hedged and deduplicated according to the operation type. The resulting message set is sorted by timestamp and pushed to the business queue. However, this patent cannot completely solve the current technical problems, nor can it meet the requirements of the present invention. Summary of the Invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for intelligent deduplication of data changes and message simplification based on timestamps.

[0005] The method for intelligent deduplication of data changes and message simplification based on timestamps provided by the present invention includes:

[0006] Step 1: Capture database change events in real time and generate change records with timestamps. The change records include source table information, change data identifiers, and timestamps.

[0007] Step 2: Parse and compare timestamps of multiple change records of the same business entity based on a preset timestamp comparison algorithm. The preset timestamp comparison algorithm supports cross-format parsing and priority sorting of decimal values, hexadecimal strings, and ordinary strings.

[0008] Step 3: Group change records according to business keys, store and dynamically update the latest change records in a hash table, retain the latest timestamp data under the same business key, and implement intelligent deduplication of redundant data. The business key is constructed by extracting the values ​​of non-timestamp fields in the change record and concatenating them with preset delimiters to form a unique identifier. The non-timestamp fields include the main table name, primary key ID, and key business fields.

[0009] Step 4: When the data volume exceeds the preset threshold, the data sharding strategy is triggered, evenly dividing the batch data into multiple subtasks. By dynamically allocating thread resources, multi-threaded parallel processing of each shard data is achieved, and the processing results are summarized. At the same time, the shard size is dynamically adjusted according to the data volume to ensure that the data of a single shard does not exceed the preset memory threshold. The consistent hashing algorithm is used to distribute the shards to different processing nodes to avoid data skew.

[0010] Step 5: Merge the deduplicated change data to construct a standardized message body. Choose to send it immediately or delay queue distribution based on the message priority, and control the network transmission load through the message batching mechanism.

[0011] Preferably, the preset timestamp comparison algorithm includes:

[0012] Try to parse the timestamp into a long integer value for size comparison;

[0013] If parsing fails, the timestamp is converted to a large integer after removing the prefix as a hexadecimal string for comparison;

[0014] If the comparison still fails, the timestamps are treated as ordinary strings and compared in lexicographic order;

[0015] Returns the comparison result to prioritize new and old data.

[0016] Preferably, the multi-threaded parallel processing includes: dynamically adjusting the thread pool capacity according to the current CPU and memory resources; binding an exception capture mechanism for each subtask to ensure that the failure of a single task does not affect the overall process; and performing secondary deduplication on the parallel processing results to eliminate duplicate records caused by sharding processing.

[0017] Preferably, the message batching mechanism includes: delayed message processing based on distributed cache, setting expiration time for non-urgent data and temporarily storing it in the cache queue; splitting large batches of messages into sub-messages that meet the middleware processing specifications through message fragmentation strategy; adding a dedicated routing identifier for delayed messages, and realizing timed distribution through the delay queue of the message middleware.

[0018] Preferably, the method further includes a step of intelligently filtering changed data: obtaining a set of processed data from a distributed cache; comparing incremental data with cached data, and retaining only unprocessed newly added or modified fields; and dynamically constructing optimized SQL query statements based on the filtering results to reduce database query load.

[0019] The SQL query optimization includes: automatically adding timestamp filtering conditions to query only changed data within a specified time window; identifying table join operations and optimizing query execution plans; and using a named parameter binding mechanism to prevent SQL injection attacks.

[0020] The timestamp-based data change intelligent deduplication and message simplification system provided by the present invention includes:

[0021] Module M1: Captures database change events in real time and generates change records with timestamps. The change records include source table information, change data identifiers, and timestamps.

[0022] Module M2: parses and compares timestamps of multiple change records of the same business entity based on a preset timestamp comparison algorithm. The preset timestamp comparison algorithm supports cross-format parsing and priority sorting of decimal values, hexadecimal strings, and ordinary strings.

[0023] Module M3: Groups change records according to business keys, stores and dynamically updates the latest change records in a hash table, retains the latest timestamp data under the same business key, and implements intelligent deduplication of redundant data. The business key is constructed by extracting the values ​​of non-timestamp fields in the change record and concatenating them with preset delimiters to form a unique identifier. The non-timestamp fields include the main table name, primary key ID, and key business fields.

[0024] Module M4: When the data volume exceeds a preset threshold, the data sharding strategy is triggered, evenly dividing the batch data into multiple subtasks. By dynamically allocating thread resources, multi-threaded parallel processing of each shard's data is implemented, and the processing results are summarized. At the same time, the shard size is dynamically adjusted according to the data volume to ensure that the data in a single shard does not exceed the preset memory threshold. The consistent hashing algorithm is used to allocate shards to different processing nodes to avoid data skew.

[0025] Module M5: Merges the deduplicated change data, constructs a standardized message body, selects immediate delivery or delayed queue distribution based on message priority, and controls network transmission load through a message batching mechanism.

[0026] Preferably, the preset timestamp comparison algorithm includes:

[0027] Try to parse the timestamp into a long integer value for size comparison;

[0028] If parsing fails, the timestamp is converted to a large integer after removing the prefix as a hexadecimal string for comparison;

[0029] If the comparison still fails, the timestamps are treated as ordinary strings and compared in lexicographic order;

[0030] Returns the comparison result to prioritize new and old data.

[0031] Preferably, the multi-threaded parallel processing includes: dynamically adjusting the thread pool capacity according to the current CPU and memory resources; binding an exception capture mechanism for each subtask to ensure that the failure of a single task does not affect the overall process; and performing secondary deduplication on the parallel processing results to eliminate duplicate records caused by sharding processing.

[0032] Preferably, the message batching mechanism includes: delayed message processing based on distributed cache, setting expiration time for non-urgent data and temporarily storing it in the cache queue; splitting large batches of messages into sub-messages that meet the middleware processing specifications through message fragmentation strategy; adding a dedicated routing identifier for delayed messages, and realizing timed distribution through the delay queue of the message middleware.

[0033] Preferably, it also includes a changed data intelligent screening module: obtains a collection of processed data from the distributed cache; compares the incremental data with the cached data, retaining only the unprocessed new or modified fields; dynamically builds optimized SQL query statements based on the screening results to reduce the database query load;

[0034] The SQL query optimization includes: automatically adding timestamp filtering conditions to query only changed data within a specified time window; identifying table join operations and optimizing query execution plans; and using a named parameter binding mechanism to prevent SQL injection attacks.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) Through the timestamp-based intelligent deduplication mechanism, the present invention can quickly identify and retain the latest valid data when processing large-scale data changes, avoiding duplicate processing and significantly improving data processing efficiency. In actual applications, for data sets containing tens of thousands of records, the processing time can be reduced by more than 70%;

[0037] (2) The use of a multi-threaded parallel processing architecture, combined with a data sharding strategy, enables the system to dynamically adjust the processing method based on the amount of data and available resources, avoiding the performance bottleneck of single-threaded processing and preventing over-allocation of resources, effectively reducing the consumption of system resources such as CPU and memory;

[0038] (3) Through message simplification and intelligent distribution mechanisms, the present invention can merge multiple changes into a single message and implement a delayed processing strategy for non-urgent data, significantly reducing the number of messages transmitted over the network. In high-frequency change scenarios, the network transmission load can be reduced by more than 50%;

[0039] (4) Based on a strict timestamp comparison algorithm, the present invention can accurately identify the order of data changes. Even if the order of data arrival in a distributed system is inconsistent with the actual change order, it can ensure that the final processing result reflects the latest data status, effectively improving the data consistency of the system;

[0040] (5) The present invention adopts a modular design with loose coupling between functional components. Configuration-driven processing logic enables the system to adapt to different business scenarios and data structures. It can support new data sources and processing rules without modifying the code, significantly enhancing the scalability of the system.

[0041] (6) By distinguishing between urgent and non-urgent data changes and implementing different processing strategies, the present invention ensures real-time processing of critical data while smoothing peak loads through a delayed processing mechanism, thereby optimizing the balance between system real-time performance and resource utilization;

[0042] (7) The present invention implements a comprehensive exception handling mechanism, which can perform reasonable downgrade processing in cases of data format abnormalities, network interruptions, etc., avoiding single point failures that cause interruption of the entire processing flow, and improving the stability and reliability of the system under abnormal conditions;

[0043] (8) Standardized data processing processes and message formats reduce the custom development work for different data sources and business scenarios. At the same time, the perfect monitoring and logging mechanism facilitates real-time monitoring of system operation status and troubleshooting, effectively reducing the development and maintenance costs of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0045] Figure 1 This is the overall system architecture diagram;

[0046] Figure 2 is the timestamp comparison algorithm;

[0047] Figure 3 Data grouping and deduplication processing;

[0048] Figure 4 It is a multi-threaded parallel processing architecture;

[0049] Figure 5It is a multi-threaded parallel query process;

[0050] Figure 6 It is a message caching and delayed sending mechanism;

[0051] Figure 7 Manage message batching and queueing processes;

[0052] Figure 8 It is a delayed message processing mechanism;

[0053] Figure 9 To change the data conversion and screening process;

[0054] Figure 10 Optimize strategies for SQL queries;

[0055] Figure 11 Provides an example of system implementation. DETAILED DESCRIPTION

[0056] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0057] Example 1

[0058] The present invention provides a timestamp-based batch data multiple change intelligent deduplication and message simplification method, the specific technical solution is as follows:

[0059] (1) System architecture design

[0060] 1.1 Overall Architecture

[0061] The method of the present invention is applied in a distributed system environment and mainly includes three core components: a data source monitoring layer, a data processing layer, and a message distribution layer. The data source monitoring layer is responsible for capturing database change events; the data processing layer is responsible for data deduplication and message streamlining; and the message distribution layer is responsible for sending processed messages to the target system.

[0062] 1.2 Key Components

[0063] 1. Data change capture component: monitors database changes and generates change records with timestamps;

[0064] 2. Timestamp comparator: compares the newness of data based on timestamp;

[0065] 3. Intelligent deduplication processor: deduplicates data based on business rules and timestamps;

[0066] 4. Message Streamliner: Merge related messages to reduce the number of messages;

[0067] 5. Multi-threaded processing engine: realizes parallel processing of large batches of data.

[0068] (2) Data change capture and recording

[0069] 2.1. Change information structure design

[0070] The present invention designs a dedicated message record structure MessageRecord for storing data change information:

[0071] 1. Record source table information: including the main table name mainTableName and the subtable name subTableName;

[0072] 2. Message metadata: including message topic, offset, and partition;

[0073] 3. Change data identification: Use the ids collection to store all master table IDs involved in the change;

[0074] 4. Change field records: Use the changeColumnRecords mapping table to store key-value pairs, where the key is the keyId monitored by the source table, and the value is the change status and timestamp of the corresponding monitored field.

[0075] 2.2 Timestamp Generation and Management

[0076] 1. Each data change record contains a timestamp to identify the exact time when the change occurred;

[0077] 2. The timestamp can be a TMSTAMP generated by the database system or a timestamp generated by the application system;

[0078] 3. For timestamps in different formats, the present invention provides a unified comparison method that supports comparison of decimal and hexadecimal timestamps.

[0079] (3) Intelligent deduplication mechanism based on timestamp

[0080] 3.1 Timestamp comparison algorithm

[0081] This invention implements a universal timestamp comparison algorithm that can process timestamps in different formats:

[0082] 1. First try to parse the two timestamps into long integer values ​​for comparison;

[0083] 2. If parsing fails, try processing the timestamp as a hexadecimal string, remove the prefix "0x", and convert it to BigInteger for comparison;

[0084] 3. If both of the above methods fail, compare the timestamps as ordinary strings in lexicographic order;

[0085] 4. Returns the comparison result: greater than 0 indicates that the first timestamp is updated, less than 0 indicates that the second timestamp is updated, and equal to 0 indicates that the two timestamps are the same.

[0086] 3.2 Data Grouping and Deduplication

[0087] The present invention adopts a grouping strategy based on business keys to perform intelligent deduplication on multiple changes to the same business entity:

[0088] 1. Build a business key: Traverse each row of data in the result set, extract the values ​​of non-timestamp fields, and concatenate them using a delimiter to form a unique business key;

[0089] 2. Group comparison: Use a hash table to store business keys and corresponding data rows. When a new data row with the same business key is encountered, its timestamp is extracted and compared with the timestamps of the stored data rows.

[0090] 3. Keep the latest record: If the timestamp of the new data row is updated, replace the corresponding record in the hash table; if the timestamp of the stored data row is updated, it remains unchanged;

[0091] 4. Result conversion: After processing all data, the values ​​in the hash table are converted into a list and returned. This deduplicates multiple changes to the same business entity and only retains the latest state.

[0092] (4) Multi-threaded parallel processing of large batches of data

[0093] 4.1 Data Sharding Strategy

[0094] For large-scale data processing scenarios, this paper designs an efficient data sharding strategy:

[0095] 1. Set the batch processing threshold: When the data volume exceeds the preset threshold BATCH_SIZE, multi-threaded processing is triggered;

[0096] 2. Data even sharding: Evenly split large batches of data into multiple small batches, with the size of each batch not exceeding a preset threshold;

[0097] 3. Dynamic thread allocation: Dynamically allocate processing threads based on data volume and system resources to avoid thread resource waste or system overload.

[0098] 4.2 Multi-threaded parallel query and processing

[0099] The present invention implements a multi-threaded parallel processing mechanism based on an asynchronous programming model:

[0100] 1. Task decomposition: Split large batch queries into multiple small batch query tasks, with each task responsible for processing one data shard;

[0101] 2. Parallel execution: Use the thread pool to execute these query tasks in parallel, with each task executed in a separate thread;

[0102] 3. Result aggregation: Use thread-safe collections to collect and merge query results from each thread to ensure data integrity;

[0103] 4. Exception handling: Capture and record exceptions during parallel execution to prevent the failure of a single task from affecting the overall processing flow;

[0104] 5. Performance monitoring: Use a timer to record the execution time of the entire parallel processing process to facilitate performance analysis and optimization;

[0105] 6. Final deduplication: Apply the aforementioned data grouping and deduplication processing algorithms to the aggregated result set to ensure the uniqueness and timeliness of the final result.

[0106] (V) Message Simplification and Intelligent Distribution Mechanism

[0107] 5.1. Message caching and delayed sending

[0108] The present invention designs a message caching mechanism based on distributed cache to reduce the sending of duplicate messages:

[0109] 1. Cache key design: Build a unique cache key based on the observer configuration, table name, and monitoring fields to ensure that caches in different business scenarios do not interfere with each other;

[0110] 2. Change data cache: When a data change message is received, the monitored field values ​​in the change information are extracted and stored in the distributed cache's collection data structure;

[0111] 3. Expiration time setting: Set a reasonable expiration time for cached data based on business configuration to avoid long-term memory resource occupation and ensure that data will not be processed repeatedly within the validity period;

[0112] 4. Cache status check: Before processing a new message, check whether the cache exists. If not, create a new cache and set the expiration time. If it already exists, add the new data directly.

[0113] 5. Performance monitoring: Record the time consumed by cache operations to facilitate system performance monitoring and optimization.

[0114] 5.2 Message Batching and Queue Management

[0115] To avoid message accumulation and system overload, the present invention implements a message batch sending mechanism:

[0116] 1. Dynamic adjustment of queue size: Dynamically determine the queue size for message batch processing based on business type and table relationship configuration;

[0117] 2. Message batch processing: Split large batches of messages into multiple small batches according to the queue size and send them one by one;

[0118] 3. Message body construction: Build a standardized message body based on business rules and data content, including necessary business data and metadata;

[0119] 4. Message header settings: Add header data such as business identification and source information to the message to facilitate message classification and processing by the recipient;

[0120] 5. Message sending: Send the constructed message to the specified topic or queue through the message service, supporting multiple message middleware.

[0121] 5.3 Delayed message processing mechanism

[0122] The present invention designs a special delayed message processing mechanism to handle non-urgent data changes:

[0123] 1. Delay queue configuration: Define the delay queue's switches, routing keys, and other parameters through the configuration file;

[0124] 2. Message classification: Classify messages into normal messages that require immediate processing and non-urgent messages that can be processed later based on business rules;

[0125] 3. Delay message construction: Construct a message body in a specific format for the delayed message, which contains all the necessary information of the original message;

[0126] 4. Message sharding: Segment large quantities of delayed messages to ensure that the size of each delayed message is suitable for processing by the message middleware;

[0127] 5. Secondary message sending: Delayed messages are encapsulated as standard messages and sent to a dedicated delay queue. The delay mechanism of the message middleware ensures that the messages are processed at the appropriate time.

[0128] (6) Intelligent screening mechanism for change data

[0129] 6.1. Change Data Conversion and Filtering

[0130] The present invention implements an intelligent filtering mechanism for changed data to avoid processing known duplicate data:

[0131] 1. Change information parsing: Parses the change information in the message into structured data, supporting case-insensitive field matching;

[0132] 2. Monitoring field extraction: Extract the corresponding change value from the change information based on the configured monitoring field list;

[0133] 3. Cache data acquisition: Obtain the processed data set from the distributed cache for subsequent incremental data identification;

[0134] 4. Incremental data identification: For changed values ​​of array types, remove the elements already in the cache and retain only the actual incremental data;

[0135] 5. Parameter mapping construction: Add the filtered incremental data to the query parameter mapping for subsequent data query operations.

[0136] 6.2 SQL query optimization

[0137] This invention implements intelligent optimization of SQL queries for different query scenarios:

[0138] 1. Dynamic construction of query statements: Dynamically construct SQL query statements based on the query template and monitoring fields configured by the observer;

[0139] 2. Join query identification: Identify whether the query contains table join operations and adopt different query strategies for different situations;

[0140] 3. Add timestamp conditions: For queries that support timestamp filtering, automatically add the condition that the timestamp is greater than the specified value, and sort by timestamp in ascending order;

[0141] 4. Query parameter binding: Use the named parameter binding mechanism to ensure query security and parameter type matching;

[0142] 5. Query result processing: Apply the aforementioned data grouping and deduplication processing algorithms to the query results to ensure that the latest valid data is returned.

[0143] Through the above six technical solutions, the present invention realizes intelligent deduplication and message simplification of batch data with multiple changes based on timestamps, effectively solving the problems of efficiency and accuracy of data processing in large-scale distributed systems.

[0144] Figure 1 The overall architecture of the present invention is illustrated, comprising three core components: the data source monitoring layer, the data processing layer, and the message distribution layer, as well as the data flow relationships between them. The diagram shows that a change event generated by a data source (such as a database) is captured by the monitoring layer, deduplicated and streamlined by the data processing layer, and finally sent to the target system via the message distribution layer.

[0145] Figure 2 This article describes the processing flow of the timestamp comparison algorithm, including three branch paths: attempting long integer comparison, hexadecimal comparison, and string comparison, and finally returning the comparison result.

[0146] Figure 3 The data grouping and deduplication process based on business keys is demonstrated, including four main steps: building business keys, group comparison, retaining the latest records, and result conversion.

[0147] Figure 4 It describes the multi-threaded parallel processing architecture for large amounts of data, showing the process of data sharding, task allocation, parallel execution and result aggregation, as well as the thread pool management mechanism.

[0148] Figure 5 The processing flow of multi-threaded parallel query is presented in detail, including five main steps: task decomposition, parallel execution, result aggregation, exception handling, and final deduplication.

[0149] Figure 6 It demonstrates the message caching mechanism based on distributed cache, including four key links: cache key design, change data caching, expiration time setting and cache status checking.

[0150] Figure 7 This paper describes the process of message batching and queue management, including five main steps: dynamic adjustment of queue size, message batch processing, message body construction, message header setting and message sending.

[0151] Figure 8 The working principle of the delayed message processing mechanism is demonstrated, including four key links: message classification, delayed message construction, message fragmentation and secondary message sending, as well as the interaction process with the message middleware.

[0152] Figure 9 The process of change data conversion and screening is described, including five main steps: change information parsing, monitoring field extraction, cache data acquisition, incremental data identification and parameter mapping construction.

[0153] Figure 10 It demonstrates various strategies for SQL query optimization, including five key steps: dynamic query statement construction, join query identification, timestamp condition addition, query parameter binding, and query result processing.

[0154] Figure 11 The paper shows an implementation example of the present invention in a practical system, including the specific implementation of each component and the interaction between them, as well as the integration method with external systems.

[0155] Example 2: Flink-based database change capture and intelligent processing system

[0156] like Figure 1 As shown, this embodiment of the database change capture and intelligent processing system built based on Apache Flink includes the following steps:

[0157] 1) System initialization configuration: configure data source monitoring parameters, processing rules and target system information;

[0158] 2) Start the Flink task: Submit the configured task to the Flink cluster for execution, and start the task through the command line or REST API;

[0159] 3) Data change capture and processing: The system automatically captures database change events and intelligently processes and distributes them according to configuration rules.

[0160] The core of the system lies in the data change capture and processing process, such as Figure 4 As shown, the following steps are included:

[0161] 3.1) Data Change Capture: Monitor database changes in real time through the Flink CDC connector;

[0162] 3.1.1) Supported data sources include MySQL, SQL Server, MongoDB and other mainstream databases;

[0163] 3.1.2) Capture change events of operation types such as INSERT, UPDATE, and DELETE;

[0164] 3.1.3) Extract the timestamp information of the changed data for subsequent deduplication processing;

[0165] 3.2) Data preprocessing and verification;

[0166] 3.2.1) Parse the change event and convert it into a standard CdcEntity structure;

[0167] 3.2.2) Verify data integrity and filter invalid data;

[0168] 3.2.3) Determine whether the current change needs to be processed based on the configured importance rules;

[0169] 3.3) Intelligent deduplication processing based on timestamp;

[0170] 3.3.1) Find the relevant task configuration based on the table name and business key;

[0171] 3.3.2) Apply timestamp comparison algorithm to identify the age of data;

[0172] 3.3.3) De-duplicate multiple changes to the same business entity and only retain the latest state;

[0173] 3.4) Parallel processing of large batches of data;

[0174] 3.4.1) When the data volume exceeds the preset threshold, multi-threaded parallel processing is triggered;

[0175] 3.4.2) Evenly divide the data into pieces and distribute them to multiple processing threads;

[0176] 3.4.3) Parallel execution of data query and processing tasks;

[0177] 3.4.4) Summarize the processing results of each thread and apply final deduplication;

[0178] 3.5) Message construction and distribution;

[0179] 3.5.1) Construct a standard message format based on the processing results;

[0180] 3.5.2) Set the message header information, including business identification, source, etc.;

[0181] 3.5.3) Choose to send immediately or delay processing based on the message priority;

[0182] 3.5.4) Send messages to the target system in batches.

[0183] Example 3: Enterprise-level data synchronization and real-time analysis platform

[0184] like Figure 11 As shown, this embodiment describes an enterprise-level data synchronization and real-time analysis platform, including the following steps: 1) platform configuration and initialization: configuring data source connection information, synchronization rules and target system;

[0185] 2) Start the monitoring service: Start the data change monitoring service and begin capturing data change events;

[0186] 3) Data processing and distribution: The system automatically processes captured change events and distributes them to the corresponding target systems. The core of the platform lies in the data processing and distribution process, such as Figure 7 and Figure 8 As shown, the following steps are included:

[0187] 3.1) Change data reception and analysis;

[0188] 3.1.1) Receive change events from different data sources;

[0189] 3.1.2) Analyze change events and extract key information;

[0190] 3.1.3) Select the appropriate processing strategy based on the data source type;

[0191] 3.2) Intelligent screening of change data;

[0192] 3.2.1) Parse the change information and extract the monitoring fields;

[0193] 3.2.2) Query the distributed cache to obtain processed data;

[0194] 3.2.3) Identify real incremental data and filter processed data;

[0195] 3.3) SQL query optimization and execution;

[0196] 3.3.1) Dynamically build SQL queries based on configuration;

[0197] 3.3.2) Identify the query type and apply the corresponding optimization strategy;

[0198] 3.3.3) Execute the optimized SQL query to obtain complete data;

[0199] 3.4) Data grouping and deduplication;

[0200] 3.4.1) Construct business keys to group query results;

[0201] 3.4.2) Compare the timestamps of the same set of data and keep the latest record;

[0202] 3.4.3) Convert the processed data into a standard format;

[0203] 3.5) Message classification and sending;

[0204] 3.5.1) Classify messages into normal messages and delayed messages according to business rules;

[0205] 3.5.2) Batch normal messages and send them immediately;

[0206] 3.5.3) Send delayed messages to a dedicated delay queue;

[0207] 3.5.4) Monitor message sending status and handle abnormal situations;

[0208] Example 4: Real-time financial data monitoring and analysis system

[0209] This embodiment describes a real-time data monitoring and analysis system for use in the financial sector, including the following steps: 1) system configuration and initialization: configuring financial data sources, monitoring rules, and analysis models;

[0210] 2) Start the monitoring service: Start the data change monitoring service to begin capturing financial data changes;

[0211] 3) Data processing and analysis: The system automatically processes captured change events and performs real-time analysis and early warning.

[0212] The core of the system lies in the real-time processing and analysis of financial data, which includes the following steps:

[0213] 3.1) Financial data change capture;

[0214] 3.1.1) Connect to the financial database and monitor changes in key tables;

[0215] 3.1.2) Capturing changes in data of financial products such as stocks, bonds, and funds;

[0216] 3.1.3) Extract the change timestamp and operation type;

[0217] 3.2) Data verification and preprocessing;

[0218] 3.2.1) Verify data integrity and validity;

[0219] 3.2.2) Determine the importance of the change based on the configured importance rules;

[0220] 3.2.3) Prioritize changes to important fields;

[0221] 3.3) Real-time data analysis;

[0222] 3.3.1) Apply financial analysis models to calculate key indicators;

[0223] 3.3.2) Compare with historical data to identify abnormal fluctuations;

[0224] 3.3.3) Generate analysis results and warning information;

[0225] 3.4) Multi-level message distribution;

[0226] 3.4.1) Set message priority based on the importance of the analysis results;

[0227] 3.4.2) High priority messages are sent immediately to the monitoring center;

[0228] 3.4.3) General messages are sent after batch processing;

[0229] 3.4.4) Low priority messages are processed through the delay queue;

[0230] 3.5) Early warning and response;

[0231] 3.5.1) The monitoring center receives and processes warning messages;

[0232] 3.5.2) Trigger the corresponding response mechanism according to the warning level;

[0233] 3.5.3) Record warning history and processing results.

[0234] Through the above embodiments, the present invention provides multiple implementation methods of a timestamp-based batch data multiple change intelligent deduplication and message simplification method, which can adapt to the needs of different industries and application scenarios and provide enterprises with efficient and reliable data processing solutions.

[0235] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0236] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for intelligent data change deduplication and message simplification based on timestamps, characterized in that: include: Step 1: Capture database change events in real time and generate change records with timestamps. The change records include source table information, change data identifiers, and timestamps. Step 2: Parse and compare timestamps of multiple change records of the same business entity based on a preset timestamp comparison algorithm. The preset timestamp comparison algorithm supports cross-format parsing and priority sorting of decimal values, hexadecimal strings, and ordinary strings. Step 3: Group change records according to business keys, store and dynamically update the latest change records in a hash table, retain the latest timestamp data under the same business key, and implement intelligent deduplication of redundant data. The business key is constructed by extracting the values ​​of non-timestamp fields in the change record and concatenating them with preset delimiters to form a unique identifier. The non-timestamp fields include the main table name, primary key ID, and key business fields. Step 4: When the data volume exceeds the preset threshold, the data sharding strategy is triggered to evenly divide the batch data into multiple subtasks. By dynamically allocating thread resources, multi-threaded parallel processing of each shard data is achieved, and the processing results are summarized. At the same time, the shard size is dynamically adjusted according to the data volume to ensure that the data of a single shard does not exceed the preset memory threshold. Use consistent hashing algorithm to distribute shards to different processing nodes to avoid data skew; Step 5: Merge the deduplicated change data to construct a standardized message body. Choose to send it immediately or delay queue distribution based on the message priority, and control the network transmission load through the message batching mechanism.

2. The method for intelligent data change deduplication and message simplification based on timestamp according to claim 1, characterized in that: The preset timestamp comparison algorithm includes: Try to parse the timestamp into a long integer value for size comparison; If parsing fails, the timestamp is converted to a large integer after removing the prefix as a hexadecimal string for comparison; If the comparison still fails, the timestamps are treated as ordinary strings and compared in lexicographic order; Returns the comparison result to prioritize new and old data.

3. The method for intelligent data change deduplication and message simplification based on timestamp according to claim 1, characterized in that: The multi-threaded parallel processing includes: dynamically adjusting the thread pool capacity according to the current CPU and memory resources; binding an exception capture mechanism for each subtask to ensure that the failure of a single task does not affect the overall process; and performing secondary deduplication on the parallel processing results to eliminate duplicate records caused by sharding processing.

4. The method for intelligent data change deduplication and message simplification based on timestamp according to claim 1, characterized in that: The message batching mechanism includes: delayed message processing based on distributed cache, setting expiration time for non-urgent data and temporarily storing it in the cache queue; splitting large batches of messages into sub-messages that meet the middleware processing specifications through message fragmentation strategy; adding dedicated routing identifiers for delayed messages, and achieving timed distribution through the message middleware's delay queue.

5. The method for intelligent data change deduplication and message simplification based on timestamp according to claim 1, characterized in that: It also includes a step for intelligent filtering of changed data: obtaining a collection of processed data from the distributed cache; comparing incremental data with cached data, and retaining only unprocessed new or modified fields; Dynamically build optimized SQL query statements based on screening results to reduce database query load; The SQL query optimization includes: automatically adding timestamp filtering conditions to query only changed data within a specified time window; identifying table join operations and optimizing query execution plans; and using a named parameter binding mechanism to prevent SQL injection attacks.

6. A timestamp-based data change intelligent deduplication and message simplification system, characterized in that: include: Module M1: Captures database change events in real time and generates change records with timestamps. The change records include source table information, change data identifiers, and timestamps. Module M2: parses and compares timestamps of multiple change records of the same business entity based on a preset timestamp comparison algorithm. The preset timestamp comparison algorithm supports cross-format parsing and priority sorting of decimal values, hexadecimal strings, and ordinary strings. Module M3: Groups change records according to business keys, stores and dynamically updates the latest change records in a hash table, retains the latest timestamp data under the same business key, and implements intelligent deduplication of redundant data. The business key is constructed by extracting the values ​​of non-timestamp fields in the change record and concatenating them with preset delimiters to form a unique identifier. The non-timestamp fields include the main table name, primary key ID, and key business fields. Module M4: When the data volume exceeds the preset threshold, the data sharding strategy is triggered, evenly dividing the batch data into multiple subtasks. By dynamically allocating thread resources, multi-threaded parallel processing of each shard data is achieved, and the processing results are summarized. At the same time, the shard size is dynamically adjusted according to the data volume to ensure that the data of a single shard does not exceed the preset memory threshold. Use consistent hashing algorithm to distribute shards to different processing nodes to avoid data skew; Module M5: Merges the deduplicated change data, constructs a standardized message body, selects immediate delivery or delayed queue distribution based on message priority, and controls network transmission load through a message batching mechanism.

7. The system for intelligent data change deduplication and message simplification based on timestamp according to claim 6, characterized in that: The preset timestamp comparison algorithm includes: Try to parse the timestamp into a long integer value for size comparison; If parsing fails, the timestamp is converted to a large integer after removing the prefix as a hexadecimal string for comparison; If the comparison still fails, the timestamps are treated as ordinary strings and compared in lexicographic order; Returns the comparison result to prioritize new and old data.

8. The system for intelligent data change deduplication and message simplification based on timestamp according to claim 6, characterized in that: The multi-threaded parallel processing includes: dynamically adjusting the thread pool capacity according to the current CPU and memory resources; binding an exception capture mechanism for each subtask to ensure that the failure of a single task does not affect the overall process; and performing secondary deduplication on the parallel processing results to eliminate duplicate records caused by sharding processing.

9. The system for intelligent data change deduplication and message simplification based on timestamp according to claim 6, characterized in that: The message batching mechanism includes: delayed message processing based on distributed cache, setting expiration time for non-urgent data and temporarily storing it in the cache queue; splitting large batches of messages into sub-messages that meet the middleware processing specifications through message fragmentation strategy; adding dedicated routing identifiers for delayed messages, and achieving timed distribution through the message middleware's delay queue.

10. The system for intelligent data change deduplication and message simplification based on timestamp according to claim 6, characterized in that: It also includes a module for intelligent filtering of changed data: it obtains a collection of processed data from the distributed cache; it compares the incremental data with the cached data and only retains the unprocessed new or modified fields; Dynamically build optimized SQL query statements based on screening results to reduce database query load; The SQL query optimization includes: automatically adding timestamp filtering conditions to query only changed data within a specified time window; identifying table join operations and optimizing query execution plans; and using a named parameter binding mechanism to prevent SQL injection attacks.

Images