A method and apparatus for real-time data stream validation

Abstract

The application discloses a kind of real-time data stream's account verification method and device, comprising: checking and configuring relevant parameters, equipment to real-time data acquisition basic environment, make equipment resource and network support real-time data acquisition;Real-time data acquisition is started on target system server, real-time capture network transaction data, and the transaction request obtained in real time is forwarded to test environment system;Real-time captured transaction data is parsed, and effective key information is extracted by analyzing and processing data;The data obtained in step 3 are layered and combined to carry out account verification;When finding that account data is abnormal or inconsistent, trigger alarm, and record abnormal information.The application introduces real-time data acquisition technology, realizes real-time capture and analysis of transaction data, improves the timeliness and accuracy of transaction verification, improves monitoring efficiency, and does not need to make a lot of modification to existing business system, reduces implementation cost.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and relates to transaction data verification and environmental testing technology, specifically to a real-time data stream tracking verification method and device. Background Technology

[0002] In existing online transaction systems, transaction verification is a crucial step in ensuring the consistency and integrity of transaction data and preventing fraud. Transaction verification requires tracking, recording, comparing, and verifying the entire lifecycle of a transaction from initiation to completion to improve online transaction security. However, traditional transaction verification methods often rely on the internal processing logic of the transaction system. Due to their deep coupling with the business system, they are susceptible to tampering or bypassing. Therefore, traditional transaction verification methods have significant risks in terms of independence, reliability, and real-time performance, necessitating a transaction verification method that can be separated from the trusted domain of the business system and perform data collection and verification from an independent perspective. Summary of the Invention

[0003] To address the aforementioned issues, this invention discloses a real-time data stream tracking verification method and apparatus. It captures transaction data in real time using real-time data acquisition technology and employs a production-like environment system for tracking verification to ensure the authenticity and integrity of the transaction data.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] A real-time data stream-based accounting verification method includes the following steps:

[0006] Step 1: Check and configure the relevant parameters and equipment for the basic environment of real-time data acquisition, so that the equipment resources and network support real-time data acquisition;

[0007] Step 2: Initiate real-time data acquisition in the target system service to capture network transaction data in real time, and forward the acquired transaction requests to the verification system; specifically including:

[0008] Step 2.1: Start the traffic replication tool on the target system server to capture and replicate traffic;

[0009] Step 2.2: Verify on the verification system server whether the received traffic matches the expectations, and check the response and performance of the target system server;

[0010] Step 2.3: Based on the analysis of the deviation between the monitoring results and the expected target, adjust the traffic replication rules and the relevant configuration of the target system server to optimize the traffic replication process;

[0011] Step 3: Analyze the real-time captured transaction data, process the data, and extract relevant key information;

[0012] Step 4: Based on different business needs and according to the importance level of the data, perform hierarchical combination and verification of the data obtained in Step 3.

[0013] Step 5: Generate execution reports based on the different business scenarios configured in Step 4. When abnormal or inconsistent accounting data is found, trigger an alarm, record the abnormal information, and notify relevant personnel.

[0014] Furthermore, the relevant parameters in step 1 include: the type of traffic to be copied, the traffic copying tool, the traffic copying rules, the traffic copying parameters, the filtering rules, and the storage and forwarding targets and policies; the devices include: servers or virtual machines, and the target and verification application systems.

[0015] Furthermore, step 1 also includes: verifying the configuration, testing the real-time data acquisition effect, verifying whether the real-time data acquisition device can acquire traffic normally, and whether the acquired traffic can be forwarded to the verification environment, and verifying the consistency of the data corresponding to the target environment and the verification environment.

[0016] Furthermore, step 2.2 is verified in the following manner:

[0017] (1) After decoding, converting and analyzing the captured data packets, verify that the verification system server can correctly copy the traffic;

[0018] (2) Use packet capture tools to verify whether TCP packets from the online server can be captured in the verification system environment. Analyze the captured packets to detect abnormal behavior or performance bottlenecks in the network.

[0019] Furthermore, step 3 includes the following sub-steps:

[0020] Step 3.1: Preprocess the data sent to the verification environment system in step 2;

[0021] Step 3.2: Based on business needs, set specific goals for data parsing, apply parsing methods to analyze transaction data, and extract key information;

[0022] Step 3.3 verifies the data parsed in Step 3.2 to ensure its accuracy and completeness;

[0023] Step 3.4: Based on the data structure and format, write the corresponding request message processing component.

[0024] Furthermore, step 4 includes the following sub-steps:

[0025] Step 4.1: Configure the tracking module with the corresponding parameters according to different business needs and the level of data consistency requirements, including: verification level, matching relationship configuration, matching type configuration, and matching logic mapping configuration.

[0026] Step 4.2: Based on the data obtained in Step 3.2, perform layered combined accounting verification using at least one accounting tracking mode.

[0027] Furthermore, the aforementioned accounting tracking modes include the following three types:

[0028] Tracking Mode 1: Extract the business return code and key fields from the message, compare the request return message data with the return data in the traffic acquisition, set the corresponding comparison rules, and set the corresponding matching threshold for comparison. When the actual calculation result exceeds the threshold, it will fail; when it is less than the threshold, it will pass.

[0029] Tracking Mode 2: For the overall business dimension, the data consistency of the databases in the real-time data acquisition target system and the real-time data verification system is verified, and the SQL statement is associated with the captured API request that triggered the operation;

[0030] Tracking Mode 3: For data reports generated by the real-time data acquisition target system, the same rules are used to calculate the values ​​of the real-time data verification system data, and then the two sides are fully matched. If they do not match, it indicates that there is a problem.

[0031] This invention also provides a real-time data stream tracking verification device for implementing a real-time data stream tracking verification method, comprising: a real-time data acquisition module, a data parsing module, a tracking verification module, and an early warning module; the real-time data acquisition module is used to capture network transaction data in real time and send the data to the real-time data verification environment according to predetermined rules; the data parsing module is used to process and analyze the requests received by the real-time data verification environment and extract key information; the tracking verification module is used to perform hierarchical tracking verification according to the importance level of the data based on different business needs; the early warning module is used to provide corresponding prompts, alarms, and monitoring reports based on the comparative analysis results of the tracking module.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. By introducing real-time data acquisition technology, transaction data can be captured and analyzed in real time, improving the timeliness and accuracy of transaction verification, and real-time analysis and recording of account change information, thereby improving monitoring efficiency. Moreover, it does not require extensive modifications to existing business systems, thus reducing implementation costs.

[0034] 2. By comparing the data with preset transaction rules or transaction records in the database, comprehensive verification of transaction data is achieved, effectively preventing fraudulent activities.

[0035] 3. An exception rule and alarm module were set up so that relevant personnel can be notified in a timely manner when abnormal transaction data is detected, thereby improving the security of the transaction system. Attached Figure Description

[0036] Figure 1 A schematic diagram of the real-time data stream tracking and verification device provided by the present invention.

[0037] Figure 2 This is a schematic diagram of the overall architecture of the present invention. Detailed Implementation

[0038] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0039] This invention provides a real-time data stream-based account verification device, the overall architecture of which is as follows: Figure 2 As shown, it includes: a real-time data acquisition module, a data parsing module, an accounting verification module, and an early warning module.

[0040] Real-time data acquisition module: mainly used to capture network transaction data in real time, including transaction requests and responses, and then send the data to the real-time data verification system (the real-time data verification system is a production-like system) according to predetermined rules.

[0041] Data parsing module: Processes and analyzes requests received from the real-time data acquisition environment, such as filtering transaction data and extracting valid and key information, such as decimal processing of transaction amounts and formatting of transaction times.

[0042] The transaction verification module primarily implements layered verification based on data importance levels. Transaction verification mode 1: Return message verification. This is the most direct and basic verification method, verifying the authenticity and integrity of transactions. It is suitable for business scenarios with relatively weak data consistency requirements. Transaction verification mode 2: Two-way database verification (supporting daily, weekly, and monthly verification). Based on mode 1 (and also supporting independent verification not bound by mode 1), it supports full comparison and sampling comparison of periodic data to verify the authenticity and integrity of transactions. Integrity mode, this mode is suitable for business scenarios with relatively high requirements for data consistency and security; tracking mode; 3: Supports report dimension verification (supports multiple verification combinations such as data verification between report files and original data via SQL script, file-to-file verification, and data verification between environment A data and real-time data verification environment via SQL script). This mode is based on mode 2 (and also supports independent tracking without being constrained by tracking mode 2). It verifies the data after data transformation. Under the premise of verifying the correctness of the basic data, it verifies the correctness of the data transformation logic. This mode is suitable for business scenarios with high requirements for data consistency and security.

[0043] Early warning module: The main function of this module is to set thresholds for the tracking module, generate corresponding prompts, alarms and monitoring reports based on the comparison and analysis results of the tracking module, and trigger the alarm mechanism when abnormal or inconsistent transaction data is detected, so as to promptly notify relevant personnel for handling.

[0044] This invention provides a real-time data stream-based accounting verification method, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0045] Step 1 involves checking and configuring the basic environment of the real-time data acquisition module, such as setting the network interface for real-time data acquisition and filtering rules. This includes the following sub-steps:

[0046] Step 1.1: Determine the type of traffic to be copied, such as HTTP, TCP, UDP, etc., and determine the proportion and scope of traffic copying.

[0047] Step 1.2: Select a suitable traffic replication tool according to your needs, such as Goreplay, TCPCopy, SD-WAN, etc.

[0048] Step 1.3: Ensure that the device resources support the real-time data acquisition engine, which is capable of capturing and analyzing data packets in real time without interrupting network traffic.

[0049] Step 1.4: Ensure a stable network connection between the real-time data acquisition module and the network, without significantly impacting network performance. To ensure network stability and data accuracy, the early warning module uses network packet capture and other tools to monitor packet loss. If the packet loss rate exceeds an acceptable threshold, an alarm is triggered, and appropriate handling strategies can be set, such as interruption or continued execution.

[0050] Step 1.5: Configure traffic replication rules according to requirements, such as selecting the target system IP address (i.e., the IP address of the target system server), test IP address (i.e., the IP address of the verification system server), port number, etc. This rule configuration supports "exact match": directly specifying one or more explicit numerical conditions, "pattern match": matching regular expressions and wildcards based on patterns / ranges, etc.

[0051] Step 1.6, set traffic replication parameters: Configure the traffic replication ratio to determine the traffic to be replicated, and configure other parameters as needed, such as session persistence and connection timeout.

[0052] Step 1.7, Set up filtering rules: Set up filtering rules as needed to capture and analyze specific network traffic; filtering rules can be configured based on conditions such as IP address, port number, and protocol type.

[0053] Step 1.8: Configure the verification system server or virtual machine resources (optional) to meet the computing power and network bandwidth required for traffic replication. Install and configure the necessary software and dependencies.

[0054] Step 1.9, Configure storage and forwarding targets and policies: Set the forwarding path and forwarding policy of the real-time data acquisition device to ensure that the captured data can be correctly forwarded to the real-time data verification environment.

[0055] Step 1.10, Configure the target system server (optional): Configure the server as a network interface capable of capturing target network traffic. This usually requires setting the network interface to promiscuous mode. Configure the target system server to receive the replicated traffic and ensure that it can handle and respond to this traffic. If necessary, configure load balancing and failover policies for the target system server.

[0056] Step 1.11, Verify Configuration: After configuration, verify whether the real-time data acquisition device can correctly capture and analyze network data; check whether the configuration items such as real-time data acquisition interface, mirror port, and filtering rules are correct.

[0057] Step 1.12, Test the real-time data acquisition effect: Test the data acquisition effect of the real-time data acquisition device by simulating network traffic or actual network activity; check whether the real-time data acquisition device can capture and report abnormal traffic or potential security threats in a timely manner. At the same time, preliminarily verify whether the real-time data verification environment can receive the captured data, and verify that the verification environment and the target environment have the same return data.

[0058] Step 2: Activate the real-time data acquisition module to capture network transaction data in real time. Simultaneously, forward the acquired transaction requests to the real-time data environment / verification environment system (the real-time data verification environment system is a production-like system). Faced with real-time correlation and comparison of massive amounts of data, the real-time data acquisition process captures a vast number of discrete data packets. To accurately and in real-time correlate all fragments of the same transaction, real-time data clock synchronization and global time series are required: deploy high-precision clock synchronization on all production servers and real-time data acquisition devices, assign a reliable timestamp to each captured event, construct a global time series view, and resolve time series dependencies across systems and requests. Specifically, this includes the following sub-steps:

[0059] Step 2.1, based on the configuration information in Steps 1.3, 1.4, 1.5, 1.6, 1.7, 1.9, and 1.12, ensure that the network connection between the target system server and the verification system service network is stable and reliable, start the real-time data acquisition module service on the target system server, and begin capturing and copying traffic, monitor the traffic copying process, and ensure that the traffic is correctly copied and sent to the target system server.

[0060] Step 2.2: After completing Step 2.1, verify the received traffic on the verification system server to ensure it matches expectations and check the server's response and performance. Verification Method 1: Use the data analysis module to decode (supports national cryptographic algorithms), convert (e.g., one-click parsing of 8583 and SOAP messages, one-click encoding conversion), and analyze (basic message information) the captured data packets to confirm the detailed network communication information, such as source IP address, destination IP address, source port number, destination port number, and data request content, to ensure they match the expected settings and can correctly handle the copied traffic. Verification Method 2: Use a packet capture tool (e.g., tcpdump) in the real-time data verification environment to verify whether TCP packets from the online server can be captured. By analyzing the captured packets, abnormal network behavior or performance bottlenecks can be detected. Based on business needs, further process and analyze the captured data packets by writing custom scripts (Python, Shell, BeanShell, etc.) or programs (Java, C++, .NET). Ensure the correctness of the information obtained from the real-time data acquisition environment and maintain data consistency between the real-time data acquisition environment and the verification environment. Both verification methods are performed simultaneously.

[0061] Step 2.3: Based on the deviation analysis between the monitoring results and the expected target (e.g., the proportion of traffic replication, session persistence, connection timeout), adjust the traffic replication rules and the relevant configuration of the target system server to optimize the traffic replication process, so as to ensure data consistency and guarantee data processing efficiency.

[0062] Step 3 involves using the data parsing module to analyze and process the captured transaction data, extracting key information. This includes the following sub-steps:

[0063] Step 3.1 involves preprocessing the data from the environment tested in Step 2, including deduplication, filling in missing values, and error correction, to improve data quality and consistency.

[0064] Step 3.2: Based on business requirements, set specific data parsing objectives and select the corresponding parsing method (this parsing method is custom-encapsulated into a component via a plugin and supports hot-reloading application capabilities) to parse the transaction data and extract key information. This includes extracting specific fields, converting data formats, and calculating derived indicators. This parsing component is based on a dynamic rule-matching message preprocessing framework. This framework adopts a modular design, abstracting a unified message reader (ABMessageReader), which is integrated into the Spring Batch batch processing environment and supports multiple running modes (normal sending, error retransmission, continuation of running with a specified serial number, sending with a specified transaction code, etc.). Its technical advantages lie in the following aspects: Through configurable regular expressions and logical inversion mechanisms, it achieves multi-dimensional intelligent diagnosis of historical message results (including transaction codes, return codes, and error messages), and makes flexible resend decisions based on the initial, final, or all historical records; the framework has a built-in unified data mapping and state management, which templates the message parsing, result comparison, and resend rule evaluation process, significantly improving the automation, maintainability, and rule adaptability of message resend processing, and effectively solving the problem of rigid message resend strategies and redundant development caused by complex error scenarios in financial trading systems.

[0065] Step 3.3 involves performing completion, consistency, and availability verification on the data parsed in Steps 3.1 and 3.2 to ensure the accuracy and integrity of the data. This may include data consistency checks, range verification, etc.

[0066] Step 3.4: Based on the data structure and format, write the corresponding request message processing component (this parsing method is custom-encapsulated into a component via a plugin and supports hot-reloading application capabilities), mainly to prepare for the successful forwarding of request messages in the verification environment. This may involve operations such as field matching, pattern recognition, and condition filtering.

[0067] Step 4 involves verification through the transaction tracking module. This step employs efficient verification algorithms and comparison mechanisms to achieve rapid verification of transaction data. Compared to traditional methods, the main advantages are reflected in four aspects: 1. Data source and collection method shifts from "active query" to "passive mirroring." Traditional method: The verification module actively pulls data generated by the business system through internal interfaces (such as database query API, log file reading, message queue subscription); Real-time data verification method: The verification module processes raw traffic (request / response messages, database access streams) passively mirrored from the network link. This data represents the "raw facts" of business interactions, independent of the business system's internal logic and storage; 2. Real-time verification and performance impact: From "asynchronous batch" to "near real-time stream." Traditional method: Daily and weekly transaction tracking are usually batch operations performed during off-peak periods, with controllable impact on the online system; Real-time data verification method: It has a natural advantage in achieving near real-time (second-level / millisecond-level) verification. Data is continuously input in the form of a stream, allowing for real-time correlation, comparison, and alarm triggering; 3. Root cause and credibility of alarms: From "system status alarms" to "factual inconsistency alarms," ​​traditional methods: alarms are based on logs or indicators generated by the business system itself (such as "reconciliation discrepancies"), and their authenticity depends on the system itself; real-time data verification methods: alarms are directly based on "observational facts not matching system status"; 4. Data alignment and association keys: From "business primary keys" to "network trace association," traditional methods: usually directly use transaction serial numbers, order numbers, etc. provided by the business system as association keys for data comparison, which is simple and direct; real-time data verification methods: initially, discrete network data packets are obtained. It is necessary to innovatively restore business semantics from network traffic. For return message verification (mode 1), it is necessary to extract business return codes and key fields from HTTP response messages; for database two-way verification (mode 2), the challenge is the greatest: it is necessary to correlate an INSERT or UPDATE SQL statement with the previously captured API request that triggered the operation. This might require analyzing sessions, transaction IDs, or performing intelligent pattern matching within a time window, which is far more technically complex than directly querying the database. For report dimension validation (mode 3), the "raw data" in the real-time data acquisition environment is itself cleaned and structured data from the traffic, rather than a copy of the business database. When executing SQL scripts for comparison, its data source is already an independent "fact data lake".Supports multiple verification rules and conditions, which can be flexibly configured according to business needs (mainly implementing layered verification based on data importance level; Tracking Mode 1: Return Message Verification: This mode is the most direct and basic verification method, verifying the authenticity and integrity of transactions. It is suitable for business scenarios with weak data consistency requirements. Tracking Mode 2: Database Two-Way Verification (supports daily, weekly, and monthly tracking, etc.), based on Mode 1 (also supports independent tracking without being constrained by Tracking Mode 1). It supports full comparison and sampling comparison of periodic data to verify the authenticity and integrity of transactions. This mode is suitable for business scenarios with relatively high requirements for data consistency and security. Tracking Mode 3: Supports report dimension verification (supports multiple verification combinations such as data verification between report files and original data via SQL scripts, file-to-file verification, and data verification between environment A and real-time data acquisition environment via SQL scripts). This mode is based on Mode 2 (and also supports independent tracking without being constrained by Tracking Mode 2). It verifies the transformed data, verifying the correctness of the data transformation logic while ensuring the basic data is correct. This mode is suitable for business scenarios with high requirements for data consistency and security. Intelligent alarm mechanisms and processing flows can promptly detect and handle abnormal transaction data. Specifically, it includes the following sub-steps:

[0068] Step 4.1: Based on different business needs and the level of data consistency requirements, configure the corresponding parameters for the tracking module, such as verification level, matching relationship configuration (e.g., data transcoding, field extraction rules, etc.), matching type configuration (e.g., exact matching, fuzzy / range matching, existence matching, set inclusion matching, etc.), matching logic mapping configuration (e.g., return message verification, database two-way verification, report dimension verification, etc.), etc.

[0069] Based on the matching level and the three main tracking modes in step 4, the device supports three categories (3+3+1) according to the data consistency requirement level: Level I, Level II, and Level III. Level I includes: Tracking Mode 1, Tracking Mode 2, and Tracking Mode 3; Level II includes: Tracking Mode 1 + Tracking Mode 2, Tracking Mode 1 + Tracking Mode 3, and Tracking Mode 2 + Tracking Mode 3; and Level III includes: Tracking Mode 1 + Tracking Mode 2 + Tracking Mode 3.

[0070] Key technical points of Tracking Accounts Mode 1: This tracking account management mode primarily verifies data consistency at a single point / single business dimension, that is, it verifies data consistency based on real-time returned messages. This mode supports parsing any data message format (such as XML, SOAP, JSON, SOPA, 8583, etc.) and supports any combination of verification conditions in the messages. Main technical means: It achieves high tracking account management performance through a multi-threaded + producer-consumer model, employing a caching + independent memory isolation strategy, while simultaneously using a real-time tracking account comparison strategy to effectively monitor and track data consistency in real time.

[0071] Key technical points of Tracking Accounts Mode 2: This mode primarily verifies data consistency across the overall business scope. It supports verification based on files (.txt, .csv, etc.) and various databases (MySQL, PG, Oracle, Hive, Hadoop, and domestic databases such as OceanBase, PolarDB, TiDB, Kingbase, openGauss, DM, GBASE, GaussDB, GoldenDB, and TDSQL). To ensure compatibility with domestic environments, the device supports three modes: entirely based on a file database (RocksDB), file-based, file-database (RocksDB) combined with memory, and entirely based on memory comparison. It also supports unidirectional comparison (left-to-right, right-to-left) and bidirectional comparison (verification of both sides). Furthermore, the device supports implementing this tracking account mode through task management.

[0072] Key technical points of Tracking Accounts Mode 3: This mode primarily verifies the consistency of data after logical transformation of the overall business process (mainly applied to business scenarios such as regulatory reporting, daily closing, and monthly closing). This tracking mode is based on report parsing (XML, HTML, CSV, etc.) and image recognition (OCR table parsing) technologies, combined with custom script parsing technology to achieve bidirectional tracking based on files and custom scripts. Furthermore, the device supports implementing this tracking mode through task management.

[0073] Step 4.2: Based on the data processed in Step 3.2, this step supports multi-dimensional analysis and comparison (two-sided return result dimension, data result dimension, report dimension, etc.). For example, in Tracking Mode 1 (verifying data consistency for a single point / single business dimension): the returned message data is compared with the returned data from traffic acquisition. Appropriate comparison rules are set, along with a matching threshold (matching degree > 80%). If the actual calculated result is > 80%, it indicates a failure; if it is <= 80%, it indicates a success. For example, in Tracking Mode 2: the logic for verifying data consistency across the overall business dimension is as follows: It can be compared from the business processing result dimension. For instance, in a bank transaction, during execution, a new account is opened with a current balance of 0 yuan. A 100 yuan recharge is made, and simultaneously, an 80 yuan withdrawal / transfer is made. Theoretically, the account balance should be 20 yuan. In this case, we can use a data comparison tool, combined with the specific business logic model, to write a corresponding script to verify the account balance in both environments. If System A (the target system for real-time data acquisition) queries an account balance of 20 yuan, but System B (the tested system for real-time data verification) queries an account balance that is not equal to 20 yuan, it indicates an error in the processing logic of the tested system. For example, in Account Tracking Mode 3: Report Dimension Processing Logic, when using the data report generated by System A, the same rules are applied to calculate the values ​​of the data in System B, and then a complete match is performed between the two. If a mismatch occurs, it indicates a problem. Based on this dimension comparison, the system supports any combination of two strategies: file database-based comparison and memory-based comparison.

[0074] Step 5: Generate relevant execution reports based on the different business scenarios configured in Step 4. When abnormal or inconsistent accounting data is detected, trigger the alarm module, record the anomaly information, and notify relevant personnel. This includes the following sub-steps:

[0075] Step 5.1: Set up monitoring rules. In the accounting system, set up reasonable monitoring rules based on business needs and the characteristics of transaction data. These rules may include transaction amount thresholds, transaction frequency, and identity verification of both parties involved in the transaction.

[0076] Step 5.2: Real-time detection data, real-time monitoring of accounting processing data, comparison with preset monitoring rules, and detection of any anomalies or inconsistencies.

[0077] Step 5.3: Anomaly Detection. Once anomalies or inconsistencies are detected in the accounting data, the system immediately determines whether to trigger the alarm module. This is typically based on factors such as the type and severity of the anomaly and its impact on business operations.

[0078] Step 5.4: Generate relevant accounting reports automatically or manually on a regular basis according to business needs.

[0079] Step 5.5: Based on the nature and severity of the anomaly, the system should identify the relevant personnel who need to be notified, such as business managers, auditors, and technical support personnel.

[0080] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A real-time data stream-based accounting verification method, characterized in that, Includes the following steps: Step 1: Check and configure the relevant parameters and equipment of the basic environment for real-time data acquisition, so that the equipment resources and network support real-time data acquisition; Step 2: Start real-time data acquisition on the target system server to capture network transaction data in real time, and forward the acquired transaction requests to the verification environment system. Specifically, it includes: Step 2.1: Start the traffic replication tool on the target system server to capture and replicate traffic; Step 2.2: Verify on the verification system server whether the received traffic matches the expectations, and check the response and performance of the target system server; Step 2.3: Based on the analysis of the deviation between the monitoring results and the expected target, adjust the traffic replication rules and the relevant configuration of the target system server to optimize the traffic replication process; Step 3: Analyze the real-time captured transaction data, process the data, and extract relevant key information; Step 4: Based on different business needs and according to the importance level of the data, perform hierarchical combination and verification of the data obtained in Step 3. Step 5: Generate execution reports based on the different business scenarios configured in Step 4. When abnormal or inconsistent accounting data is found, trigger an alarm, record the abnormal information, and notify relevant personnel.

2. The real-time data stream tracking verification method according to claim 1, characterized in that, The relevant parameters in step 1 include: the type of traffic to be copied, the traffic copying tool, the traffic copying rules, the traffic copying parameters, the filtering rules, and the storage and forwarding targets and policies; the devices include: servers or virtual machines, and the target and verification application systems.

3. The real-time data stream tracking verification method according to claim 1, characterized in that, Step 1 further includes: verifying the configuration, testing the real-time data acquisition effect, verifying whether the real-time data acquisition device can acquire traffic normally, and whether the acquired traffic can be forwarded to the verification environment, and verifying the consistency of the data corresponding to the target environment and the verification environment.

4. The real-time data stream tracking verification method according to claim 1, characterized in that, Step 2.2 is verified in the following way: (1) After decoding, converting and analyzing the captured data packets, verify that the verification system server can correctly copy the traffic; (2) Use packet capture tools to verify whether TCP packets from the online server can be captured in the verification environment. Analyze the captured packets to detect abnormal behavior or performance bottlenecks in the network.

5. The real-time data stream tracking verification method according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Preprocess the data sent to the verification environment system in step 2; Step 3.2: Based on business needs, set specific objectives for data parsing, parse the transaction data, and extract key information; The parsing function is encapsulated as a component, using an evaluation process template. The process includes: message parsing, result comparison, and retransmission rules. Step 3.3 verifies the data parsed in Step 3.2 to ensure its accuracy and completeness; Step 3.4: Based on the data structure and format, write the corresponding request message processing component.

6. The real-time data stream tracking verification method according to claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Configure the tracking module with the corresponding parameters according to different business needs and the level of data consistency requirements, including: verification level, matching relationship configuration, matching type configuration, and matching logic mapping configuration. Step 4.2: Based on the data obtained in Step 3.2, perform layered combined accounting verification using at least one accounting tracking mode.

7. The real-time data stream tracking verification method according to claim 1, characterized in that, The accounting tracking modes include the following three: Tracking Mode 1: Extract the business return code and key fields from the message, compare the request return message data with the return data in the traffic acquisition, set the corresponding comparison rules, and set the corresponding matching threshold for comparison. When the actual calculation result exceeds the threshold, it will fail; when it is less than the threshold, it will pass. Tracking Mode 2: For the overall business dimension, the data consistency of the databases in the real-time data acquisition target system and the real-time data verification system is verified, and the SQL statement is associated with the captured API request that triggered the operation; Tracking Mode 3: For data reports generated by the real-time data acquisition target system, the same rules are used to calculate the values ​​of the real-time data verification system data, and then the two sides are fully matched. If they do not match, it indicates that there is a problem.

8. A real-time data stream tracking and verification device, characterized in that, The method for tracking and verifying real-time data streams according to any one of claims 1-7 comprises: a real-time data acquisition module, a data parsing module, a tracking and verification module, and an early warning module; the real-time data acquisition module is used to capture network transaction data in real time and send the data to the real-time data verification environment according to predetermined rules; the data parsing module is used to process and analyze the requests received by the real-time data verification environment and extract key information; the tracking and verification module is used to perform hierarchical tracking and verification according to the importance level of the data based on different business needs; and the early warning module is used to generate corresponding prompts, alarms, and monitoring reports based on the comparative analysis results of the tracking and verification module.

Images