Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a program product, and the method comprises the steps: converting to-be-processed data generated by a business system into a standardized check event, and uploading the standardized check event to a check event queue; monitoring the checking event queue, searching an associated target checking task according to the standardized checking event, and sending the target checking task to a task delay queue according to the dynamic delay time corresponding to the target checking task; monitoring the task delay queue, querying and processing the data of each check data source according to the configuration information of the target check task, and executing the check script through dynamic compiling to complete multi-source data consistency check; and correcting the check data source according to a data check result and a data correction script in the configuration information of the target check task, and carrying out early warning processing on a data abnormal condition. The invention provides a closed-loop checking system integrating data detection, restoration and early warning, which can quickly realize checking and correction of multi-type source data.

Description

Technical Field

[0001] This application relates to the field of distributed system data processing technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and program product applied to a near real-time verification platform. Background Technology

[0002] As enterprises deepen their digital transformation, business systems are showing a trend towards distributed and microservice architectures. In this architecture, a complete business process often spans multiple independent business systems, which interact with each other through asynchronous messaging, database changes, and service calls. Due to factors such as network latency, system failures, and concurrency conflicts, data inconsistencies between different systems are easily generated. Traditional solutions suffer from several technical limitations. For example, using nighttime batch jobs for data verification means that by the time problems are discovered, the business impact has already lasted for several hours, lacking real-time performance. Existing verification systems are typically triggered by a single event source (such as database changes or message queues), making them unsuitable for complex multi-system collaboration scenarios. Using a uniform latency time for verification cannot be dynamically adjusted according to business characteristics, easily leading to false positives or false negatives. The lack of a unified data acquisition framework makes it difficult to support the flexible acquisition of multi-source data such as SQL queries, API calls, and event data. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium, and program product that can respond in real time, has rich triggering mechanisms, and has rich scalability to address the above-mentioned technical problems.

[0004] Firstly, this application provides a data processing method using a near real-time verification platform, the method comprising:

[0005] The pending data generated by the business system is transformed into standardized verification events, and the standardized verification events are uploaded to the verification event queue; wherein, the pending data includes business logs, business data, and business messages;

[0006] Monitor the verification event queue, find the associated target verification task based on the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task;

[0007] The task delay queue is monitored, and the data of each verification data source is queried and processed according to the configuration information of the target verification task. The verification script is dynamically compiled and executed to complete the multi-source data consistency verification.

[0008] The verification data source is corrected based on the data verification results and the data correction script in the configuration information of the target verification task, and an early warning is issued for data anomalies.

[0009] In one embodiment, the step of converting the pending data generated by the business system into standardized verification events includes:

[0010] The system uses a Python script to parse MySQL binlog to obtain business data, parses business logs using a custom delimiter, obtains business messages by listening to a business message queue, and sends the business data, business logs, and business messages to a Kafka message queue.

[0011] The target data that meets the criteria is filtered by a screening script, and the filtered target data is transformed into standardized verification events by a transformation script.

[0012] The standardized verification events are stored in the verification event configuration table and cached in Redis, and simultaneously sent to the verification event queue.

[0013] In one embodiment, the step of finding the associated target verification task based on the standardized verification event and sending the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task includes:

[0014] The event-task association mapping table is queried based on the event code of the standardized verification event. The event-task association mapping table is a key-value pair structure, where the key is the event code of the standardized verification time and the value is the task list of the associated target verification task.

[0015] The standardized verification events are written to the verification event record table, which includes event ID, event code, event attributes, and write time.

[0016] The configuration information of the target verification task is queried according to the event-task association mapping table to obtain the dynamic delay time corresponding to each target verification task;

[0017] According to the dynamic delay time, each target verification task is sent to the task delay queue.

[0018] In one embodiment, the types of data sources to be verified include verification event data sources, application system data sources, and database data sources;

[0019] The step of querying and processing data from each verification data source according to the configuration of the target verification task includes:

[0020] For the data source of the verification event, a preset data processing script is executed to perform secondary processing on the raw event data;

[0021] For the data source of the application system, generate interface input parameters and initiate interface calls. After obtaining the returned data, execute the preset data processing script for processing.

[0022] For a database data source, execute a preset SQL script to query the data, and process the query results using a preset data processing script.

[0023] In one embodiment, the step of executing the verification script through dynamic compilation includes:

[0024] The text content of the verification script is embedded into a predefined Java class template to generate complete and compilable Java source code;

[0025] The Java source code is compiled into bytecode by calling the Java compiler, and the bytecode is loaded into the currently running Java Virtual Machine by a custom class loader;

[0026] The verification processing object is instantiated through reflection so that the corresponding verification logic can be called and executed.

[0027] In one embodiment, the configuration information of the target verification task includes a task name, a verification trigger event, a dynamic delay time, a verification data source type, a data processing script, and a verification script. The method further includes:

[0028] The target verification task is configured through a visual interface to obtain the configuration information, which is then stored in the task configuration table.

[0029] Secondly, this application also provides a data processing apparatus that employs a near real-time verification platform, the apparatus comprising:

[0030] The event receiving module is used to convert the pending data generated by the business system into standardized verification events and upload the standardized verification events to the verification event queue; wherein, the pending data includes business logs, business data and business messages;

[0031] The event handling module is used to listen to the verification event queue, find the associated target verification task according to the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0032] The task processing module is used to monitor the task delay queue, query and process the data of each verification data source according to the configuration information of the target verification task, and complete the multi-source data consistency verification by dynamically compiling and executing the verification script.

[0033] The correction and early warning module is used to correct the data source based on the data verification results and the data correction script in the configuration information of the target verification task, and to provide early warning for data anomalies.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data processing method described in the first aspect.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data processing method described in the first aspect.

[0036] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data processing method described in the first aspect.

[0037] In summary, this application proposes a data processing method, apparatus, computer equipment, storage medium, and program product, comprising: converting data to be processed generated by a business system into standardized verification events and uploading them to a verification event queue; monitoring the verification event queue, searching for associated target verification tasks based on the standardized verification events, and sending the target verification tasks to a task delay queue according to the dynamic delay time corresponding to the target verification tasks; monitoring the task delay queue, querying and processing data from various verification data sources based on the configuration information of the target verification tasks, and completing multi-source data consistency verification by dynamically compiling and executing verification scripts; correcting the verification data sources based on the data verification results and the data correction scripts in the configuration information of the target verification tasks, and providing early warning processing for data anomalies. This application provides a closed-loop verification system integrating data detection, repair, and early warning, which can quickly realize the verification and correction of multi-type source data. Attached Figure Description

[0038] Figure 1 This is an application environment diagram of a data processing method in one embodiment;

[0039] Figure 2 This is an application environment diagram of the data processing method in another embodiment;

[0040] Figure 3 This is a flowchart illustrating a data processing method in one embodiment;

[0041] Figure 4 A schematic diagram of a visual interface for verifying event configuration information in one embodiment;

[0042] Figure 5This is a schematic diagram of a visual interface for the configuration information of the target verification task in one embodiment;

[0043] Figure 6 This is a schematic diagram of the structure of the event task associated cache in one embodiment;

[0044] Figure 7 This is a schematic diagram of the visualization interface for the configuration information of the target verification task in another embodiment;

[0045] Figure 8 This is a schematic diagram of the near real-time verification platform in one embodiment;

[0046] Figure 9 This is a structural block diagram of a data processing device in one embodiment;

[0047] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0050] The data processing method provided in this application embodiment can be applied to, for example... Figure 2 The near real-time verification platform shown can be installed on the aforementioned terminal or server to perform detection, verification, and correction processing on multiple types of data sources in the business system. In this embodiment, the near real-time verification platform includes an event receiver, an event processor, a task processor, and a correction early warning device.

[0051] In this embodiment, the event receiver includes a three-source data processing unit and an event transformation unit. The three-source data processing unit is responsible for collecting business logs, business data, and business messages from the business system, obtaining the data through specialized parsing methods, and sending it to the Kafka message queue. The event transformation unit transforms the raw data into standardized verification events through filtering and transformation scripts, completing the unified formatting of the data, and realizing the storage and caching of events.

[0052] The event handler consists of an association query unit, an event storage unit, and a delay scheduling unit. The association query unit quickly matches associated verification tasks using event codes; the event storage unit persistently stores verification events for easy traceability; and the delay scheduling unit sends verification tasks to a task delay queue based on the dynamic delay configuration of the tasks, ensuring the accuracy of verification timing.

[0053] The task processor includes a data source configuration unit, a data acquisition unit, a dynamic execution unit, and a load balancing unit. The data source configuration unit stores configuration information for various data sources; the data acquisition unit collects and processes multi-source data according to the configuration; the dynamic execution unit executes data processing scripts and verification scripts through a dynamic compilation mechanism; and the load balancing unit ensures the stability and efficiency of data source calls by the application system.

[0054] The correction and early warning system comprises a correction configuration unit, a data correction unit, and an early warning unit. The correction configuration unit stores correction configuration information for different types of data sources. The data correction unit executes correction scripts to automatically repair data. The early warning unit pushes tiered alarm information through multiple channels, forming a complete early warning mechanism.

[0055] Based on the aforementioned near real-time verification platform, the efficient execution of multi-source data verification and correction logic can be ensured through the collaborative work of four core modules: event receiver, event processor, task processor, and correction early warning system.

[0056] In one embodiment, such as Figure 3 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the application environment in [the document] as an example, the following steps are included:

[0057] S301 transforms the pending data generated by the business system into standardized verification events and uploads the standardized verification events to the verification event queue; the pending data includes business logs, business data, and business messages.

[0058] In this embodiment, the business system generates a large amount of business processing data during daily business operations, and this data is divided into various data types. Business processing data includes business logs, business data (database binlog), and business messages. In related technologies, a single data source triggers the verification logic for the aforementioned business processing data. That is, when business data in the database changes or business messages are updated in the message queue, a verification task for the business data or business information is triggered. This triggering mechanism only supports one event source—database change or message queue—and cannot cover complex data streams such as business logs and API calls.

[0059] In this embodiment, the three-source unified event transformation model is used to link the dynamic script engine and combine the filtering script and the transformation script to realize the transformation of the data to be processed into standardized verification events.

[0060] In practical applications, standardized verification events can be achieved through methods such as... Figure 4 The visual interface shown completes the information configuration. The configuration information for standardized verification events includes the event name, event source, filtering script, and transformation script. The event source is categorized into business data, business logs, and business messages. The filtering script is used to filter out messages that meet the criteria; the rule engine executes the script to select messages that meet the conditions. The transformation script is used to convert the messages into verification events.

[0061] After completing the information configuration for standardized verification times, the configuration information for standardized verification events can be stored in the verification event configuration table. The structure of the verification event configuration table is shown in Table 1 below:

[0062] Table 1

[0063]

[0064] In practical applications, the configuration information for standardized verification events can also be stored in a Redis cache to ensure that the configuration information can be retrieved instantly and accurately during the execution of the verification logic. Simultaneously, standardized verification events are also uploaded to the verification event queue so that subsequent event handlers can execute the corresponding steps.

[0065] S302 listens to the verification event queue, finds the associated target verification task based on the standardized verification event, and sends the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0066] In this embodiment, by listening to the verification event queue and finding the associated target verification task based on the standardized verification events, the verification task that the platform needs to execute can be determined, thereby realizing the atomicity capability of the verification platform.

[0067] In practical applications, the verification task can be carried out in advance by technical personnel through methods such as... Figure 5 The visualization interface shown is used for information configuration. In one embodiment, the configuration information of the target verification task includes the task name, verification trigger event, dynamic delay time, verification data source type, data processing script, and verification script. The data processing method further includes: configuring the target verification task through the visualization interface to obtain configuration information, and storing the configuration information in the task configuration table.

[0068] In this embodiment, the verification trigger event corresponds to the standardized verification event generated in the previous embodiment, representing the starting condition for the verification task. The delay time represents the delay period for task verification. For example, in the accounting entry verification task, since the generation of accounting entries after loan completion is asynchronous, a 2-minute delay is required to avoid verification noise. A single verification event can correspond to multiple verification tasks. It should be noted that in this embodiment, the delay time of the target verification task can dynamically change according to the task chain. If the task chain corresponding to the verification event is task 1-task 2, then the delay time of task 1 is the first value. If the task chain corresponding to the verification event is task 3-task 1-task 2, then the delay time of task 1 will dynamically change to the second value. The first value and the second value are different values. It should be noted that the specific value of the dynamic delay time can be customized according to the needs of the actual application scenario, and is not specifically limited here.

[0069] The data source type verification is used to configure the data sources participating in the verification. The data processing method provided in this embodiment supports multi-data source verification. Different types of data sources support different configuration methods, so that differentiated verification can be achieved when integrated into the same verification platform. Users can write custom data verification scripts through a visual interface, or the system can automatically import data verification scripts corresponding to the data source type. The data verification script is used to verify multiple data sources.

[0070] The task configuration table structure for the verification task is shown in Table 2 below:

[0071] Table 2

[0072]

[0073] In this embodiment, after the information configuration for the verification task is completed, the event task association cache is automatically updated. The event task association cache is as follows: Figure 6 As shown, the event-task associated cache adopts a key-value structure, where the key represents the event code and the value represents the task list.

[0074] In this embodiment, the event handler listens to the verification event message queue, then queries the event task association cache based on the event code to quickly locate the task list associated with the event. When a task exists in the task list, the event is written to the database and recorded in the verification event record table. The structure of the verification event record table is shown in Table 3 below:

[0075] Table 3

[0076]

[0077] Among them, event_id represents a unique ID randomly generated when the standardized event is generated, event_code represents the code generated when the standardized event is configured, one event_code can correspond to multiple event_ids, and event_attribute represents all the data of the standardized event. Event attributes will be used when the event data is used during subsequent verification.

[0078] In this embodiment, all target verification tasks in the list are processed sequentially, the task configuration table is queried to obtain the task delay time, and then the triggered task is sent to the verification task delay queue according to the delay time for subsequent task processors to process.

[0079] S303 is a monitoring task delay queue. It queries and processes data from each verification data source based on the configuration information of the target verification task, and executes the verification script through dynamic compilation to complete the multi-source data consistency verification.

[0080] In this embodiment, multiple verification data sources can be set during task configuration. Each type of data source has a distinct configuration method. Therefore, during task configuration, data for each verification data source needs to be stored simultaneously. Each data source record is associated with the task configuration table via task_code. The types of verification data sources include verification event data sources, application system data sources, and database data sources.

[0081] In this embodiment, the event data source is associated with the task configuration table through task_event_source, the application system data source is associated with the task configuration table through task_system_source, and the database data source is associated with the task configuration table through task_database_source.

[0082] In this embodiment, the task processor monitors the task delay queue. When it detects a target verification task that needs processing, it triggers processing for that target verification task. First, it queries the task configuration table of the target verification task and the data source configuration table associated with the task configuration table. Then, it processes the data in each data source configuration table sequentially. Finally, it executes the verification scripts related to the data in the data source configuration tables through dynamic compilation to complete the consistency verification of multi-source data.

[0083] Consistency verification refers to the process of verifying the consistency and accuracy of similar data from different data sources in terms of logic, numerical values, or format.

[0084] S304: Correct the data source according to the data verification results and the data correction script in the configuration information of the target verification task, and issue an early warning for data anomalies.

[0085] In this embodiment, if the data verification result shows data consistency, no further correction or warning processing is required. If the data verification result shows data inconsistency, it is determined whether data correction is needed based on the configuration information of the target verification task. If the configuration information of the target verification task includes a data correction script, the data in the verification data source is corrected according to the corresponding data correction script. In this embodiment, the information configuration table structure of the target verification task including the data correction script is as follows: Figure 7 As shown.

[0086] In this embodiment, if the data verification result shows inconsistency, it can be determined that there is an anomaly in the data source, and corresponding early warning processing needs to be performed. In practical application scenarios, early warning processing can be carried out through SMS, telephone, or other means. The SMS type can be mobile SMS or app notification. It should be noted that this embodiment does not limit the specific implementation method of early warning processing. In a feasible embodiment, early warning can also be graded according to the specific characteristics of the data anomaly, that is, data inconsistency can be divided into level one, level two, and level three anomalies, and different early warning methods can be set for different levels of anomalies, so that users can immediately understand the degree of data anomaly after receiving an early warning.

[0087] In summary, this embodiment provides a data processing method for a near real-time verification platform. Employing a near real-time event-driven architecture, the latency for problem detection is reduced from hours in traditional solutions to seconds, significantly improving real-time performance. Through dynamic latency strategies, verification timing is configured differently based on business link characteristics, reducing invalid verifications and effectively lowering the false alarm rate. Based on a scripted data source adaptation framework, the time for adding a new data source is shortened from 3 person-days to 0.5 person-days. Hot script loading supports zero-downtime iteration, significantly reducing maintenance costs. Replacing traditional full table scans with event-driven verification effectively reduces system resource consumption. The platform can automatically complete data repair, significantly reducing manual intervention costs and significantly improving business stability.

[0088] In one embodiment, the step of converting pending data generated by the business system into standardized verification events includes:

[0089] First, the system uses a Python script to parse MySQL binlog to obtain business data, parses business logs using a custom delimiter, retrieves business messages by listening to the business message queue, and then sends the business data, business logs, and business messages to the Kafka message queue.

[0090] Second, the target data that meets the conditions is filtered by the screening script, and the filtered target data is transformed into standardized verification events by the transformation script.

[0091] Third, standardized verification events are stored in the verification event configuration table and cached in Redis, while also being sent to the verification event queue.

[0092] In this embodiment, regarding business data, a Python script is used to parse the MySQL binlog and send the changed data to Kafka. The script uses the python-mysql-replication library to parse the binlog and the kafka-python library to send messages.

[0093] Regarding business log data, each business system prints logs in a specific format and uses a custom delimiter to split the logs. A Python script is used to parse the business logs using this custom delimiter format. This script can monitor the log files in real time, parse log lines according to the specified delimiter, and send the results to Kafka.

[0094] Regarding business messages, these are messages sent by various business systems when processing relevant business logic. For example, after an order is placed, a "successful order" message is sent. Then, the points and fulfillment systems will listen for the order message and execute their corresponding logic. The verification platform can also listen for relevant business messages to trigger verification events.

[0095] In this embodiment, business data, business logs, and business messages are converted into verification events. Under the same triggering condition, multiple verification events may be generated. For example, a loan message can be converted into a product A loan event, a product B loan event, a loan exceeding 100,000 event, and so on, as needed. Verification events are a data structure unique to the verification platform, which stores the relevant verification events. Verification events have a certain degree of reusability; one verification event may trigger multiple verification tasks. For example, a "loan successful" verification event may trigger three verification tasks: accounting reconciliation, financial reconciliation, and payment reconciliation.

[0096] In this embodiment, the specific implementation methods of the second and third steps described above can be referred to the foregoing embodiments, and will not be repeated here.

[0097] In one embodiment, the step of finding the associated target verification task based on the standardized verification event and sending the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task includes:

[0098] First, query the event-task association mapping table based on the event code of the standardized verification event. The event-task association mapping table is a key-value pair structure, where the key is the event code of the standardized verification time and the value is the task list of the associated target verification task.

[0099] Second, standardized verification events are written to the verification event record table, which includes event ID, event code, event attributes, and write time.

[0100] Third, query the configuration information of the target verification task based on the event-task association mapping table to obtain the dynamic delay time corresponding to each target verification task.

[0101] Fourth, according to the dynamic delay time, send each target verification task to the task delay queue respectively.

[0102] In this embodiment, the standardized event code is the event code used to verify the event. The event-task association mapping table can be stored in a Redis cache, and its map structure is as follows: Figure 6 As shown.

[0103] In this embodiment, the verification events are stored in a verification event record table, which records key information such as event ID, event code, and event attributes for easy traceability. The configuration information of each target verification task is queried to obtain the corresponding dynamic delay time. This delay time is configured differently based on the characteristics of the business process; for example, a 2-minute delay for loan transactions and a 30-second delay for membership points. Finally, each target verification task is sent to the task delay queue according to the dynamic delay time.

[0104] Based on the above steps, by configuring corresponding dynamic delay strategies, the timing of verification can be configured differently according to the characteristics of the business link, thereby reducing invalid verifications.

[0105] In one embodiment, the step of querying and processing data from each verification data source according to the configuration of the target verification task includes:

[0106] First, for the data source of the verification event, execute the preset data processing script to perform secondary processing on the raw event data.

[0107] Second, for the data source of the application system, generate interface input parameters and initiate interface calls. After obtaining the returned data, execute the preset data processing script for processing.

[0108] Third, for the database data source, execute a preset SQL script to query the data, and process the query results through a preset data processing script.

[0109] In this embodiment, the event data source configuration table is checked as shown in Table 4 below:

[0110] Table 4

[0111]

[0112] For verification events, a specific event needs to be selected, and a data processing script needs to be configured. The data processing script is used to perform secondary processing on the raw event data. In a specific application scenario, after the task processor listens for the target verification task that needs to be processed, if the data source associated with the target verification task is the verification event data source, the data processing script is directly executed to process the data.

[0113] The application system data source configuration table is shown in Table 5 below:

[0114] Table 5

[0115]

[0116] For each application system, specific system information, interface input parameter scripts, and configuration data processing scripts need to be selected. The interface input parameter script generates the interface input parameters, while the data processing script performs secondary processing on the raw interface output data. In a specific application scenario, after the task processor detects a target verification task that needs to be processed, if the data source associated with the target verification task is the application system's data source, it first obtains the service address list and queries the target service instance list using the API or SDK provided by the registry center. Then, it performs load balancing to select an instance, choosing an available node from the address list (using strategies such as round-robin, random, or weighted). Next, it executes the script using the interface input parameters to obtain the interface input parameter information, constructs a request using the selected machine information (IP + port), and initiates the call through the RPC framework. Finally, it executes the data processing script to process the data.

[0117] The database data source configuration table is shown in Table 6 below:

[0118] Table 6

[0119]

[0120] For databases, a specific database needs to be selected. SQL is executed on the database, and a data processing script is used to further process the data retrieved from the original database query. In a specific application scenario, after the task processor detects a target verification task that needs to be processed, if the data source associated with the target verification task is a database data source, then the data configuration information is obtained according to the configured database information. Afterwards, the SQL script is executed directly, and finally, the data processing script is executed to process the data.

[0121] In one embodiment, the step of executing the verification script through dynamic compilation includes:

[0122] First, the text content of the verification script is embedded into a predefined Java class template to generate complete and compilable Java source code.

[0123] In this embodiment, the system embeds the user-entered script text into a predefined Java class template based on the configured data source type. For example, the data processing script is embedded into a template named `DataProcessor`, generating the `process(ObjectinputData)` method body. The data verification script is embedded into a template named `DataComparator`, generating the `compare(ObjectprocessedData)` method body. Finally, a syntactically complete and compilable Java source code string is generated.

[0124] Second, the Java compiler is invoked to compile the Java source code and generate bytecode, which is then loaded into the currently running Java Virtual Machine using a custom class loader.

[0125] In this embodiment, the system can use the JavaCompiler API to start the compilation process in memory or a temporary directory, taking the Java source code string generated in the previous step as input. Upon successful compilation, Java bytecode is generated. The system then dynamically loads this bytecode into the currently running Java Virtual Machine (JVM) using a custom URLClassLoader or InMemoryClassLoader.

[0126] Third, the verification processing object is instantiated through reflection so that the corresponding verification logic can be called and executed.

[0127] In this embodiment, objects such as DataProcessor and DataComparator are instantiated from the loaded class using Java reflection, and then called by the execution engine.

[0128] In a more detailed embodiment, such as Figure 8 As shown, the near real-time verification platform of this application can be deployed on a server cluster, adopts a microservice architecture design, and each module is deployed independently and communicates through a service registration and discovery mechanism.

[0129] The system deployment is as follows: Two event receiver nodes are deployed, achieving high availability through load balancing, responsible for the collection and transformation of multi-source data; three event processor nodes are deployed, handling verification events and task scheduling; four task processor nodes are deployed, processing various verification tasks in parallel to improve processing efficiency; and two correction and early warning nodes are deployed to ensure the stable operation of correction and early warning functions. A Redis cluster is used to cache the event-task association mapping table and verification events, a MySQL database stores various configuration tables and event record tables, and a Kafka cluster is used for high-availability deployment of the message queue.

[0130] The system operation flow is as follows: After the business system starts, the event receiver continuously monitors various data sources, collects business data, business logs, and business messages in real time, and transforms them into standardized verification events; the event processor monitors the verification event queue in real time, completing task association and dynamic scheduling; the task processor executes data collection and verification operations on time according to the tasks in the task delay queue; and the correction and early warning device automatically performs data correction based on the verification results and pushes alarm information. During system operation, the operating status of each module, task execution status, and anomaly handling records can be viewed through a visual monitoring interface, facilitating operation and maintenance management.

[0131] In a more detailed embodiment, the data processing method of the near real-time verification platform, taking the loan business in the financial industry as an example, illustrates the implementation process of the method in this application in detail:

[0132] The first step involves multi-source data collection and event transformation. The data to be processed generated by the loan business system includes binlog logs (business data) from the loan business database, system operation logs (business logs), and loan completion messages (business messages). The event receiver's three-source data processing unit uses Python scripts to parse MySQL binlogs to obtain business data such as loan amount and loan time, parses business logs using custom delimiters to obtain operation records, and listens to the RabbitMQ message queue to obtain loan completion messages. All three types of data are then sent to the Kafka message queue. The event transformation module uses a filtering script to select target data with loan amounts exceeding 10,000 yuan. This data is then transformed into "Loan Exceeding 10,000 Yuan Events" using a transformation script, containing standardized attributes such as event ID, event code, loan amount, and user ID. This data is stored in the event configuration table, cached in Redis, and simultaneously sent to the event verification queue.

[0133] The second step is to perform task association and dynamic scheduling. The event handler listens to the event queue. Upon receiving an event indicating a loan exceeding 10,000, it queries the Redis event-task association mapping table based on the event code, matching three target verification tasks: accounting reconciliation, financial reconciliation, and payment record reconciliation. This event is then written to the verification event record table, recording the event ID, event code, and event attributes. The dynamic delay time configured for each task is then queried: the accounting reconciliation task requires a 2-minute delay (due to asynchronous accounting entry generation), the financial reconciliation task requires a 1-minute delay, and the payment record reconciliation task requires a 30-second delay. Based on the respective delay times, the three verification tasks are sent to the task delay queue.

[0134] The third step is multi-source data verification and execution. The task processor listens to the task delay queue. When the payment record verification task reaches its execution time, it collects data from three types of data sources according to the task configuration: verification event data source (data on "loans exceeding 10,000", executing processing scripts to extract key fields), application system data source (payment system interface, querying the service instance list through Nacos, selecting nodes using a polling strategy, generating interface input parameters and calling through the Dubbo framework to obtain payment records and then executing processing scripts), and database data source (payment database, executing SQL queries to retrieve data, and formatting the results through processing scripts). After data collection is complete, the verification script is embedded into the DataComparator class template to generate source code. After compilation and loading, an object is instantiated through reflection and the verification logic is executed to compare whether the key fields such as payment amount and transaction time in the three types of data are consistent.

[0135] The fourth step is correction and early warning processing. If the verification reveals a discrepancy between the amount in the payment database and the event data, the correction and early warning system executes an SQL write-back script according to its configuration to update the erroneous data in the payment database. Simultaneously, based on the anomaly level, an alert is pushed to the financial administrator and technical maintenance personnel via WeChat Work. The alert includes the event ID, anomaly type, inconsistent fields, and automatic correction results, ensuring relevant personnel are promptly informed and can track the issue.

[0136] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0137] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0138] In one embodiment, such as Figure 9 As shown, a data processing device 900 is provided, including: an event receiving module 910, an event processing module 920, a task processing module 930, and a correction and early warning module 940, wherein:

[0139] The event receiving module 910 is used to convert the pending data generated by the business system into standardized verification events and upload the standardized verification events to the verification event queue; wherein, the pending data includes business logs, business data and business messages;

[0140] The event processing module 920 is used to listen to the verification event queue, find the associated target verification task according to the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0141] The task processing module 930 is used to monitor the task delay queue, query and process the data of each verification data source according to the configuration information of the target verification task, and complete the multi-source data consistency verification by dynamically compiling and executing the verification script.

[0142] The correction and early warning module 940 is used to correct the verification data source according to the data verification results and the data correction script in the configuration information of the target verification task, and to provide early warning processing for data anomalies.

[0143] In summary, this embodiment provides a data processing device for a near real-time verification platform. Employing a near real-time event-driven architecture, the latency for problem detection is reduced from hours in traditional solutions to seconds, significantly improving real-time performance. Through dynamic latency strategies, verification timing is configured differently based on business link characteristics, reducing invalid verifications and effectively lowering the false alarm rate. Based on a scripted data source adaptation framework, the time for adding a new data source is shortened from 3 person-days to 0.5 person-days. Hot script loading supports zero-downtime iteration, significantly reducing maintenance costs. Using an event-driven verification mode instead of traditional full table scans effectively reduces system resource consumption. The platform can automatically complete data repair, significantly reducing manual intervention costs and significantly improving business stability.

[0144] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0146] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0148] The pending data generated by the business system is transformed into standardized verification events, and the standardized verification events are uploaded to the verification event queue; the pending data includes business logs, business data, and business messages.

[0149] Listen to the verification event queue, find the associated target verification task based on the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0150] The system monitors the task delay queue, queries and processes data from each verification data source based on the configuration information of the target verification task, and executes the verification script through dynamic compilation to complete the multi-source data consistency verification.

[0151] The data source is corrected based on the data verification results and the data correction script in the configuration information of the target verification task, and an early warning is issued for data anomalies.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0153] The pending data generated by the business system is transformed into standardized verification events, and the standardized verification events are uploaded to the verification event queue; the pending data includes business logs, business data, and business messages.

[0154] Listen to the verification event queue, find the associated target verification task based on the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0155] The system monitors the task delay queue, queries and processes data from each verification data source based on the configuration information of the target verification task, and executes the verification script through dynamic compilation to complete the multi-source data consistency verification.

[0156] The data source is corrected based on the data verification results and the data correction script in the configuration information of the target verification task, and an early warning is issued for data anomalies.

[0157] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0158] The pending data generated by the business system is transformed into standardized verification events, and the standardized verification events are uploaded to the verification event queue; the pending data includes business logs, business data, and business messages.

[0159] Listen to the verification event queue, find the associated target verification task based on the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task.

[0160] The system monitors the task delay queue, queries and processes data from each verification data source based on the configuration information of the target verification task, and executes the verification script through dynamic compilation to complete the multi-source data consistency verification.

[0161] The data source is corrected based on the data verification results and the data correction script in the configuration information of the target verification task, and an early warning is issued for data anomalies.

[0162] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method, employing a near real-time verification platform, includes: The pending data generated by the business system is transformed into standardized verification events, and the standardized verification events are uploaded to the verification event queue; wherein, the pending data includes business logs, business data, and business messages; Monitor the verification event queue, find the associated target verification task based on the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task; The task delay queue is monitored, and the data of each verification data source is queried and processed according to the configuration information of the target verification task. The verification script is dynamically compiled and executed to complete the multi-source data consistency verification. The verification data source is corrected based on the data verification results and the data correction script in the configuration information of the target verification task, and an early warning is issued for data anomalies.

2. The method according to claim 1, characterized in that, The process of converting pending data generated by the business system into standardized verification events includes: The system uses a Python script to parse MySQL binlog to obtain business data, parses business logs using a custom delimiter, obtains business messages by listening to a business message queue, and sends the business data, business logs, and business messages to a Kafka message queue. The target data that meets the criteria is filtered by a screening script, and the filtered target data is transformed into standardized verification events by a transformation script. The standardized verification events are stored in the verification event configuration table and cached in Redis, and simultaneously sent to the verification event queue.

3. The method according to claim 1, characterized in that, The step of finding the associated target verification task based on the standardized verification event and sending the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task includes: The event-task association mapping table is queried based on the event code of the standardized verification event. The event-task association mapping table is a key-value pair structure, where the key is the event code of the standardized verification time and the value is the task list of the associated target verification task. The standardized verification events are written to the verification event record table, which includes event ID, event code, event attributes, and write time. The configuration information of the target verification task is queried according to the event-task association mapping table to obtain the dynamic delay time corresponding to each target verification task; According to the dynamic delay time, each target verification task is sent to the task delay queue.

4. The method according to claim 1, characterized in that, The types of data sources to be verified include data sources from verification events, data sources from application systems, and data sources from databases. The step of querying and processing data from each verification data source according to the configuration of the target verification task includes: For the data source of the verification event, a preset data processing script is executed to perform secondary processing on the raw event data; For the data source of the application system, generate interface input parameters and initiate interface calls. After obtaining the returned data, execute the preset data processing script for processing. For a database data source, execute a preset SQL script to query the data, and process the query results using a preset data processing script.

5. The method according to claim 1, characterized in that, The process of dynamically compiling and executing the verification script includes: The text content of the verification script is embedded into a predefined Java class template to generate complete and compilable Java source code; The Java source code is compiled into bytecode by calling the Java compiler, and the bytecode is loaded into the currently running Java Virtual Machine by a custom class loader; The verification processing object is instantiated through reflection so that the corresponding verification logic can be called and executed.

6. The method according to claim 3, characterized in that, The configuration information for the target verification task includes the task name, verification trigger event, dynamic delay time, verification data source type, data processing script, and verification script. The method further includes: The target verification task is configured through a visual interface to obtain the configuration information, which is then stored in the task configuration table.

7. A data processing apparatus, characterized in that, The device employs a near real-time verification platform, comprising: The event receiving module is used to convert the pending data generated by the business system into standardized verification events and upload the standardized verification events to the verification event queue; wherein, the pending data includes business logs, business data and business messages; The event handling module is used to listen to the verification event queue, find the associated target verification task according to the standardized verification event, and send the target verification task to the task delay queue according to the dynamic delay time corresponding to the target verification task. The task processing module is used to monitor the task delay queue, query and process the data of each verification data source according to the configuration information of the target verification task, and complete the multi-source data consistency verification by dynamically compiling and executing the verification script. The correction and early warning module is used to correct the data source based on the data verification results and the data correction script in the configuration information of the target verification task, and to provide early warning for data anomalies.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1 to 6.

Images