Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and relates to the technical field of distributed system data processing. According to the method, the financial detail data of each product application system is uniformly acquired and analyzed, so that centralized management and efficient utilization of the data are realized, and the data processing efficiency and accuracy are remarkably improved. And the analyzed data is stored in a contract detail list of an accounting system database, so that the integrity and traceability of the data are ensured, and a solid foundation is provided for subsequent financial analysis and decision making. When an exception occurs in the analysis process, the method can timely record data of an abnormal batch into an error information data warehouse, so that data loss and error accumulation are avoided, and problems can be timely found and solved. And finally, through day-end batch job processing, traversing an error information data warehouse and processing the recorded data, the accuracy and integrity of the data are further ensured, and the stability and reliability of the system are improved at the same time.

Description

Technical Field

[0001] The present application relates to the technical field of distributed system data processing, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of the digital economy and the rapid increase of financial data, major financial institutions are facing huge challenges of limited host system processing capacity and high maintenance costs under the centralized architecture. As a result, the industry has undergone a transformation and upgrade from a centralized architecture to a distributed architecture.

[0003] Distributed architecture has its unique advantages, which are reflected in the fact that the various components and services of the system are distributed on multiple independent computing nodes. Through network communication collaboration, the system load is distributed to multiple nodes to achieve parallel job processing and load balancing. Since multiple job requests can be processed simultaneously, the system's processing power and response speed have been greatly improved.

[0004] However, the complexity of the distributed architecture also brings some additional problems. The communication between different microservices will inevitably face a series of data consistency issues. Especially for financial transaction systems with extremely high requirements for real-time, reliability, and accuracy, how to ensure the integrity and accuracy of data transmission under the distributed architecture is an issue that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, this application provides a data processing method, including the following contents:

[0006] In a first aspect, the present application provides a data processing method, the method comprising:

[0007] Obtain detailed financial data from each product application system;

[0008] Parsing the financial detailed data and storing the parsed data in a contract detail table of an accounting system database;

[0009] When an exception occurs during the parsing process, the abnormal batch of data is recorded in the error information data warehouse;

[0010] Perform end-of-day batch job processing, traverse the error information data warehouse, and process the data recorded therein.

[0011] Optionally, before obtaining the financial detailed data from each product application system, the method further includes:

[0012] Set up multiple message partitions in the distributed publish-subscribe messaging system Kafka, each partition is used to store financial detailed data of different product application systems;

[0013] Each product application corresponds to a Kafka topic, and each topic is set with one or more partitions.

[0014] Optionally, when an exception occurs during the parsing process, recording the abnormal batch of data into the error information data warehouse includes:

[0015] If there is a primary key conflict or data format error, the abnormal data will be recorded in the error information data warehouse;

[0016] When a Kafka consumer rebalances, the relevant batch message data is recorded in the error information data warehouse.

[0017] Optionally, before performing the end-of-day batch operation, the method further includes:

[0018] An end mark message is obtained, and end-of-day batch operations are performed after receiving the end mark message.

[0019] Optionally, traversing the error information data warehouse and processing the data recorded therein includes:

[0020] For each piece of data in the error information data warehouse, first check whether there is a primary key conflict in the accounting system database. If so, discard the data; otherwise, rewrite the data into the database.

[0021] In a second aspect, the present application provides a data processing device, the device comprising:

[0022] An acquisition unit, used to acquire financial detailed data from each product application system;

[0023] A parsing unit, used to parse the financial detailed data and store the parsed data into a contract detail table of the accounting system database;

[0024] A first processing unit is used to record the abnormal batch of data into an error information data warehouse when an abnormality occurs during the parsing process;

[0025] The second processing unit is used to perform end-of-day batch job processing, traverse the error information data warehouse, and process the data recorded therein.

[0026] Optionally, the device further comprises:

[0027] A setting unit is used to set multiple message partitions in the distributed publish-subscribe message system Kafka, each partition is used to store financial detailed data of different product application systems;

[0028] Each product application corresponds to a Kafka topic, and each topic is set with one or more partitions.

[0029] Optionally, the first processing unit is specifically configured to record abnormal data into an error information data warehouse if there is a primary key conflict or a data format error;

[0030] When a Kafka consumer rebalances, the relevant batch message data is recorded in the error information data warehouse.

[0031] Optionally, the acquisition unit is further used to acquire an end mark message, and perform end-of-day batch operations after receiving the end mark message.

[0032] Optionally, the second processing unit traverses the error information data warehouse and processes the data recorded therein, including:

[0033] For each piece of data in the error information data warehouse, first check whether there is a primary key conflict in the accounting system database. If so, discard the data; otherwise, rewrite the data into the database.

[0034] In a third aspect, the present application provides a device, comprising a memory and a processor, the memory being used to store instructions or codes, and the processor being used to execute the instructions or codes, so that the device executes the data processing method introduced in any implementation of the first aspect.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, in which codes are stored. When the codes are executed, a device executing the codes implements the data processing method introduced in any implementation of the first aspect.

[0036] The present application provides a data processing method. When executing the method, first obtain the financial detailed data from each product application system, then parse the financial detailed data, store the parsed data in the contract detail table of the accounting system database, and when an exception occurs during the parsing process, record the abnormal batch of data in the error information data warehouse, and finally perform the end-of-day batch operation processing, traverse the error information data warehouse, and process the data recorded therein. The method can ensure the comprehensiveness and accuracy of the data by systematically acquiring and parsing the financial detailed data from multiple product application systems, laying a solid foundation for subsequent accounting processing and decision analysis. Secondly, the method directly stores the parsed data in the contract detail table of the accounting system database, realizes the instant update and efficient management of data, and greatly improves the speed and efficiency of data processing. More importantly, the method has excellent exception handling capabilities in the process of parsing data. Once encountering abnormal situations such as data format errors and primary key conflicts, the system can quickly identify and record the abnormal batch of data in the error information data warehouse. This design not only avoids system crashes or data loss caused by abnormal data, but also provides great convenience for subsequent error troubleshooting and data repair. Finally, the method also includes a batch job processing link at the end of the day, which can automatically traverse the error information data warehouse and reprocess the data recorded in it. This function not only ensures the integrity and consistency of the data, but also further improves the robustness and reliability of the system. Through this method, the error rate in the data processing process can be effectively reduced, the data quality can be improved, and strong support can be provided for the financial management and decision-making of the enterprise.

[0037] In summary, the data processing method provided in this application shows significant advantages in data processing efficiency, exception handling capability and data quality, providing a strong technical guarantee for the financial management and decision-making of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] As described in the background technology of this application, the complexity of the distributed architecture brings some additional problems. For example, when communicating between different microservices, a series of data consistency issues will be faced. Specifically, the existing distributed public accounting system is coupled with a distributed publish-subscribe message system. Through an infinite loop, data records related to detailed topics are continuously pulled from the message queue, and the corresponding message processing instance is used to process the Kafka records. The Kafka offset is submitted only after the data is successfully stored in the database to prevent unprocessed data. If the database crashes, the program will not submit the Kafka offset, and will continue to process messages normally after the database is restored; if the Pod is restarted or killed, the program will no longer continue to consume data, ensuring a smooth exit of the program, but it may cause data loss.

[0042] In order to solve the above technical problems, an embodiment of the present application provides a data processing method, which obtains the financial detail data generated by each product application in real time through the distributed publish-subscribe message system Kafka, combines production practice, considers the deadlock problem that may be caused by the Reblance feature of the third-party component, establishes a trusted message error storage center to record the detail data that has not been processed due to deadlock, and configures additional job nodes in the daily end-of-day batch job template to read the message error data, so as to ensure the integrity of the account detail data received.

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application. Figure 1 As shown, the data processing method provided in the embodiment of the present application may include:

[0045] S101. Obtain detailed financial data from each product application system.

[0046] The public accounting system of commercial banks obtains detailed financial data from various product application systems. The public accounting system of commercial banks provides basic accounting capability component packages for various financial product applications to support the application to complete various account-related business functions, process detailed financial data from various product application systems, complete general ledger processing and account reconciliation, and transmit data to subsequent downstream systems.

[0047] The public accounting system in this application is a distributed public accounting system, which is based on a cloud platform and uses containerization technology. It has high performance, is scalable, and can quickly build a highly reliable container cluster. The public accounting system introduces two independent microservices: a message consumer module and a batch data processing module. The message consumer module is configured with a Kafka native API package to receive messages and write them to the local database, and the batch processing module reads detailed data for subsequent data processing.

[0048] Product application systems can be various business systems within the bank, such as loan systems, deposit systems, and wealth management product sales systems, or systems developed in cooperation with external institutions, such as payment and clearing systems that connect with third-party payment institutions. Each product application system is responsible for processing financial transactions within its specific business scope and generating corresponding detailed data. These data may include key information such as transaction amount, transaction time, counterparty information, and transaction type. In order to pass this information to subsequent accounting systems or other processing systems, each product application system will send the generated detailed data to the Kafka message queue.

[0049] As a distributed publish-subscribe messaging system, Kafka provides a reliable messaging mechanism for these product application systems. Through Kafka, each product application system can asynchronously send data to the message queue without waiting for the processing results of the subsequent system. This not only improves the processing efficiency of the system, but also ensures the timeliness and integrity of the data. The Kafka cluster stores and organizes data stream information in the form of topics, where each product application corresponds to an independent topic.

[0050] In order to manage and access this data more efficiently, Kafka sets one or more partitions for each topic. These partitions not only help improve the parallelism of data processing, but also ensure that data is stored in a certain order within each partition. The financial detail data sent by each product application will be distributed to these partitions, and each detail data is assigned a unique offset, which is used to accurately locate and obtain historical messages in the subsequent data consumption process.

[0051] In summary, by setting up multiple message partitions in the distributed publish-subscribe messaging system Kafka, each partition is used to store detailed financial data from different product application systems. Each product application corresponds to a Kafka topic, which contains one or more partitions for receiving and storing detailed data sent by the application. In this way, through Kafka's partition and offset mechanism, it is possible to effectively manage and access detailed financial data from various product application systems.

[0052] S102: Analyze the financial detailed data, and store the analyzed data into a contract detail table of the accounting system database.

[0053] The public accounting system message consumer module will first retrieve the detailed data sent by each product application from the Kafka message queue. As the core basic application module of commercial banks, the public accounting system is used to provide basic accounting capabilities for various core system applications (such as credit cards, corporate loans, personal liabilities, corporate deposits, wealth management, etc.) under a distributed architecture, and interact with various applications through distributed message middleware.

[0054] As a message middleware, Kafka can reliably store and transmit these messages to ensure data consistency and integrity. After retrieving the message, the message consumer module will parse the message format. Since different product applications may use different data formats to send messages, the message consumer module needs to be able to identify and parse these different formats in order to extract useful information.

[0055] After parsing is completed, the message consumer module will insert the extracted data into the contract details table of the accounting system database. This table is a key component of the accounting system for storing and managing contract-related detailed data. It records the detailed transaction information of each contract, including transaction amount, transaction time, transaction type, etc.

[0056] S103. When an exception occurs during the parsing process, the data of the abnormal batch is recorded in the error information data warehouse.

[0057] When parsing financial detail data from the Kafka message queue, various anomalies may be encountered. In order to ensure the integrity and traceability of the data, the system will take a series of measures to handle these anomalies and record the abnormal data in the error information data warehouse.

[0058] Error Information Data Warehouse: This is a database or data warehouse dedicated to storing abnormal data. When an exception occurs during data parsing or processing, the relevant batch of data will be recorded here for subsequent error analysis and processing.

[0059] Primary key conflict: In a database table, the primary key is a field that uniquely identifies each row of data in the table. A primary key conflict occurs when you try to insert a new record and the primary key value for that record already exists in the table. In financial data processing, this can be caused by duplicate transaction records or data errors.

[0060] Data format error: When the data received from the Kafka message queue does not conform to the expected format, a data format error occurs. This may be caused by changes in the sender's data format, errors in the data transmission process, or defects in the parser itself.

[0061] Kafka consumer rebalance: Consumers in a Kafka cluster may trigger a rebalance process due to the increase or decrease of consumer instances. During the rebalance process, Kafka will reallocate the partitions of the topics subscribed by consumers to different consumer instances. This may cause some consumer instances to experience interruptions when processing messages, thus causing exceptions.

[0062] Program exceptions: During data processing, the program may throw exceptions due to code errors, insufficient resources, external system problems, etc. These exceptions need to be caught and handled to ensure system stability and data integrity.

[0063] When an exception occurs during the parsing process (such as primary key conflict, data format error, etc.), the exception handling module will capture these exceptions and record the relevant batches of data in the error information data warehouse. When the Kafka consumer is rebalanced, the partition redistribution may cause the consumer to interrupt when processing messages, so the relevant batches of message data also need to be recorded in the error information data warehouse. If the program has an exception at any time (including deadlock caused by rebalance, etc.), the exception handling module will also capture these exceptions and record the abnormal batches of data in the error information data warehouse. It should be noted that in the case of deadlock caused by rebalance, since the offset submitted to Kafka by the database may be inconsistent with the actual consumption progress, Kafka may include some of the data that has been consumed when resending the message. This part of the duplicate data will also be recorded in the error information data warehouse for subsequent deduplication and analysis.

[0064] Through the above method, it can be ensured that any exceptions encountered during the data processing process can be captured and recorded in a timely manner, thereby providing strong support for subsequent error analysis and processing.

[0065] S104, performing end-of-day batch job processing, traversing the error information data warehouse, and processing the data recorded therein.

[0066] Since each product application system will continuously send financial detailed data to the public accounting system, in order to ensure that the accounting system can accurately process this data, each product application system will send an end mark message after completing the sending of all detailed data. The purpose of this message is to notify the public accounting system that all relevant detailed data have been sent and data processing can begin.

[0067] After receiving this end message, the public accounting system will trigger the end-of-day batch job processing flow to traverse and process the data in the error information data warehouse. Since the data in these information warehouses may be abnormal data due to various reasons (such as primary key conflicts, data format errors, Kafka consumer rebalance, etc.), special processing is required.

[0068] During the processing, for each piece of data in the error information data warehouse, the system will first check whether there is a primary key conflict in the accounting system database. If there is a primary key conflict, it means that the data may be duplicated or have been processed, so the system will directly discard the data to avoid unnecessary impact on the accounting system. If there is no primary key conflict in the data, the system will rewrite the data into the contract details table of the accounting system database. This process is carried out in batches, which can greatly improve the efficiency of data processing.

[0069] It is worth noting that the public accounting system will trigger the execution of end-of-day batch jobs at regular intervals every day. This application expands on the original end-of-day accounting processing logic and adds a new trigger condition: that is, all detailed data has been sent and received. In this way, regardless of whether the system has experienced abnormal situations such as deadlock, the program will directly start from the error information data warehouse, traverse the entire table, and parse and rewrite each piece of data. If an abnormal situation such as deadlock occurs, there will be a corresponding record in the error information data warehouse. By processing these records, the system can ensure the accuracy and completeness of the accounting data.

[0070] The data processing method proposed in the above embodiment of the present invention can be used to prevent the loss of account details data caused by deadlock of the bank system in a distributed environment. This method effectively responds to the consumer rebalance phenomenon caused by common scenarios such as expansion and contraction, restart of services, etc. in distributed systems by increasing redundancy and setting up an error information data warehouse, ensuring the integrity of information transmission and providing recoverable data backup when the system is deadlocked. During the end-of-day batch processing, the deadlock missed data can be re-entered into the warehouse by adding fixed read-write jobs, thereby enhancing the robustness and fault tolerance of the system. This method enables various applications of commercial banks to safely and reliably transmit data through a distributed messaging system, providing an innovative solution to the system security and reliability issues faced by commercial banks in the process of transformation and upgrading of distributed architectures.

[0071] The above are some specific implementations of a data processing method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.

[0072] Figure 2 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2 As shown, the data processing device 200 provided in the embodiment of the present application includes:

[0073] An acquisition unit 210 is used to acquire financial detailed data from each product application system;

[0074] The parsing unit 220 is used to parse the financial detailed data and store the parsed data into the contract details table of the accounting system database;

[0075] The first processing unit 230 is used to record the abnormal batch of data into the error information data warehouse when an abnormality occurs during the parsing process;

[0076] The second processing unit 240 is used to perform end-of-day batch job processing, traverse the error information data warehouse, and process the data recorded therein.

[0077] In one implementation of the embodiment of the present application, the device further includes:

[0078] A setting unit is used to set multiple message partitions in the distributed publish-subscribe message system Kafka, each partition is used to store financial detailed data of different product application systems;

[0079] Each product application corresponds to a Kafka topic, and each topic is set with one or more partitions.

[0080] In an implementation of the embodiment of the present application, the first processing unit is specifically configured to record abnormal data into an error information data warehouse if there is a primary key conflict or a data format error;

[0081] When a Kafka consumer rebalances, the relevant batch message data is recorded in the error information data warehouse.

[0082] In an implementation of the embodiment of the present application, the acquisition unit is further used to acquire an end mark message, and perform end-of-day batch operations after receiving the end mark message.

[0083] In an implementation of the embodiment of the present application, the second processing unit traverses the error information data warehouse and processes the data recorded therein, including:

[0084] For each piece of data in the error information data warehouse, first check whether there is a primary key conflict in the accounting system database. If so, discard the data; otherwise, rewrite the data into the database.

[0085] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0086] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.

[0087] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method described in any embodiment of the present application.

[0088] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0089] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0090] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and apparatus embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain detailed financial data from each product application system; Parsing the financial detailed data and storing the parsed data in a contract detail table of an accounting system database; When an exception occurs during the parsing process, the abnormal batch of data is recorded in the error information data warehouse; Perform end-of-day batch job processing, traverse the error information data warehouse, and process the data recorded therein.

2. The method according to claim 1, characterized in that Before obtaining the financial detailed data from each product application system, the method further includes: Set up multiple message partitions in the distributed publish-subscribe messaging system Kafka, each partition is used to store financial detailed data of different product application systems; Each product application corresponds to a Kafka topic, and each topic is set with one or more partitions.

3. The method according to claim 1, characterized in that When an exception occurs during the parsing process, recording the abnormal batch of data into the error information data warehouse includes: If there is a primary key conflict or data format error, the abnormal data will be recorded in the error information data warehouse; When a Kafka consumer rebalances, the relevant batch message data is recorded in the error information data warehouse.

4. The method according to claim 1, characterized in that: Before performing the end-of-day batch operation, the method further includes: An end mark message is obtained, and end-of-day batch operations are performed after receiving the end mark message.

5. The method according to claim 4, characterized in that The traversing the error information data warehouse and processing the data recorded therein comprises: For each piece of data in the error information data warehouse, first check whether there is a primary key conflict in the accounting system database. If so, discard the data; otherwise, rewrite the data into the database.

6. A data processing device, characterized in that: The device comprises: An acquisition unit, used to acquire financial detailed data from each product application system; A parsing unit, used to parse the financial detailed data and store the parsed data into a contract detail table of the accounting system database; A first processing unit is used to record the abnormal batch of data into an error information data warehouse when an abnormality occurs during the parsing process; The second processing unit is used to perform end-of-day batch job processing, traverse the error information data warehouse, and process the data recorded therein.

7. The device according to claim 6, characterized in that The first processing unit is specifically used to record abnormal data into the error information data warehouse if there is a primary key conflict or data format error; When a Kafka consumer rebalances, the relevant batch message data is recorded in the error information data warehouse.

8. The device according to claim 6, characterized in that The acquisition unit is further used to acquire an end mark message, and perform end-of-day batch operations after receiving the end mark message.

9. A computing device, characterized in that The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images