A Redis-based account reconciliation method and system

Abstract

The present invention discloses a reconciliation method and system based on Redis, the method comprising: relying on a scheduling framework, regularly triggering a reconciliation task initialization process, generating a reconciliation task; writing the reconciliation task into a delay queue, regularly pulling a reconciliation file; parsing the reconciliation file and converting the data into a unified format; pulling, cleaning, and filtering system transaction data; comparing the reconciliation data with the system transaction data through Redis, obtaining intersections and differences, and completing the reconciliation. The present invention is based on a message processing mechanism of a distributed scheduling framework and a delay queue, has the advantages of low development difficulty, less code intrusion, prevention of single point failures, and higher timeliness, and prevents repeated polling and scanning of databases from giving system and database performance pressure; the present invention is also based on Redis data comparison, and compared with traditional reconciliation methods, when the amount of data is large, the time consumption is significantly reduced, and the pressure on the database is also reduced, reducing the deployment cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of payment system data reconciliation, and in particular relates to a Redis-based reconciliation method and system. Background Art

[0002] In non-cloud architecture applications, traditional payment systems store end-of-day reconciliation files on local disks or disk arrays, and use relational databases to perform data matching loops one by one.

[0003] This solution has the following problems:

[0004] 1. The storage method has a single point of failure. If the disk or array fails, the entire reconciliation process will fail.

[0005] 2. The storage method has high deployment and maintenance costs, and subsequent capacity expansion is difficult; the disaster recovery capability is poor, and data is easily lost when a disk fails.

[0006] 3. Traditional storage empowers the entire system and cannot perform file permission segmentation and file isolation based on payment type or payment channel. Basically, permission management is almost zero. In addition, file storage paths and methods are managed by the payment system and are heavily coupled with the payment system. At the same time, the disk or array has no high-availability processing, which has a greater impact on the system when problems occur.

[0007] 4. Data matching is based on database comparison, which has low performance and long time consumption. When the amount of data is huge, the comparison time is uncontrollable and occupies the database I / O performance for a long time. If there is real-time processing operation on the database in the same time period, the processing performance will be reduced, which will have a greater impact on the system. Summary of the invention

[0008] The purpose of the present invention is to provide a reconciliation method and system based on Redis to solve the problems in the existing end-of-day reconciliation scheme mentioned in the above background technology, such as high deployment and maintenance cost, single point failure of files, poor disaster recovery capability, and reconciliation file management relying on the system with heavy dependence, high coupling, and almost zero file isolation and permission control capabilities.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] According to one aspect of the present invention, a Redis-based account reconciliation method is provided, the method comprising the following steps:

[0011] Relying on the scheduling framework, the reconciliation task initialization process is triggered regularly, and reconciliation tasks are generated according to the reconciliation file configuration template of each payment channel;

[0012] Write the reconciliation task into the delay queue. When the pull rule is met, the delay queue is triggered to send the corresponding payment channel reconciliation file pull message to pull the reconciliation file.

[0013] After configuring the reconciliation file type and message format pulled by each channel, convert the reconciliation data of each channel into a unified format;

[0014] Pull the system transaction data corresponding to the reconciliation data of each channel, and clean and filter the system transaction data;

[0015] The corresponding channel reconciliation data and system transaction data are pushed to two collections of the Redis cluster according to the specified format of Redis comparison, and then the collections are compared and the reconciliation results are generated.

[0016] Furthermore, after pulling the reconciliation file, the pulled reconciliation file is stored in MinIO, specifically including:

[0017] Calculate the MD5 value of the pulled reconciliation file and use the MD5 value as the MinIO object name;

[0018] Call the MinIO encapsulation method through the object name to check the reconciliation file for duplication;

[0019] If it already exists, it will directly return that the reconciliation file has been successfully pulled; if it does not exist, the reconciliation file will be uploaded to the MinIO specified object bucket for storage through the MinIO upload encapsulation method. After the reconciliation file is successfully uploaded, MinIO returns the object name and then returns that the reconciliation file has been successfully pulled;

[0020] Update the relevant data to associate with the MinIO object name.

[0021] Furthermore, if the pulling of the reconciliation file fails, the pulling message is rolled back to the delay queue message queue and triggered again after a delay, and the delay time can be configured.

[0022] Furthermore, the conversion of the reconciliation data of each channel into a unified format is performed through a file parsing engine, which parses and converts the reconciliation data according to the reconciliation file channel type, MinIO object bucket name, and MinIO object name.

[0023] Furthermore, converting the reconciliation data of each channel into a unified format specifically includes:

[0024] The file parsing engine downloads the corresponding account file through the MinIO object bucket name and MinIO object name, and then converts it into a file stream and pushes it into the file parsing engine;

[0025] The file parsing engine determines whether the file is a Txt file, and if not, converts it to Txt;

[0026] After the files are unified into Txt, the file parsing engine obtains the mapping relationship between the reconciliation file fields of the corresponding channel and the unified format fields;

[0027] The parsing factory inside the engine uses the dynamic proxy framework to dynamically create the parsing implementation class for the corresponding channel according to the configuration information and parsing template;

[0028] The dynamic implementation class is called to parse the file line by line, convert it into a unified format, and write it into a database, which includes but is not limited to MySQL, Oracle, SQL Server, DB2, DAMO, Nanda, Gauss and other relational databases, preferably MySQL database.

[0029] Furthermore, the above file parsing process also includes:

[0030] If the current parsing template is not applicable to the current channel, write specific parsing code, package it into a Jar file, and upload it;

[0031] The file parsing engine determines whether there is a custom implementation method in the current channel during parsing. If so, the corresponding Class file in the Jar file is loaded into the system through the self-encapsulated class loader;

[0032] The file parsing operation is completed by customizing the implementation method for the parsing interface proxy through the dynamic proxy framework.

[0033] If an error or exception occurs in the current analysis, the current analysis event message is rolled back to the delay queue and a delay trigger is set. At the same time, the task processing exception message is pushed through the message system to relevant personnel for observation and processing.

[0034] Furthermore, comparing the sets and generating the reconciliation results includes:

[0035] Compare the two sets, get the intersection, and store it in the Redis intersection set;

[0036] Compare the two sets, get the difference set, and store it in the Redis difference set;

[0037] Get the data in the Redis intersection and update the reconciliation data of the corresponding data to reconciliation completed;

[0038] Get the data in the Redis difference set and determine what kind of difference the corresponding data belongs to. If it is a difference that the system can correct by itself, the system will correct it by itself and update the reconciliation data of the corresponding data to reconciliation completed;

[0039] If self-correction is not possible, the reconciliation data of the corresponding data will be updated as a reconciliation error and written into the reconciliation error table.

[0040] Furthermore, after comparing the sets and generating the reconciliation results, the process also includes issuing a reconciliation result report: collecting statistics on the reconciliation data and error data of each channel, processing the data and generating a report.

[0041] Furthermore, each step of the Redis-based reconciliation method described in the present invention is provided with corresponding monitoring and event alarms.

[0042] In some embodiments of the present invention, the scheduling framework is an XXL-JOB distributed scheduling framework.

[0043] According to another aspect of the present invention, a Redis-based account reconciliation system is provided, the system comprising the following modules:

[0044] Task initialization module: used to trigger the reconciliation task initialization process on a regular basis and generate reconciliation tasks according to the reconciliation file configuration templates of each payment channel;

[0045] The first acquisition module is used to send a corresponding payment channel reconciliation file pull message according to the delay queue to pull the reconciliation file;

[0046] File parsing module: used to configure the reconciliation file type and message format pulled by each channel, and convert the reconciliation data of each channel into a unified format;

[0047] The second acquisition module is used to pull the system transaction data corresponding to the reconciliation file pulled by the first acquisition module, and clean and filter it;

[0048] Reconciliation module: used to push the data acquired by the first acquisition module and the second acquisition module to two sets of the Redis cluster according to the Redis comparison specified format, compare the sets, and complete the reconciliation.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] (1) The reconciliation method of the present invention is based on the XXL-JOB distributed scheduling framework, which has the advantages of lower development difficulty, less code intrusion, and prevention of single point failure compared with the traditional scheduling framework, greatly improving the development and implementation efficiency.

[0051] (2) The reconciliation method of the present invention is based on a message processing mechanism of a delay queue, which is more timely than a traditional scheduling mechanism and prevents repeated polling and scanning of the database from putting pressure on the system and database performance.

[0052] (3) The reconciliation method of the present invention adopts MinIO distributed object storage, which has flexible expansion, strong disaster recovery capability, low maintenance cost, and strong file isolation and permission control compared with traditional file storage methods.

[0053] (4) The reconciliation method of the present invention is based on Redis data comparison. Compared with the traditional method of comparing one by one through the database, the time consumption is significantly reduced when the data volume is huge, and it also reduces the pressure on the database and reduces the deployment cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 The figure is a schematic diagram of the steps of the reconciliation method based on Redis of the present invention.

[0056] Figure 2 The figure is a schematic diagram of the steps of a reconciliation method based on Redis according to an embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram of the steps of storing the pulled reconciliation file into MinIO storage according to an embodiment of the present invention.

[0058] Figure 4 The figure is a schematic diagram of the steps of converting the reconciliation data of each channel into a unified format according to an embodiment of the present invention.

[0059] Figure 5 The figure is a schematic diagram of the steps of comparing two Redis sets according to an embodiment of the present invention.

[0060] Figure 6 The figure is a flowchart of a reconciliation method based on Redis according to an embodiment of the present invention.

[0061] Figure 7 This is a schematic diagram of the reconciliation system module based on Redis in the present invention. DETAILED DESCRIPTION

[0062] The technical scheme in the embodiment of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides a reconciliation method based on Redis, which includes the following steps:

[0065] S1. Relying on the scheduling framework, the reconciliation task initialization process is triggered regularly, and the reconciliation tasks are generated according to the reconciliation file configuration templates of each payment channel.

[0066] Specifically, relying on the XXL-JOB distributed scheduling framework, the reconciliation task initialization process is triggered regularly at dawn every day, and the reconciliation tasks are generated according to the reconciliation file configuration templates of various payment channels (such as Alipay, WeChat, UnionPay, various banks, etc.).

[0067] S3. Write the reconciliation task into the delay queue. When the pull rule is met, the delay queue is triggered to send the corresponding payment channel reconciliation file pull message to pull the reconciliation file.

[0068] Specifically, the end-of-day reconciliation task initialization operation is executed through XXL-JOB scheduled scheduling, and the pull execution time is generated according to the configuration time of each channel; the pull task and pull time are written into the RocketMQ delay queue; after the specified pull time is reached, the RocketMQ delay queue triggers the message sending, and sends the corresponding channel pull message; after receiving the channel reconciliation file pull message, the reconciliation file is pulled through the corresponding pull interface.

[0069] In this embodiment, if the reconciliation file pull fails, the pull message is rolled back to the RocketMQ message queue and triggered again after a delay of 5 minutes. It should be noted that the delay time can be configured. After all the reconciliation file pull tasks are successful, the next task execution event information is pushed through the RocketMQ message middleware.

[0070] S5. After configuring the reconciliation file type and message format pulled by each channel, convert the reconciliation data of each channel into a unified format.

[0071] In this embodiment, if an error or exception occurs in the current analysis, the current analysis event message is rolled back to the RocketMQ delay queue, and a delay trigger is set. At the same time, the task processing exception message is pushed through the message system to relevant personnel for observation and processing.

[0072] After the file parsing and conversion task is completed, check whether the file parsing and conversion tasks of all channels have been completed. If they have been completed, push the next task execution event message through RocketMQ message middleware.

[0073] S7. Use the ETL tool to pull the system transaction data corresponding to the reconciliation data of each channel, clean and filter the system transaction data, and store it in a separate reconciliation database. If successful, push the next task execution event message through the RocketMQ message middleware.

[0074] S9. Push the corresponding channel reconciliation data and system transaction data to two collections of the Redis cluster according to the Redis comparison specified format, and then compare the collections and generate the reconciliation results.

[0075] In this embodiment, the comparison of the sets specifically includes the following steps:

[0076] Compare the two sets, get the intersection, and store it in the Redis intersection set;

[0077] Compare the two sets, get the difference set, and store it in the Redis difference set;

[0078] Get the data in the Redis intersection and update the reconciliation data of the corresponding data to reconciliation completed;

[0079] Get the data in the Redis difference set and determine what kind of difference the corresponding data belongs to. If it is a difference that the system can correct by itself, the system will correct it by itself and update the reconciliation data of the corresponding data to reconciliation completed;

[0080] If the error cannot be corrected by itself, the reconciliation data of the corresponding data will be updated as a reconciliation error and written into the reconciliation error table. The error message will be pushed to the relevant personnel through the message system for manual processing.

[0081] After the reconciliation results are generated, the reconciliation data and error data of each channel on that day are counted, and the data is processed and a report on the reconciliation results of the day is issued to facilitate operations personnel to check the reconciliation status.

[0082] XXL-JOB is a distributed task scheduling platform with a built-in visual interface. It can dynamically modify task information, status, and start and stop tasks. It also supports elastic scaling, routing strategies, failover, congestion handling, task timeout control, task failure warnings, sharding broadcast tasks, dynamic sharding, and event triggering. This embodiment is based on the XXL-JOB distributed scheduling framework, which has the advantages of low development difficulty, less code intrusion, and prevention of single point failures compared to traditional scheduling frameworks, greatly improving development and implementation efficiency.

[0083] As a pure Java, distributed, queue-based open source messaging middleware, RocketMQ supports transactional messages, sequential messages, batch messages, scheduled messages, delayed messages, message backtracking, etc. This embodiment is based on the message processing mechanism of RocketMQ delay queue, which is more timely than the traditional scheduling mechanism and prevents repeated polling and scanning of the database from putting pressure on the system and database performance.

[0084] As a NoSQL database, Redis provides database solutions to solve a series of problems such as high concurrency, high expansion, and large data storage. At the same time, Redis is a database structure server based on Key-Value storage. The data structure types it supports include: string, list, hash table, set, ordered set, etc. In order to ensure the efficiency of reading, Redis stores all data objects in memory. It can support periodic writing of updated data to disk files, and it also provides intersection and difference sets, as well as some operations of sorting in different ways. This embodiment is based on Redis data comparison. Compared with the traditional method of comparing one by one through the database, when the amount of data is huge, the time consumption is significantly reduced, and the pressure on the database is also reduced, reducing the deployment cost.

[0085] Example 2

[0086] like Figures 2 to 6 As shown, this embodiment provides a reconciliation method based on Redis, which includes the following steps:

[0087] S1. Relying on the scheduling framework, the reconciliation task initialization process is triggered regularly, and the reconciliation tasks are generated according to the reconciliation file configuration templates of each payment channel.

[0088] Specifically, in this embodiment, relying on the XXL-JOB distributed scheduling framework, it is checked whether the task has arrived at the specified time, and the end-of-day reconciliation task initialization is triggered after the specified time is reached. The end-of-day reconciliation task data is generated according to the reconciliation file configuration template of each payment channel (such as Alipay, WeChat, UnionPay, various banks, etc.), and written into the database.

[0089] S3. Write the reconciliation task into the delay queue. When the pull rule is met, the delay queue is triggered to send the corresponding payment channel reconciliation file pull message to pull the reconciliation file.

[0090] Specifically, the system pushes the corresponding channel reconciliation file pull message to the RocketMQ message middleware. RocketMQ checks whether the message arrives at the specified time. After the specified time arrives, it pushes the specified channel reconciliation file pull message to the system. After receiving the channel reconciliation file pull message, the file is pulled through the corresponding pull interface.

[0091] In this embodiment, if the reconciliation file fails to be pulled, the pulled message is rolled back to the RocketMQ message queue and is triggered again after a delay of 5 minutes. It should be noted that the delay time can be configured.

[0092] S4. Store the pulled reconciliation file into MinIO.

[0093] Specifically, storing the pulled reconciliation file into MinIO storage includes the following steps:

[0094] S41. Calculate the MD5 value of the pulled reconciliation file and use the MD5 value as the MinIO object name;

[0095] S42. Call the MinIO encapsulation method through the object name to check the duplication of the reconciliation file;

[0096] S43. If it already exists, it directly returns that the reconciliation file has been successfully pulled; if it does not exist, the reconciliation file is uploaded to the MinIO specified object bucket for storage through the MinIO upload encapsulation method. After the reconciliation file is successfully uploaded, MinIO returns the object name and then returns that the reconciliation file has been successfully pulled;

[0097] S44. Update the relevant data to associate with the MinIO object name.

[0098] After updating the pull information and associated object names, the file parsing event message is pushed through the RocketMQ message middleware.

[0099] S5. After configuring the reconciliation file type and message format pulled by each channel, convert the reconciliation data of each channel into a unified format.

[0100] Specifically, after the system receives the file parsing event message pushed by RocketMQ, it obtains the corresponding reconciliation file from the MinIO storage. The file parsing engine parses and converts the reconciliation data according to the file channel type, MinIO object bucket name, and MinIO object name.

[0101] In this embodiment, converting the reconciliation data of each channel into a unified format specifically includes the following steps:

[0102] S51. The file parsing engine downloads the corresponding account file through the MinIO object bucket name and MinIO object name, and then converts it into a file stream and pushes it into the file parsing engine.

[0103] S52: The file parsing engine determines whether the file is a Txt file, and if not, converts it into Txt.

[0104] It should be noted that the system has integrated the Apache POI open source component to convert Excel to Csv and then to Txt, and also reserved an interface to implement custom conversion processing.

[0105] S53. After the files are unified into Txt, the file parsing engine obtains the mapping relationship between the reconciliation file fields of the corresponding channel and the unified format fields through parameters such as channel type.

[0106] S54. The parsing factory inside the engine uses the CGLIB dynamic proxy framework to dynamically create the parsing implementation class of the corresponding channel according to the configuration information and parsing template.

[0107] S55. Call the dynamic implementation class to parse the file line by line, convert it into a unified format, and write it into the MySQL database.

[0108] It should be noted that in addition to the MySQL database, other relational databases such as Oracle, SQL Server, DB2, DAMO, Nanda, and Gauss are also applicable.

[0109] If the current parsing template is not suitable for the current channel, you can implement interface parsing, write specific parsing code, and then package it into a Jar file, and upload the Jar file through the back-end management platform; when parsing, the file parsing engine will determine whether there is a custom implementation method for the current channel. If so, the corresponding Class file in the Jar file will be dynamically loaded into the system through the self-encapsulated class loader; then the CGLIB dynamic proxy framework is used to parse the interface proxy custom implementation method to complete the file parsing operation.

[0110] If an error or exception occurs in the current analysis, the current analysis event message is rolled back to the RocketMQ delay queue, and a delay trigger is set. At the same time, the task processing exception message is pushed through the message system to relevant personnel for observation and processing.

[0111] After the file parsing and conversion task is completed, the system checks whether the file parsing and conversion tasks of all channels have been completed. If they have been completed, a system data pull event message is sent to RocketMQ, and the RocketMQ message middleware pushes the next task execution event message.

[0112] S7. Pull the system transaction data corresponding to the reconciliation data of each channel, clean and filter the system transaction data, and store it in a separate reconciliation database.

[0113] Specifically, the system uses ETL tools to pull, clean, and filter the system transaction data of the specified date corresponding to the reconciliation data of each channel, and stores the processed data in a separate reconciliation database, and then sends a data comparison event message to RocketMQ, and the RocketMQ message middleware pushes the data comparison event message.

[0114] S9. Push the corresponding channel reconciliation data and system transaction data to two collections of the Redis cluster according to the specified format of Redis comparison, and then compare the collections and generate reconciliation results.

[0115] Specifically, the system obtains the corresponding channel reconciliation data and system transaction data from the database, generates Redis string data one by one according to the format, and then pushes the data to the Redis Set collection. After calculating the intersection and difference of the data Set collection, the intersection and difference data are returned; the system processes the intersection and difference data, updates the reconciliation data information to the database, and checks whether the reconciliation is completed.

[0116] In this embodiment, the comparison of the sets specifically includes the following steps:

[0117] S91, compare the two sets;

[0118] S92, obtain the intersection and store it in the Redis intersection set; obtain the difference and store it in the Redis difference set, determine what kind of difference the corresponding data belongs to, and if it is a difference that the system can correct itself, the system corrects it itself;

[0119] S93. Obtain the data in the Redis intersection set and the data after the system self-corrects the error in the Redis difference set, and update the reconciliation data of the corresponding data to reconciliation completed; update the reconciliation data of the corresponding data in the Redis difference set that cannot be self-corrected to reconciliation error.

[0120] After updating the reconciliation data information, the system checks whether the reconciliation is completed. If it is completed, it sends a report statistics event message to RocketMQ. RocketMQ message middleware pushes the report statistics event message. The system counts the reconciliation data and error data of each channel on the day, processes the data and generates a reconciliation statistics report, and then saves the statistical data and updates the reconciliation task data to the database to facilitate operators to view the reconciliation status.

[0121] MinIO is a high-performance, distributed object storage service that is compatible with the Amazon S3 cloud storage interface. It is suitable for storing unstructured data such as pictures, audio, video, files, etc. The maximum size of an object file can reach 5TB. At the same time, MinIO is simple to deploy, has good compatibility, supports various platforms, and has massive storage compared to traditional storage. It supports a single object up to 5TB, is compatible with the AmazonS3 interface, has low redundancy, comes with its own error correction algorithm, has high tolerance for disk damage, and has excellent read and write performance. The SpringCloud microservice system used in the payment system of this embodiment is decoupled and zero-intrusive from the payment system when using MinIO file storage. At the same time, when the MinIO cluster uses the default configuration, the system can connect to MinIO to achieve zero-configuration automatic assembly. The AutoConfiguration specification that complies with SpringBoot is used to encapsulate MinIO, which can be quickly deployed and developed in systems that need to use MinIO. This embodiment uses MinIO distributed object storage. Compared with traditional file storage methods, it has flexible expansion, strong disaster tolerance, low maintenance cost, and strong file isolation and permission control.

[0122] Example 3

[0123] like Figure 7 As shown, this embodiment provides a reconciliation system based on Redis, which includes the following modules:

[0124] Task initialization module: used to trigger the reconciliation task initialization process at a scheduled time, and generate reconciliation tasks according to the reconciliation file configuration template of each payment channel.

[0125] In this embodiment, the system relies on the XXL-JOB distributed scheduling framework to trigger the reconciliation task initialization process at regular intervals every morning, and generates reconciliation tasks based on the reconciliation file configuration templates of various payment channels (such as Alipay, WeChat, UnionPay, various banks, etc.).

[0126] The first acquisition module is used to send a corresponding payment channel reconciliation file pull message according to the delay queue to pull the reconciliation file.

[0127] In this embodiment, the end-of-day reconciliation task initialization operation is performed through XXL-JOB timer scheduling, and the pull execution time is generated according to the configuration time of each channel; the pull task and the pull time are written into the RocketMQ delay queue; after the specified pull time is reached, the RocketMQ delay queue triggers the message sending, and sends the corresponding channel pull message; after receiving the channel reconciliation file pull message, the file is pulled through the corresponding pull interface.

[0128] If the reconciliation file pull fails, the pull message is rolled back to the RocketMQ message queue and triggered again after a certain delay. After all the reconciliation file pull tasks are successful, the next task execution event information is pushed through the RocketMQ message middleware.

[0129] File parsing module: used to configure the reconciliation file type and message format pulled by each channel, and convert the reconciliation data of each channel into a unified format.

[0130] In this embodiment, if an error or exception occurs in the current analysis, the current analysis event message is rolled back to the RocketMQ delay queue, and a delay trigger is set. At the same time, the task processing exception message is pushed through the message system to relevant personnel for observation and processing. It should be noted that the system has integrated the Apache POI open source component to convert Excel to Csv and then to Txt, and also reserved an interface to implement custom conversion processing.

[0131] After the file parsing and conversion task is completed, check whether the file parsing and conversion tasks of all channels have been completed. If they have been completed, push the next task execution event message through RocketMQ message middleware.

[0132] The second acquisition module is used to pull the system transaction data corresponding to the reconciliation file pulled by the first acquisition module, and clean and filter it.

[0133] In this embodiment, the system uses the ETL tool to pull the system transaction data corresponding to the reconciliation data of each channel (i.e., the data pulled by the first acquisition module), cleans and filters the system transaction data, and stores it in a separate reconciliation database. After success, the next task execution event message is pushed through the RocketMQ message middleware.

[0134] Reconciliation module: used to push the data acquired by the first acquisition module and the second acquisition module to two sets of the Redis cluster according to the Redis comparison specified format, compare the sets, and complete the reconciliation.

[0135] In this embodiment, the comparison of the sets specifically includes the following steps:

[0136] Compare the two sets, get the intersection, and store it in the Redis intersection set;

[0137] Compare the two sets, get the difference set, and store it in the Redis difference set;

[0138] Get the data in the Redis intersection and update the reconciliation data of the corresponding data to reconciliation completed;

[0139] Get the data in the Redis difference set and determine what kind of difference the corresponding data belongs to. If it is a difference that the system can correct by itself, the system will correct it by itself and update the reconciliation data of the corresponding data to reconciliation completed;

[0140] If the error cannot be corrected by itself, the reconciliation data of the corresponding data will be updated as a reconciliation error and written into the reconciliation error table. The error message will be pushed to the relevant personnel through the message system for manual processing.

[0141] After updating the reconciliation data information, the system checks whether the reconciliation is completed. If it is completed, it sends a report statistics event message to RocketMQ. RocketMQ message middleware pushes the report statistics event message. The system counts the reconciliation data, error data, and data processing of each channel on the day and generates a reconciliation statistics report. It then saves the statistical data and updates the reconciliation task data to the database to facilitate operations personnel to view the reconciliation status.

[0142] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A reconciliation method based on Redis, characterized in that: The following steps are involved: Relying on the scheduling framework, the reconciliation task initialization process is triggered regularly, and reconciliation tasks are generated according to the reconciliation file configuration template of each payment channel; Write the reconciliation task into the delay queue. When the pull rule is met, the delay queue is triggered to send the corresponding payment channel reconciliation file pull message to pull the reconciliation file. After configuring the reconciliation file type and message format pulled by each channel, convert the reconciliation data of each channel into a unified format; Pull the system transaction data corresponding to the reconciliation data of each channel, and clean and filter the system transaction data; The corresponding channel reconciliation data and system transaction data are pushed to two sets of the Redis cluster in the specified format of Redis comparison, and then the sets are compared and the reconciliation results are generated; After pulling the reconciliation file, the pulled reconciliation file is also stored in MinIO, specifically including: Calculate the MD5 value of the pulled reconciliation file and use the MD5 value as the MinIO object name; Call the MinIO encapsulation method through the object name to check the reconciliation file for duplication; If it already exists, it will directly return that the reconciliation file has been successfully pulled; if it does not exist, the reconciliation file will be uploaded to the MinIO specified object bucket for storage through the MinIO upload encapsulation method. After the reconciliation file is successfully uploaded, MinIO returns the object name and then returns that the reconciliation file has been successfully pulled; Update the relevant data to associate with the MinIO object name.

2. The account reconciliation method based on Redis according to claim 1, characterized in that: The conversion of the reconciliation data of each channel into a unified format is performed by a file parsing engine, which parses and converts the reconciliation data according to the reconciliation file channel type, MinIO object bucket name, and MinIO object name.

3. The account reconciliation method based on Redis according to claim 2, characterized in that: The converting of the reconciliation data of each channel into a unified format specifically includes: The file parsing engine downloads the corresponding account file through the MinIO object bucket name and MinIO object name, and then converts it into a file stream and pushes it into the file parsing engine; The file parsing engine determines whether the file is a Txt file, and if not, converts it to Txt; After the files are unified into Txt, the file parsing engine obtains the mapping relationship between the reconciliation file fields of the corresponding channel and the unified format fields; The parsing factory inside the engine uses the dynamic proxy framework to dynamically create the parsing implementation class for the corresponding channel according to the configuration information and parsing template; Call the dynamic implementation class to parse the file line by line and convert it into a unified format.

4. The account reconciliation method based on Redis according to claim 3, characterized in that: The method further comprises: If the current parsing template is not applicable to the current channel, write specific parsing code, package it into a Jar file, and upload it; The file parsing engine determines whether there is a custom implementation method in the current channel during parsing. If so, the corresponding Class file in the Jar file is loaded into the system through the self-encapsulated class loader; The file parsing operation is completed by customizing the implementation method for the parsing interface proxy through the dynamic proxy framework.

5. The account reconciliation method based on Redis according to claim 1, characterized in that: The comparing the sets and generating the reconciliation results comprises: Compare the two sets, get the intersection, and store it in the Redis intersection set; Compare the two sets, get the difference set, and store it in the Redis difference set; Get the data in the Redis intersection and update the reconciliation data of the corresponding data to reconciliation completed; Get the data in the Redis difference set and determine what kind of difference the corresponding data belongs to. If it is a difference that the system can correct by itself, the system will correct it by itself and update the reconciliation data of the corresponding data to reconciliation completed; If self-correction is not possible, the reconciliation data of the corresponding data will be updated as a reconciliation error and written into the reconciliation error table.

6. The account reconciliation method based on Redis according to claim 1, characterized in that: If the pulling of the reconciliation file fails, the pulling message is rolled back to the delay queue message queue and triggered again after a delay.

7. The account reconciliation method based on Redis according to claim 1, characterized in that: After comparing the sets and generating the reconciliation results, it also includes issuing a reconciliation result report: counting the reconciliation data and error data of each channel, processing the data and generating a report.

8. The Redis-based account reconciliation method according to any one of claims 1 to 7, characterized in that: Each step of the method is equipped with corresponding monitoring and event alarm.

9. A reconciliation system based on Redis, characterized in that: Includes the following modules: Task initialization module: used to trigger the reconciliation task initialization process on a regular basis and generate reconciliation tasks according to the reconciliation file configuration templates of each payment channel; The first acquisition module is used to send a corresponding payment channel reconciliation file pull message according to the delay queue, pull the reconciliation file and store the pulled reconciliation file into MinIO; the pulling reconciliation file is stored in MinIO, specifically including: Calculate the MD5 value of the pulled reconciliation file and use the MD5 value as the MinIO object name; Call the MinIO encapsulation method through the object name to check the reconciliation file for duplication; If it already exists, it will directly return that the reconciliation file has been successfully pulled; if it does not exist, the reconciliation file will be uploaded to the MinIO specified object bucket for storage through the MinIO upload encapsulation method. After the reconciliation file is successfully uploaded, MinIO returns the object name and then returns that the reconciliation file has been successfully pulled; Update the relevant data to associate with the MinIO object name; File parsing module: used to configure the reconciliation file type and message format pulled by each channel, and convert the reconciliation data of each channel into a unified format; The second acquisition module is used to pull the system transaction data corresponding to the reconciliation file pulled by the first acquisition module, and clean and filter it; Reconciliation module: used to push the data acquired by the first acquisition module and the second acquisition module to two sets of the Redis cluster according to the Redis comparison specified format, compare the sets, and complete the reconciliation.

Images