A method for reconciling massive data based on large memory

By introducing random forest model technology into the multi-channel management module and large-memory computing module, the problem of insufficient complexity and real-time in multi-channel data reconciliation is solved, efficient and accurate reconciliation processing is achieved, and manual intervention is reduced. Database dependence improves the adaptability and operational efficiency of the system.

CN118761747BActive Publication Date: 2024-11-26BEIJING HUITONG JINCAI INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411253374.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-26
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

In the process of multi-channel data reconciliation, existing systems face insufficient complexity and real-timeness caused by the diversity of data formats and transmission methods, resulting in low reconciliation efficiency, increased manual intervention, and insufficient timeliness on database resources. It cannot meet the real-time processing needs of modern payment services.

Method used

Adopt massive data reconciliation method based on large memory, and realize intuitive configuration and efficient entry through visualization technology of multi-channel management modules. Combined with large memory computing module and random forest model technology, the precise management and real-time processing of multi-channel data is achieved, reducing Manual operation to optimize operational processes.

Benefits of technology

Improve reconciliation efficiency and real-time, reduce manual operation and labor costs, realize efficient processing and accuracy of massive data, support the processing of structured and unstructured data, break the dependence on traditional databases, and improve the System adaptability and response speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761747B_ABST
    Figure CN118761747B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for reconciling massive data based on large memory, which relates to the technical field of reconciliation of massive data, and includes the following steps: S1, realizing intuitive configuration and efficient input of multi-channel related information through a multi-channel management module; S2, adopting a batch processing mechanism through an initialization module to efficiently convert the configuration into a monitorable task with key attributes; S3, implementing a multi-channel sub-library and sub-table strategy through a channel data module to store recent key data, and importing other data into a large data warehouse. The present invention realizes multi-channel data management through visualization technology, reduces access costs and improves system response speed. Utilizing large memory computing module technology, the efficiency and real-time performance of data reconciliation are improved, efficient processing of structured and unstructured data is supported, and dependence on traditional databases is reduced. The introduction of large model technology reduces manual operations and improves work efficiency and accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of massive data reconciliation, and in particular to a massive data reconciliation method based on a large memory. Background Art

[0002] In the modern business environment, as a system enterprise that handles payment business links, it is usually necessary to process massive amounts of data from multiple channels (such as Alipay, WeChat, banks, and third-party payment systems such as YeePay and KuaiBill). These data reconciliations are crucial to ensuring the accuracy of corporate financial statements, preventing financial risks, and optimizing operational efficiency. With the rapid growth of data volume and the diversification of channels, system enterprises face severe challenges in ensuring the accuracy and consistency between their internal order data and data obtained from external channels. Especially in the process of multi-channel data reconciliation, the data formats and transmission methods of different channels are different, which not only increases the complexity of data integration and comparison, but also puts higher requirements on the speed and accuracy of data processing.

[0003] Currently, most companies still rely on traditional databases (such as Oracle) to implement reconciliation functions. Figure 2 As shown. These databases use system memory resources to compare data from different channels. Although they can provide certain computing power, they are prone to excessive consumption of memory and CPU resources when faced with massive amounts of data. In addition, traditional reconciliation is usually performed on a TN day and follows an offline data processing process. This method has poor real-time performance and cannot meet the real-time processing requirements of modern payment services. In terms of data difference processing, since there may still be a large number of account differences after the reconciliation is completed, traditional solutions often rely on manual intervention to eliminate these differences, resulting in a large workload and low efficiency.

[0004] However, with the expansion of business scale and the increase of data complexity, the existing reconciliation system based on traditional databases such as Oracle has gradually exposed some limitations, such as insufficient timeliness of multi-channel data processing, poor flexibility in accessing new channels, and high dependence on database resources. In the scenario of reconciliation of differential data, the existing methods usually require manual intervention of operators to perform complex verification and processing, which not only increases the workload, but may also lead to inaccurate accounting processing due to human errors. Therefore, how to improve reconciliation efficiency while reducing manual intervention has become a key issue that payment business system companies need to solve urgently.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a method for reconciling massive data based on large memory, which reduces the channel access cost, improves the adaptability and response speed of the system, and realizes the precise management of multi-channel data through the visualization technology of the multi-channel management module. The large memory computing module technology is used to improve the efficiency and real-time performance of reconciliation of massive data, ensure the comprehensiveness and integrity of the data, and break the dependence on traditional databases, supporting the efficient processing of structured and unstructured data. In addition, the introduction of large model technology based on random forest reduces manual operations, improves work efficiency and accuracy of account processing, optimizes the operation process, and reduces the labor cost of the enterprise to solve the problems in the above-mentioned background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a method for reconciling massive data based on a large memory, comprising the following steps:

[0008] S1. Realize intuitive configuration and efficient input of multi-channel related information through the multi-channel management module;

[0009] S2, adopting a batch processing mechanism through the initialization module to efficiently convert the configuration into monitorable tasks with key attributes;

[0010] S3. Implement a multi-channel database and table sharding strategy through the channel data module to store recent key data, and import other data into a large data warehouse;

[0011] S4. Real-time and non-real-time data are quickly imported through the real-time data module and the offline data module. Real-time data is imported instantly through uninterrupted data flow to ensure the freshness and immediacy of data. For non-real-time data, a sharding machine is used to extract data through a multi-process and multi-threaded mechanism.

[0012] S5. Reconcile the data of both parties through the reconciliation module to achieve fast and accurate reconciliation of the data of both parties and ensure the consistency and integrity of the data;

[0013] S6. The difference data generated in the reconciliation module is resolved through the difference cancellation module, and a secondary reconciliation process is performed based on the query difference information and the reconciliation identifier.

[0014] Preferably, in step S1, intuitive configuration and efficient input of multi-channel related information are achieved through a multi-channel management module, and the specific steps are:

[0015] First, collect basic channel information through a visual interface;

[0016] Secondly, for the detailed information of the channel data, the file name, format, content format and mapping of each field in the content are accurately configured, and flexible configuration options are provided for the content in the file that needs to be escaped.

[0017] Preferably, the text data provided by the channel is written into a file, and the original channel providing the file is renamed in the format of "channel abbreviation_business type_time"; the matched compressed files are decompressed, including ordinary compressed files and special encrypted compressed files.

[0018] Preferably, in step S2, based on the scale of historical data, a channel-based divide-and-conquer strategy is adopted to implement precise task management, classify tasks according to their sources, and based on the preset reconciliation frequency, to ensure the concurrency security of the system and prevent repeated execution of tasks, all subsequent operations will rely on the unique ID of the task as the key of the distributed lock, thereby implementing functional interception and protection.

[0019] Preferably, in step S3, with the help of the ElasticJob framework, the other party's data is processed and stored in real time through the interface. For channels that need to provide data independently, the multi-sharding mechanism of ElasticJob is used to quickly import data.

[0020] Preferably, the entire data import process includes the following key steps:

[0021] Accurately collect unstructured data from multiple channels to ensure the comprehensiveness and timeliness of the data;

[0022] Before importing, unstructured data should be carefully cleaned to remove duplicate, incomplete or erroneous data, laying a solid foundation for subsequent processing;

[0023] The key to converting unstructured data into structured data is to accurately parse the collected data based on the configuration information provided by the initialization module, extract valuable information, and convert it into a structured format for subsequent processing and analysis;

[0024] Before data is imported, secondary verification is performed to ensure the accuracy and completeness of the data;

[0025] Finally, by using multi-threaded concurrent technology, the processed data can be efficiently imported into the data storage center and reconciliation module to meet the needs of real-time processing and business analysis.

[0026] Preferably, the process of data mapping is:

[0027] First, the files to be processed are counted and child threads are created based on the principle that one thread processes one million rows to ensure that multiple tasks are carried out simultaneously. The mapping relationships of the fields configured in the channel management module are retrieved, and the data in each row is separated according to the corresponding rules. Ordinary string types are used directly, date and time types are formatted, digital amount types are converted to yuan or standardized, data that needs to be desensitized is desensitized by intermediate data or specified length, data that needs to be intercepted and spliced ​​is removed by removing empty spaces and escapes before use, and enumerations are interpreted according to predetermined enumeration types before use.

[0028] Preferably, in step S4, the relevant data of the party are all structured data stored in the database and do not need to be processed. For offline data, the database streaming query technology is adopted to return the query results in the form of a stream. For the real-time data of the party, the required data is quickly and accurately imported into the reconciliation module to ensure the integrity and availability of the data.

[0029] Preferably, in step S5, a high-performance memory computing cluster is constructed to support large-scale data computing needs, which is achieved by the following steps:

[0030] The first step is to verify the reconciliation input;

[0031] Step 2: Clear the reconciliation cache and difference information. First, clear the key value in the inherent memory computing cluster, and then clear the information in the difference table according to the task ID.

[0032] Step 3: Query the total number of transactions, obtain the total number of transactions of the party and the channel, and calculate the grouping according to the total number of transactions and the principle of 100,000 transactions as one group;

[0033] Step 4: Intermediate verification: if the transaction records of both parties are empty, the reconciliation is successful and the status is updated;

[0034] Step 5: Cache preparation: record the transaction in the cache key value prefix, generate the cache key to be used later, set the expiration time for the generated key and record it;

[0035] Step 6: Prepare reconciliation data, obtain the transaction flow of the party, and store it in the cache; obtain the transaction flow of the channel party, and store it in the cache;

[0036] Step 7: Compare and analyze the differences. Perform positive and negative comparisons based on the positive and negative account switches. Compare the two sides to find out the missing parts and content differences. The other side compares with the two sides to find out the missing parts and content differences. Merge the differences between the two sides. Then, remove duplicates based on the reconciliation mark and count the total number of generated differences. Then, determine whether to block the program based on the number of difference blocks. The reconciliation mark after removal of duplicates is stored in the cache.

[0037] Step 8. Update the status of successful reconciliation. Update the task table and change the task status to reconciled. The update conditions are the status "unreconciled" and the version number field of the optimistic lock.

[0038] Step 9: Update the reconciliation status of the bank statement and the bank statement of the party to complete, where the update condition is that the status is "reconciliation in progress";

[0039] Step 10. Clear the used memory computing cluster groups. Each group of keys will be cleared after a delay of delayTime+i seconds. After the reconciliation is successfully completed, the redis group key expiration time will be reset.

[0040] Preferably, in step S6, in the process of realizing the automatic account balancing function through the difference settlement module, the pre-trained advanced model is used to process the newly input account data. When the new account data enters the system, it will be directly input into these carefully trained and optimized models to obtain accurate prediction results or classification labels. The specific steps are as follows:

[0041] The first step is to select random forest as the model for data processing tasks;

[0042] Step 2: Before using the model, make sure the data is prepared and adapted to the model’s input requirements.

[0043] Step 3: Reuse TensorFlow to load the weights and parameters of the model;

[0044] Step 4: Fine-tune the model to adapt to the balance task and data, and fine-tune the model using historical labeled data to optimize the model's performance on the task;

[0045] Step 5. Use the model's prediction results to classify or mark new data, and mark and update the data that needs to be settled. In the initial stage of use, manual secondary confirmation is required, and manual secondary confirmation will be gradually phased out in the later stages.

[0046] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0047] The present invention realizes intuitive configuration and efficient input of multi-channel information through the visualization technology of the multi-channel management module. This technology solves the problem of high channel access cost in traditional systems, especially the limitations of previously relying on hard-coded or program configuration files to process channel configuration information. Through the visual interface, operators can not only monitor the data flow of each channel in real time, but also customize the management interface according to their own needs, set monitoring indicators and alarm thresholds, so as to achieve accurate management and optimization of different channels. This flexible configuration capability greatly reduces the cost of system development and maintenance, improves the adaptability and response speed of the system, and provides an efficient solution for enterprises to quickly access and manage multi-channel data.

[0048] The present invention utilizes large memory computing module technology, through powerful computing power and efficient algorithms, to quickly complete data comparison and verification, greatly improving the efficiency of reconciliation. In addition, the system also supports the real-time extraction and processing of the latest data of all parties to ensure the real-time and accuracy of reconciliation. By combining the two methods of offline data reconciliation and real-time data reconciliation supported by large memory, the system can quickly respond to data changes and ensure the comprehensiveness and integrity of the data. This multi-level data processing mechanism provides reliable data support for the operational efficiency and decision-making quality of the enterprise, and effectively reduces the risk of manual operation.

[0049] The present invention breaks the reliance on databases in traditional reconciliation processes through large memory computing module technology, creates a new data processing model, and realizes unlimited expansion and efficient processing of massive data. The large memory computing module can not only process structured data, but also flexibly load unstructured data, make full use of the computing power of memory, and quickly process and analyze massive data. This new data processing model greatly improves the processing power and scalability of the system, ensuring that enterprises can still operate efficiently and stably when facing data growth and complex business scenarios.

[0050] The present invention introduces advanced large model technology based on random forests, and has achieved remarkable results in reducing manual participation. Through the intelligent design of the multi-channel management module and the conventional difference elimination module, the system can automatically handle most reconciliation tasks, and manual intervention is only required in necessary links. This automated processing method significantly reduces the workload of manual operations in daily operations, improves work efficiency, and reduces the possibility of human errors. At the same time, through the automatic account balancing function, the system can quickly and accurately handle account differences, ensure the accuracy and consistency of accounts, and provide users with better quality and more efficient services. This method of optimizing operational processes not only meets the growing business needs, but also saves labor costs for enterprises and improves overall operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0052] Figure 1 The present invention is a method flow chart of a method for reconciling massive data based on large memory.

[0053] Figure 2 It is a schematic diagram of a traditional database computing center represented by Oracle in the prior art. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0055] The present invention provides Figure 1 A method for reconciling massive data based on a large memory is shown, comprising the following steps:

[0056] S1. Realize intuitive configuration and efficient input of multi-channel related information through the multi-channel management module;

[0057] First, collect basic channel information through a visual interface;

[0058] Configure the merchant ID, interface path, interface mode, and corresponding interface protocol and format and other key attributes in detail;

[0059] Secondly, for the detailed information of the channel data, the file name, format, content format and mapping of each field in the content are accurately configured, and flexible configuration options are provided for the content in the file that needs to be escaped.

[0060] The main contents are as follows:

[0061] For the text data provided by the channel, write the text data into a file, rename the original file channel, and use the format of "channel abbreviation_business type_time" to match the compressed files, ordinary files, and special compressed and encrypted files of the channel. The compressed files matched will be decompressed, including ordinary compressed files and special encrypted compressed files. There are zip and gz types. Finally, they are processed into ordinary files such as txt, csv, and xls.

[0062] Match the file content format rules of the file, including the number of lines that the file header occupies, the characters that separate the fields, etc. For example, configure the number of lines to skip the file header line, the vertical bar separator to separate the fields, and the fixed length of 10 characters to separate the contents of each field.

[0063] Map the contents of each field, and support defining the field content type as string, date, digital unit conversion, data desensitization, enumeration escape, partial field interception, multi-field splicing, etc.

[0064] Assemble the unique identification field, and form a global unique identification according to the unique identification of each channel, the business type, and the relevant order number.

[0065] Finally, a full-process batch information management function is provided, and users can intuitively set the number of threads for channel data import and the frequency of querying data results to achieve more efficient data processing process management.

[0066] S2. Through the initialization module, a batch processing mechanism is adopted to efficiently convert the configuration into monitorable tasks with key attributes such as date. In order to ensure the stability and efficiency of the system, based on the scale of historical data, a divide-and-conquer strategy based on channels is adopted to implement precise task management. Furthermore, tasks are classified according to their sources, covering various types such as Alipay, WeChat, Cloud QuickPass, and other banks, and detailed task record data is generated based on the preset reconciliation frequency (such as daily, hourly, monthly, etc.). To ensure the concurrent security of the system and prevent repeated execution of tasks, all subsequent operations will rely on the unique ID of the task as the key of the distributed lock, thereby implementing functional interception and protection.

[0067] S3. Implement a multi-channel database and table sharding strategy through the channel data module to store recent key data, and import other data into a large data warehouse;

[0068] To ensure the real-time and high efficiency of data flow processing, the ElasticJob framework is used to process and store the other party's data in real time through the interface (incremental provision). For specific channels that need to provide data independently, the multi-sharding mechanism of ElasticJob (i.e. distributed multi-threaded processing mechanism) is used for fast data import.

[0069] The entire data import process has been carefully designed and optimized, including the following key steps:

[0070] Data collection: Accurately collect unstructured data from multiple channels to ensure the comprehensiveness and timeliness of the data.

[0071] Data cleaning: Before importing, unstructured data is carefully cleaned to remove duplicate, incomplete or erroneous data, laying a solid foundation for subsequent processing.

[0072] Data parsing and mapping: This step is the key to converting unstructured data into structured data. Based on the configuration information provided by the initialization module, the collected data is accurately parsed, valuable information is extracted, and converted into a structured format for subsequent processing and analysis.

[0073] The main process of data mapping is:

[0074] First, the files to be processed are counted and sub-threads are created based on one thread processing one million lines to ensure that multiple tasks are carried out simultaneously. The mapping relationships of each field configured in the channel management module are taken out, and the data of each line is separated according to the corresponding rules. For direct use of ordinary string types, date formatting is performed for date and time types, yuan conversion or standardized processing is performed for digital amount types, intermediate data or specified length is desensitized for data that needs to be desensitized, and for data that needs to be intercepted and spliced, empty and escaped are removed before use, and enumerations are interpreted according to the pre-defined enumeration types before use.

[0075] Basic data verification: Before data is imported, secondary verification is performed to ensure the accuracy and completeness of the data. Any errors or inconsistencies will be repaired in a timely manner to ensure data quality.

[0076] Data import: Finally, using multi-threaded concurrent technology, the processed data is efficiently imported into the data storage center and reconciliation module to meet the needs of real-time processing and business analysis.

[0077] S4. Real-time and non-real-time data are quickly imported through our real-time data module and offline data module respectively. Real-time data is imported instantly through uninterrupted data flow to ensure the freshness and immediacy of the data. For non-real-time data, a sharding machine is used to extract data through a multi-process and multi-threaded mechanism to greatly improve the import efficiency.

[0078] All relevant data on our side are structured data stored in the database, and no formatting or other processing is required. For offline data, database streaming query technology is adopted, which allows the database system to return query results in the form of streams instead of returning all results at once. It is N times faster than conventional paging queries. In contrast, streaming queries read data from the database server in batches, and the amount of data read each time is not a row, but part of the data is read from the TCP stream channel as needed. This difference makes streaming queries more efficient when processing large amounts of data because it avoids memory overflow (OOM) problems caused by excessive data volume.

[0079] For our real-time data, we use efficient data migration tools, API (application programming interface) or manual methods to quickly and accurately import the required data into the reconciliation module to ensure data integrity and availability.

[0080] S5. Reconcile the data of both parties through the reconciliation module to achieve fast and accurate reconciliation of the data of both parties and ensure the consistency and integrity of the data;

[0081] To support large-scale data computing, the first and most critical step is to build a high-performance memory computing cluster. This will ensure that the system can efficiently and stably handle huge data loads. To achieve the goals of this module, the following steps are required:

[0082] The first step is to verify the reconciliation input parameters. First, determine the reconciliation task status, determine the positive and negative reconciliation switches of the reconciliation task, etc. After all the above verifications are passed, the reconciliation status is updated to Reconciling.

[0083] The second step is to clear the reconciliation cache and difference information. First, clear the key value in the inherent memory computing cluster, and then clear the information in the difference table according to the task ID.

[0084] Step 3: Query the total number of transactions, obtain the total number of transactions on your side, and the total number of transactions on the channel side, and calculate the grouping based on the total number of transactions and the principle of 100,000 (configurable) transactions as a group.

[0085] Step 4: Intermediate verification: if the transaction flow of both parties (i.e. the assembly unique identification field) is empty, the reconciliation is successful and the status is updated.

[0086] Step 5: Cache preparation: add the transaction record to the cache key value prefix to generate the cache key used later. Set the expiration time (default 2 days) for the generated key and record it.

[0087] Step 6: Prepare reconciliation data, obtain the transaction flow of the party, and store it in the cache. Obtain the transaction flow of the channel party, and store it in the cache.

[0088] Step 7: Compare and analyze the differences. Perform positive and negative comparisons based on the positive and negative account switches. Compare with the other party to find out the missing parts and content differences (difference set). The other party compares with ours to find out the missing parts and content differences (difference set). Combine the differences between ours and the other party (union set). Then, remove duplicates based on the reconciliation mark and count the total number of differences generated. Then, determine whether to block the program based on the number of difference blocks. The reconciliation mark after removal of duplicates is stored in the cache.

[0089] Step 8. Update the status of successful reconciliation. Update the task table and change the task status to reconciled. The update conditions are the status "unreconciled" and the version number field of the optimistic lock.

[0090] Step 9. Update the reconciliation status of the transaction sheet and the transaction reconciliation status of the party to complete, where the update condition is that the status is "reconciling".

[0091] Step 10: Clear the used memory computing cluster groups. Each group key is cleared after a delay of delayTime+i seconds. After the reconciliation is successfully completed, reset the redis group key expiration time.

[0092] S6. Resolve the difference data generated in the reconciliation module through the difference cancellation module, and perform secondary reconciliation processing based on the query difference information and reconciliation identifier;

[0093] It is mainly used to handle scenarios such as inconsistent financial dates and inconsistent status, and to write off the records that have been successfully matched by this module. This module will attempt to match multiple times over a period of time, and data that cannot be processed within a week or a period of time will be transferred to the balance module for processing.

[0094] In the process of implementing the automatic reconciliation function through the difference settlement module, pre-trained advanced models are fully utilized to process newly input account data. When new account data enters the system, it is directly input into these carefully trained and optimized models to obtain accurate prediction results or classification labels. This process is not only fast, but also ensures the reliability and consistency of the results.

[0095] The first step is to select random forest as the model for the data processing task.

[0096] Step 2: Data preparation: Before using the model, you need to ensure that the data is prepared and adapted to the model's input requirements. This includes steps such as data cleaning, standardization, and feature engineering. Ensure that historical labels are correctly processed and compatible with the input format.

[0097] Step 3: Load the pre-trained model: reuse TensorFlow to load the weights and parameters of the model.

[0098] Step 4: Fine-tune the model: Fine-tune the model to adapt to the balance task and data. Fine-tuning involves training the model using historical labeled data to optimize the model's performance on the task.

[0099] Step 5. Data processing: Use the model's prediction results to classify or mark new data, and mark and update the data that needs to be settled. In the initial stage of use, manual secondary confirmation is required, and manual secondary confirmation will be gradually phased out in the later stage.

[0100] The present invention realizes intuitive configuration and efficient input of multi-channel information through the visualization technology of the multi-channel management module. This technology solves the problem of high channel access cost in traditional systems, especially the limitations of previously relying on hard-coded or program configuration files to process channel configuration information. Through the visual interface, operators can not only monitor the data flow of each channel in real time, but also customize the management interface according to their own needs, set monitoring indicators and alarm thresholds, so as to achieve accurate management and optimization of different channels. This flexible configuration capability greatly reduces the cost of system development and maintenance, improves the adaptability and response speed of the system, and provides an efficient solution for enterprises to quickly access and manage multi-channel data.

[0101] The present invention utilizes large memory computing module technology, through powerful computing power and efficient algorithms, to quickly complete data comparison and verification, greatly improving the efficiency of reconciliation. In addition, the system also supports the real-time extraction and processing of the latest data of all parties to ensure the real-time and accuracy of reconciliation. By combining the two methods of offline data reconciliation and real-time data reconciliation supported by large memory, the system can quickly respond to data changes and ensure the comprehensiveness and integrity of the data. This multi-level data processing mechanism provides reliable data support for the operational efficiency and decision-making quality of the enterprise, and effectively reduces the risk of manual operation.

[0102] The present invention breaks the reliance on databases in traditional reconciliation processes through large memory computing module technology, creates a new data processing model, and realizes unlimited expansion and efficient processing of massive data. The large memory computing module can not only process structured data, but also flexibly load unstructured data, make full use of the computing power of memory, and quickly process and analyze massive data. This new data processing model greatly improves the processing power and scalability of the system, ensuring that enterprises can still operate efficiently and stably when facing data growth and complex business scenarios.

[0103] The present invention introduces advanced large model technology based on random forests, and has achieved remarkable results in reducing manual participation. Through the intelligent design of the multi-channel management module and the conventional difference elimination module, the system can automatically handle most reconciliation tasks, and manual intervention is only required in necessary links. This automated processing method significantly reduces the workload of manual operations in daily operations, improves work efficiency, and reduces the possibility of human errors. At the same time, through the automatic account balancing function, the system can quickly and accurately handle account differences, ensure the accuracy and consistency of accounts, and provide users with better quality and more efficient services. This method of optimizing operational processes not only meets the growing business needs, but also saves labor costs for enterprises and improves overall operational efficiency.

[0104] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for reconciling massive data based on large memory, characterized in that: The following steps are involved: S1. Realize intuitive configuration and efficient input of multi-channel related information through the multi-channel management module; In step S1, intuitive configuration and efficient input of multi-channel related information are achieved through the multi-channel management module, and the specific steps are as follows: First, collect basic channel information through a visual interface; Secondly, for the detailed information of the channel data, the file name, format, content format and mapping of each field in the content are precisely configured, and flexible configuration options are provided for the content in the file that needs to be escaped; S2, adopting a batch processing mechanism through the initialization module to efficiently convert the configuration into monitorable tasks with key attributes; In step S2, based on the scale of historical data, a divide-and-conquer strategy based on channels is adopted to implement precise task management, classify tasks according to their sources, and based on the preset reconciliation frequency, to ensure the concurrent security of the system and prevent repeated execution of tasks, all subsequent operations will rely on the unique ID of the task as the key of the distributed lock, thereby implementing functional interception and protection; S3. Implement a multi-channel database and table sharding strategy through the channel data module to store recent key data, and import other data into a large data warehouse; In step S3, with the help of the ElasticJob framework, the other party's data is processed and stored in real time through the interface. For channels that need to provide data independently, the multi-sharding mechanism of ElasticJob is used to quickly import data; The entire data import process includes the following key steps: Accurately collect unstructured data from multiple channels to ensure the comprehensiveness and timeliness of the data; Before importing, unstructured data should be carefully cleaned to remove duplicate, incomplete or erroneous data, laying a solid foundation for subsequent processing; The key to converting unstructured data into structured data is to accurately parse the collected data based on the configuration information provided by the initialization module, extract valuable information, and convert it into a structured format for subsequent processing and analysis; Before data is imported, secondary verification is performed to ensure the accuracy and completeness of the data; Finally, using multi-threaded concurrent technology, the processed data is efficiently imported into the data storage center and reconciliation module to meet the needs of real-time processing and business analysis; The process of data mapping is: First, the files to be processed are counted and sub-threads are created based on one thread processing one million lines to ensure that multiple tasks are carried out simultaneously. The mapping relationships of the fields configured in the channel management module are taken out, and the data of each line is separated according to the corresponding rules. For ordinary string types, they are used directly, for date and time types, date formatting is performed, for digital amount types, yuan conversion or standardized processing is performed, for data that needs to be desensitized, intermediate data or specified length is desensitized, for data that needs to be intercepted and spliced, empty and escaped are removed before use, and for enumerations, they are interpreted according to the pre-defined enumeration type before use; S4. Real-time and non-real-time data are quickly imported through the real-time data module and the offline data module. Real-time data is imported instantly through uninterrupted data flow to ensure the freshness and immediacy of data. For non-real-time data, a sharding machine is used to extract data through a multi-process and multi-threaded mechanism. In step S4, the relevant data of the party are all structured data stored in the database and do not need to be processed. For offline data, the database streaming query technology is adopted to return the query results in the form of a stream. For the real-time data of the party, the required data is quickly and accurately imported into the reconciliation module to ensure the integrity and availability of the data; S5. Reconcile the data of both parties through the reconciliation module to achieve fast and accurate reconciliation of the data of both parties and ensure the consistency and integrity of the data; In step S5, a high-performance memory computing cluster is constructed to support large-scale data computing. This is achieved by the following steps: The first step is to verify the reconciliation input; Step 2: Clear the reconciliation cache and difference information. First, clear the key value in the inherent memory computing cluster, and then clear the information in the difference table according to the task ID. Step 3: Query the total number of transactions, obtain the total number of transactions of the party and the channel, and calculate the grouping according to the total number of transactions and the principle of 100,000 transactions as one group; Step 4: Intermediate verification: if the transaction records of both parties are empty, the reconciliation is successful and the status is updated; Step 5: Cache preparation: record the transaction in the cache key value prefix, generate the cache key to be used later, set the expiration time for the generated key and record it; Step 6: Prepare reconciliation data, obtain the transaction flow of the party, and store it in the cache; obtain the transaction flow of the channel party, and store it in the cache; Step 7: Compare and analyze the differences. Perform positive and negative comparisons based on the positive and negative account switches. Compare the two sides to find out the missing parts and content differences. The other side compares with the two sides to find out the missing parts and content differences. Merge the differences between the two sides. Then, remove duplicates based on the reconciliation mark and count the total number of generated differences. Then, determine whether to block the program based on the number of difference blocks. The reconciliation mark after removal of duplicates is stored in the cache. Step 8. Update the status of successful reconciliation. Update the task table and change the task status to reconciled. The update conditions are the status "unreconciled" and the optimistic lock version number field. Step 9: Update the reconciliation status of the transaction sheet and the transaction reconciliation status of the party to complete, where the update condition is that the status is "reconciliation in progress"; Step 10. Clear the used memory computing cluster groups. Each group key is cleared after a delay of delayTime+i seconds. After the reconciliation is successfully completed, reset the redis group key expiration time. S6. Resolve the difference data generated in the reconciliation module through the difference cancellation module, and perform secondary reconciliation processing based on the query difference information and reconciliation identifier; In step S6, the pre-trained advanced models are used to process the newly input account data in the process of realizing the automatic account balance function through the difference settlement module. When new account data enters the system, it will be directly input into these carefully trained and optimized models to obtain accurate prediction results or classification labels. The specific steps are as follows: The first step is to select random forest as the model for data processing tasks; Step 2: Before using the model, make sure the data is prepared and adapted to the model’s input requirements. Step 3: Reuse TensorFlow to load the weights and parameters of the model; Step 4: Fine-tune the model to adapt to the balance task and data, and fine-tune the model using historical labeled data to optimize the model's performance on the task; Step 5. Use the model's prediction results to classify or mark new data, and mark and update the data that needs to be settled. In the initial stage of use, manual secondary confirmation is required, and manual secondary confirmation will be gradually phased out in the later stages.