Multi-source data processing method and device based on multi-supervision-caliber report output requirement

Abstract

The invention discloses a multi-source data processing method and device based on a multi-supervision-caliber report output demand, which can be used in the technical field of information security, and the method comprises the following steps: carrying out semantic association analysis on different supervision caliber requirements, and determining a wide table field; for each supervision caliber, determining a difference table field of the supervision caliber, and determining a difference field processing rule for calculating the difference table field by the wide table field; determining a report processing rule for calculating the supervision caliber report by the wide table field and the difference table field; processing the data of the multiple data sources according to the wide table fields and storing the data to the wide table; reading data from the wide table, processing the data according to a difference field processing rule, and storing the data in a difference table field; and after a submission request of a target supervision caliber is received, reading data from the wide table, reading data in the difference table, processing the data according to a report processing rule, storing the processed data in a report, and outputting the report to the target supervision caliber. According to the method, multi-supervision-caliber report processing can be efficiently completed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a multi-source data processing method and device based on the reporting requirements of multiple regulatory calibers. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention described in the claims. The description herein is not admitted to be prior art merely by including it in this section.

[0003] Generally, a financial institution is regulated by multiple regulatory agencies. Different regulatory agencies have different data reporting requirements for the management of related transactions, but the designed field ranges are roughly the same. For a financial institution with a relatively wide business scope, there are many types of transaction products involved, and the data structures of the original transaction data of different products are inconsistent. If the multi-source transaction data is directly processed into reporting data of multiple regulatory calibers, for example, for n data sources and m regulatory calibers, n×m times of processing are required, and m copies of data calculated according to different regulatory calibers need to be stored. As the number of calibers increases, the space cost also increases linearly. In addition, due to the wide scope and complex scenarios involved in related transactions, the processed data still needs to be checked and maintained by users. For users, m copies of data need to be maintained in the system, which is a large workload and prone to errors. Summary of the Invention

[0004] Embodiments of the present invention provide a multi-source data processing method based on the reporting requirements of multiple regulatory calibers, which is used to efficiently complete the processing of reporting data of multiple regulatory calibers in the face of multiple related transaction data sources, reduce the storage space, effectively reduce the workload of users, improve the response speed to changes in regulatory rules, and effectively reduce the associated impact between calculations of each caliber, and reduce the impact of a change in the rules of one regulatory caliber on the processing of other regulatory calibers. The method includes:

[0005] Performing semantic association analysis on different regulatory caliber requirements to determine wide table fields, where the wide table fields include fields storing the same data;

[0006] For each regulatory caliber, determine the difference table fields of the regulatory caliber according to the requirements of the regulatory caliber and the wide table fields, and determine the difference field processing rules for calculating the difference table fields from the wide table fields according to the difference table fields and the wide table fields, and store the difference field processing rules in a rule table;

[0007] Determine the report processing rules for calculating the regulatory caliber reports from the wide table fields and the difference table fields, and store them in the rule table;

[0008] Process the data of multiple data sources according to the wide table fields and store them in the wide table;

[0009] Query the rule table to obtain the processing rules for the differential fields of the regulatory caliber. Read data from the wide table, process it according to the queried differential field processing rules, and store it in the differential table fields corresponding to the regulatory caliber;

[0010] After receiving the reporting request for the target regulatory caliber, query the rule table to obtain the report processing rules for the target regulatory caliber. Read data from the wide table and read the data in the differential table corresponding to the target regulatory caliber. Process it according to the queried report processing rules and store it in the report, and output the report to the target regulatory caliber.

[0011] The invention embodiment also provides a multi-source data processing device based on the reporting requirements of multiple regulatory calibers, which is used to efficiently complete the processing of reporting data for multiple regulatory calibers in the face of multiple associated transaction data sources, reduce the storage space, effectively reduce the workload of users, improve the response speed to changes in regulatory rules, and effectively reduce the associated impact between calculations of each caliber, and reduce the impact of a change in the rules of one regulatory caliber on the processing of other regulatory calibers. The device includes:

[0012] A wide table field determination module for performing semantic association analysis on the requirements of different regulatory calibers to determine the wide table fields, where the wide table fields include fields storing the same data;

[0013] A differential table field determination module for, for each regulatory caliber, determining the differential table fields of the regulatory caliber according to the requirements of the regulatory caliber and the wide table fields, and determining the differential field processing rules for calculating the differential table fields from the wide table fields based on the differential table fields and the wide table fields, and storing the differential field processing rules in the rule table;

[0014] A rule determination module for determining the report processing rules for calculating the regulatory caliber report from the wide table fields and the differential table fields and storing them in the rule table;

[0015] A source data processing module for processing the data of multiple data sources according to the wide table fields and storing them in the wide table;

[0016] A wide table data processing module for querying the rule table to obtain the differential field processing rules of the regulatory caliber, reading data from the wide table, processing it according to the queried differential field processing rules, and storing it in the differential table fields corresponding to the regulatory caliber;

[0017] A report generation module for, after receiving the reporting request for the target regulatory caliber, querying the rule table to obtain the report processing rules of the target regulatory caliber, reading data from the wide table, reading the data in the differential table corresponding to the target regulatory caliber, processing it according to the queried report processing rules, storing it in the report, and outputting the report to the target regulatory caliber.

[0018] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the multi-source data processing method based on the multi-regulatory-caliber report output requirement described above is implemented.

[0019] An embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the multi-source data processing method based on the multi-regulatory-caliber report output requirement described above is implemented.

[0020] An embodiment of the present invention further provides a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the multi-source data processing method based on the multi-regulatory-caliber report output requirement described above is implemented.

[0021] In an embodiment of the present invention, semantic association analysis is performed on different regulatory-caliber requirements to determine wide-table fields, where the wide-table fields include fields storing the same data; for each regulatory caliber, according to the requirements of the regulatory caliber and the wide-table fields, determine the difference-table fields of the regulatory caliber, and according to the difference-table fields and the wide-table fields, determine the difference-field processing rules for calculating the difference-table fields from the wide-table fields, and store the difference-field processing rules in a rule table; determine the report processing rules for calculating the regulatory-caliber reports from the wide-table fields and the difference-table fields, and store them in the rule table; process the data of multiple data sources according to the wide-table fields and store them in the wide-table; query the rule table to obtain the difference-field processing rules of the regulatory caliber, read the data from the wide-table, process them according to the queried difference-field processing rules, and store them in the difference-table fields corresponding to the regulatory caliber; after receiving a reporting request for a target regulatory caliber, query the rule table to obtain the report processing rules of the target regulatory caliber, read the data from the wide-table, and read the data in the difference table corresponding to the target regulatory caliber, process them according to the queried report processing rules, and store them in the report, and output the report to the target regulatory caliber. In the above process, for calculations of different calibers, the user only needs to maintain the first-layer wide-table, and for changes in the rules, only needs to maintain the processing rules of each regulatory caliber in the second and third layers. This enables efficient completion of the processing of multi-regulatory-caliber reporting data in the face of multiple associated transaction data sources, reduces storage space, effectively reduces the workload of users, improves the response speed to changes in regulatory rules, and effectively reduces the associated impact between calculations of each caliber, reducing the impact of a change in the rules of one regulatory caliber on the processing of other regulatory calibers, and is convenient for maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0023] Figure 1 It is a flowchart of the multi-source data processing method based on the report output requirements of multiple regulatory calibers in the embodiments of the present invention;

[0024] Figure 2 It is a data flow diagram of the multi-source data processing based on the report output requirements of multiple regulatory calibers in the embodiments of the present invention;

[0025] Figure 3 It is a flowchart of determining the wide table fields in the embodiments of the present invention;

[0026] Figure 4 It is a schematic diagram of job dependencies in the embodiments of the present invention;

[0027] Figure 5 It is a data flow diagram of the wide table, difference table, rule table, and report in the distributed relational database in the embodiments of the present invention;

[0028] Figure 6 It is a schematic diagram of the multi-source data processing device based on the report output requirements of multiple regulatory calibers in the embodiments of the present invention;

[0029] Figure 7 It is a schematic diagram of the computer device in the embodiments of the present invention. Detailed implementation manners

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following further elaborates on the embodiments of the present invention with reference to the drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0031] In the technical solutions of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations.

[0032] It should be noted that in the embodiments of this application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary. The purpose is only to illustrate the feasibility in the implementation of the technical solutions of this application, but it does not mean that the applicant has already or necessarily used this solution.

[0033] Figure 1 It is a flowchart of the multi-source data processing method based on the report output requirements of multiple regulatory calibers in the embodiments of the present invention, including:

[0034] Step 101: Conduct semantic association analysis on different regulatory caliber requirements to determine wide table fields, where the wide table fields include fields storing the same data.

[0035] Step 102: For each regulatory caliber, determine the differential table fields according to the regulatory caliber requirements and the wide table fields, and determine the differential field processing rules for calculating the differential table fields from the wide table fields. Store the differential field processing rules in the rule table.

[0036] Step 103: Determine the report processing rules for calculating the regulatory caliber reports from the wide table fields and the differential table fields, and store them in the rule table.

[0037] Step 104: Process the data from multiple data sources according to the wide table fields and store them in the wide table.

[0038] Step 105: Query the rule table to obtain the differential field processing rules for the regulatory caliber. Read the data from the wide table, process them according to the queried differential field processing rules, and store them in the differential table fields corresponding to the regulatory caliber.

[0039] Step 106: After receiving the reporting request for the target regulatory caliber, query the rule table to obtain the report processing rules for the target regulatory caliber. Read the data from the wide table and read the data in the differential table corresponding to the target regulatory caliber. Process them according to the queried report processing rules and store them in the report, and output the report to the target regulatory caliber.

[0040] In the embodiments of the present invention, for the calculations of different calibers, users only need to maintain the first-layer wide table well, and for the changes in the rules, they only need to maintain the processing rules of each regulatory caliber in the second and third layers. This enables efficient completion of the processing of reporting data for multiple regulatory calibers in the face of multiple associated transaction data sources, reduces the storage space, effectively reduces the workload of users, improves the response speed to changes in regulatory rules, and effectively reduces the associated impact between the calculations of each caliber, reducing the impact of the change of one regulatory caliber rule on the processing of other regulatory calibers, which is convenient for maintenance.

[0041] The following describes each step in detail.

[0042] Analyze the reporting requirements for related party transactions under different regulatory calibers. There are many fields that can be shared, such as the occurrence time, transaction amount, transaction currency, etc. However, there will also be different requirements for some special fields. For example, for a certain period of time, a regulatory agency requires the contract amount of a time deposit to be the principal + expected return instead of the principal. Therefore, the embodiments of the present invention adopt shared fields and differential fields. To reduce the workload of users, lower the storage cost, and facilitate maintenance, the embodiments of the present invention design a three - level data management structure. The first layer is the common basic data layer, which designs a unified data interface. The second layer is differential processing, and the third layer is the report layer. Figure 2 This is the data flow diagram of multi - source data processing based on the report output requirements of multiple regulatory calibers in the embodiments of the present invention. Refer to Figure 2 , the distributed database integrates data from multiple data sources such as deposits, corporate credit lines, and consignment financial management. These data are processed into a wide table of basic data.

[0043] In step 101, perform semantic association analysis on the requirements of different regulatory calibers to determine the wide - table fields, where the wide - table fields include fields storing the same data;

[0044] The wide table contains shared reporting fields required by different regulatory calibers or basic fields required for processing the reporting fields of each caliber. The following gives the steps to determine the wide - table fields.

[0045] Figure 3 This is the flowchart for determining the wide - table fields in the embodiments of the present invention. The specific steps include:

[0046] Step 301, perform semantic analysis on the requirements of different regulatory calibers and obtain the fields required by each regulatory caliber according to the smallest disassembly granularity;

[0047] For example, there are three regulatory calibers A, B, and C. By disassembling the regulatory caliber requirements, the reporting fields of regulatory caliber A can be disassembled into Xa = {i1, i2, i3,..., in, a1, a2, a3,..., an}, the reporting fields of regulatory caliber B can be disassembled into Xb = {i1, i2, i3,..., in, b1, b2, b3,..., bn}, and the reporting fields of regulatory caliber C can be disassembled into Xc = {i1, i2, i3,..., in, c1, c2, c3,..., cn}.

[0048] If there are fields that can be disassembled into the results calculated by other fields in the set according to specific rules, disassemble them according to the smallest granularity. For example, if ax = i1 + i2 and bx = i1 + i3, then only i1, i2, and i3 need to be included in the set.

[0049] Step 302, take the common fields required by all regulatory calibers as the wide - table fields;

[0050] The union of the de-duplicated Xa, Xb, and Xc, which is the common field, is directly used as the wide table field. Y = {i1, i2, i3, …, in, a1, a2, a3, …, an, b1, b2, b3, …, bn, c1, c2, c3, …, cn}. In Y, all fields are independent and unrelated fields, there is no mutual dependence in field values, and there are no redundant fields, ensuring the minimization of data processing and data consistency.

[0051] Step 303: Conduct semantic association analysis on the non-common fields required by all regulatory calibers. For multiple fields representing the same data, determine the main field for storing the data from the multiple fields, and use the other fields except the main field in the multiple fields as secondary fields.

[0052] For fields that seem different but are semantically similar under different regulatory calibers, integrate them through semantic analysis techniques. For example, a regulatory caliber of a certain regulatory agency uses "total transaction volume", while another regulatory caliber uses "total business transaction funds". Through semantic association analysis, it can be identified that they essentially describe the same type of data. At this time, "total transaction volume" can be used as the main field, and "total business transaction funds" can be used as the secondary field.

[0053] Step 304: Establish a mapping relationship between the secondary field and the main field, and store the mapping relationship in the secondary field. The mapping relationship is represented by the hash value of the main field.

[0054] Step 305: Use all the main fields as the wide table fields.

[0055] Step 306: Store all the secondary fields in the secondary field registration form.

[0056] In one embodiment, the method further includes:

[0057] After a new regulatory caliber appears, conduct semantic analysis on the new regulatory caliber to obtain the fields required by the new regulatory caliber.

[0058] Conduct semantic association analysis on the obtained fields and the wide table fields of the basic data.

[0059] If the obtained fields represent the same data as the wide table fields of the basic data, use the obtained fields as secondary fields, establish a mapping relationship between the secondary fields and the main fields, store the mapping relationship in the secondary fields, and store the secondary fields in the secondary field registration form.

[0060] If the obtained fields do not represent the same data as all the wide table fields of the basic data, use the obtained fields as the wide table fields of the basic data.

[0061] Through the above steps, dynamic adjustment of wide table fields can be achieved. Of course, the basic data wide table fields and the secondary field registration form can be updated regularly according to all current regulatory caliber requirements through a preset algorithm, reducing manual intervention and improving the timeliness and accuracy of data management.

[0062] Version management can be carried out to ensure that the data before the updated fields is not lost. For the modified or deleted fields, their historical version information is retained. This helps to quickly obtain the wide table field status in different periods when it is necessary to trace back data or analyze the evolution process of regulatory requirements, providing strong support for compliance auditing and data analysis.

[0063] When conducting semantic association analysis, knowledge graph technology can be introduced to construct a knowledge graph of associated transaction data. Using the wide table fields as nodes and the semantic associations, data dependency relationships, etc. between fields as edges, a visual knowledge network is formed. This not only helps to better understand and manage the data in the wide table, but also can discover potential cross-regulatory caliber requirements or data processing optimization points through the reasoning ability of the knowledge graph.

[0064] For sensitive fields in the wide table, such as transaction participant information involving customer privacy and transaction details of certain high-risk businesses, advanced encryption algorithms are used for encrypted storage. For example, homomorphic encryption technology is used, enabling certain computing operations to be performed on the encrypted data, ensuring data security while meeting data processing requirements.

[0065] In step 102, for each regulatory caliber, according to the requirements of the regulatory caliber and the wide table fields, determine the differential table fields of the regulatory caliber, and based on the differential table fields and the wide table fields, determine the differential field processing rules for calculating the differential table fields from the wide table fields, and store the differential field processing rules in the rule table;

[0066] In step 103, determine the report processing rules for calculating the regulatory caliber reports from the wide table fields and the differential table fields, and store them in the rule table;

[0067] Since there are differences in the calibers of certain fields for each regulatory caliber, the data based on the wide table fields at the first layer can process these fields based on the differential field processing rules of each regulatory caliber to form differential table fields, which are stored in the differential table (different regulatory calibers have different differential tables). For example, regulatory caliber A requires one of the reported fields to be ra, and the differential table field of regulatory caliber A is determined to be ra, ra = i1 + i2. Regulatory caliber B requires one of the reported fields to be rb, and the differential table field of regulatory caliber B is determined to be rb, rb = i1 + i3. These two additions are the differential field processing rules for regulatory calibers A and B respectively, and a rule set R can be formed and stored in the rule table.

[0068] In step 104, process the data from multiple data sources according to the wide table fields and store it in the wide table.

[0069] In one embodiment, processing the data from multiple data sources according to the wide table fields and storing it in the wide table includes:

[0070] After obtaining the data of each data source, use a machine learning model to perform standard preprocessing on the data of each data source.

[0071] Parallelly store the standardized data of each data source in the distributed database nodes corresponding to each data source, and use the blockchain to record the source and transfer process of the data; among them, when the distributed database nodes perform data storage, they are partitioned according to customers.

[0072] Perform association and dependency verification on the standardized data of each data source. If the verification fails, generate a reminder message.

[0073] After meeting the data processing trigger condition, process the data in the distributed database nodes according to the wide table fields and store it in the wide table.

[0074] In specific implementation, intelligent processing such as predictively filling missing values and automatically adjusting the data format according to data characteristics. Train the model based on historical data to improve the accuracy and intelligence of data processing. In addition, a real-time data monitoring mechanism can be introduced to monitor the quality indicators of data, such as data integrity, accuracy, consistency, etc., during the process of data transmission from multiple source systems to the wide table. Once data quality problems are found, immediately trigger an automatic repair process or send a warning message to relevant personnel to ensure the high quality of the wide table data.

[0075] Optionally, use a machine learning model to perform intelligent processing on the data of each data source, and then store it in a distributed manner, such as predictively filling missing values and automatically adjusting the data format according to data characteristics. The model can be trained based on historical data to improve the accuracy and intelligence of data processing.

[0076] Utilize blockchain technology to record the processing track and key operations of the data on the blockchain. The immutable feature of the blockchain ensures the authenticity and integrity of the audit trail, enabling regulatory agencies and enterprises themselves to use the data in the wide table with more confidence, and at the same time enhancing the transparency and credibility of data management.

[0077] When the distributed database stores data, it is partitioned according to the customer number as the partition key, and the data volume can be basically guaranteed to be equal in the distribution of the full customer volume data range, and the data volume of each shard is ensured to be roughly the same in terms of business.

[0078] The above parallel data processing improves data processing efficiency by distributing the workload across multiple nodes. For the upstream data of the related transaction system, which involves daily bank transaction records, accounting records, etc., the data sources are extensive, the daily new data volume is large, and the designed fields are complex. However, a distributed database can effectively handle the scenarios of continuous growth in the scale of the data set and increasing complexity.

[0079] Performing correlation and dependency verification on the standardized data of each data source can improve the accuracy and reliability of the data. For example, in related transactions, there may be a mutually corroborative relationship between bank transaction record data and enterprise accounting record data. Through this framework, these relationships can be automatically identified, and during the process of data processing into a wide table, joint processing and verification are carried out to improve the accuracy and reliability of the data.

[0080] In one embodiment, after the data processing trigger condition is met, the data in the distributed database nodes is processed according to the wide table fields and stored in the wide table, including:

[0081] Forming a trigger event according to the data processing trigger condition;

[0082] Generating a job for each wide table field to be processed under each regulatory caliber;

[0083] Determining the dependency relationship and fault tolerance mechanism between jobs;

[0084] Adopting a job execution prediction model to predict the execution time and resource requirements of each job. The job execution prediction model is obtained by training a machine learning model based on historical job execution data and the current system state;

[0085] When the trigger event is met, job scheduling and execution are carried out based on the execution time and resource requirements of all jobs.

[0086] The above steps are an event-based data batch processing scheduling scheme. Jobs include data transmission, data loading, data processing, etc. Figure 4 This is a schematic diagram of job dependencies in the embodiments of the present invention. It can be seen that data loading depends on data transmission, and data processing depends on data loading. Among them, the job execution prediction model predicts the execution time and resource requirements of jobs, and optimizes the job scheduling plan in advance. For example, when it is predicted that a pre-job on which a certain job depends will be delayed in completion, the execution order of subsequent jobs can be automatically adjusted, and those jobs that do not depend on the pre-job are preferentially executed to improve the execution efficiency of the overall job flow. At the same time, the fault tolerance mechanism of the scheduling platform is strengthened. When a job fails to execute, it can be automatically retried, and intelligent diagnosis and repair are carried out according to the failure reason. For example, if the job fails due to data quality problems, the scheduling platform can automatically trigger the data quality inspection and repair process to ensure that the job can be successfully completed.

[0087] In the event-based data batch processing scheduling solution, events generated during the continuous monitoring of each data source and job execution can be listened to. It will collect this event information in real time and determine whether events on which a job depends have occurred. For example, when the systems where multiple data sources are located complete daily data generation and transmit it to the temporary storage area, an event of "data transmission completed" will be generated, and then this event will be captured.

[0088] For the upstream data of the related party transaction system, such as daily bank transaction statements, accounting statements, etc., the sources are extensive, the daily new data volume is large, and the designed fields are complex. The distributed database can effectively handle the scenario of continuous growth in the scale of the data set and increasing complexity. For the data of different data sources, according to the characteristics of each data source, it is stored in the distributed database, and the data is transmitted through file transfer and processed into the distributed database daily. After the scheduling checks that all the data for the day is ready, the data processing of the common basic data layer is first completed in the distributed database, and the data in the distributed database nodes is processed according to the wide table fields and stored in the wide table.

[0089] In one embodiment, the method further includes:

[0090] After determining the wide table fields, determine the encryption requirements for different wide table fields;

[0091] Determine the encryption algorithm for the encryption requirements;

[0092] Processing the data in the distributed database nodes according to the wide table fields and storing it in the wide table includes:

[0093] After processing the data in the distributed database nodes according to the wide table fields, encrypt it using the corresponding encryption algorithm according to the encryption requirements of the wide table fields, and then store it in the wide table.

[0094] In step 105, query the rule table to obtain the processing rules for the differential fields of the regulatory caliber, read the data from the wide table, process it according to the queried differential field processing rules, and store it in the differential table fields corresponding to the regulatory caliber;

[0095] In step 106, after receiving the reporting request for the target regulatory caliber, query the rule table to obtain the report processing rules for the target regulatory caliber, read the data from the wide table, and read the data in the differential table corresponding to the target regulatory caliber, process it according to the queried report processing rules, and store it in the report, and output the report to the target regulatory caliber.

[0096] In one embodiment, the wide table fields further include regulatory caliber fields;

[0097] Read data from the wide table, including: read the data corresponding to the regulatory caliber from the wide table according to the regulatory caliber fields.

[0098] The regulatory caliber fields are used to record which regulatory agencies each transaction belongs to, facilitating the subsequent calculation of statistical reports for different regulatory calibers.

[0099] In one embodiment, the wide table, difference table, rule table, and report are stored in a distributed relational database;

[0100] Read data from the wide table and process it according to the query difference field processing rules, including: using the distributed computing framework of the distributed relational database, distribute the processing tasks of different difference field processing rules to multiple nodes for parallel execution;

[0101] Read data from the wide table and read the data in the difference table corresponding to the target regulatory caliber, and process it according to the query report processing rules, including:

[0102] Using the distributed computing framework of the distributed relational database, distribute the processing tasks of different report processing rules to multiple nodes for parallel execution, and process the data read from the wide table and the data in the difference table corresponding to the target regulatory caliber read.

[0103] Figure 5 This is the data flow diagram of the wide table, difference table, rule table, and report in the distributed relational database in the embodiment of the present invention. The first part, the wide table, is used to store the basic transaction data that can be maintained by the business personnel with the smallest granularity. The second part is the rule table used to store the processing formulas for the difference table fields and report fields maintained by the business. The third part is the difference table of the special field values calculated using the basic data of the wide table and the formulas of the rule table. The fourth part is the report of each regulatory caliber calculated based on the difference table and the basic transaction data. The advantage of this flow path is that if the requirements for adjusting the report processing caliber by the supervision are changed, only the data in the rule table and the interface table need to be maintained, which is highly flexible and convenient to operate.

[0104] Specifically, during the processing, cache some frequently used intermediate result data to avoid repeated calculations. For example, the calculation results of some fields are used in multiple rules. Caching them can reduce the waste of computing resources. In addition, the above parallel execution process can improve the efficiency of data processing.

[0105] The distributed relational database combines the advantages of traditional relational databases and distributed databases, can perform online transaction processing / online analytical processing, can be horizontally scaled, provides high availability and strong consistency, and is suitable for application scenarios of various scales, especially for industries such as the Internet and finance with high requirements for data scale and performance.

[0106] Distributed relational databases adopt an architecture that separates storage from computing. Data is stored in a distributed storage engine, and the computing layer can flexibly expand the number of nodes according to business requirements to achieve linear growth in computing power. When the data volume and business load increase, simply adding new nodes can easily handle the situation without performance bottlenecks. It can automatically disperse data across multiple nodes, and the data sharding strategy is based on the hash algorithm or range partitioning to ensure uniform data distribution, avoid data hotspots, enable each node to fully play its role, and improve the concurrent processing ability of the overall system.

[0107] Data is stored in multiple copies on multiple nodes. By default, each data shard has three copies distributed on different nodes. When a node fails, other copies can immediately take over the service to ensure data availability and business continuity, with a data reliability as high as 99.9999%. The node status is monitored in real time. Once a node failure is detected, the system will automatically perform a failover, switching the traffic to healthy nodes. The entire process is transparent to the application program and requires no manual intervention, greatly improving the stability and reliability of the system.

[0108] Distributed relational databases support distributed transactions based on distributed consistency algorithms, ensuring data consistency in a distributed environment. Whether it is a transaction operation on a single node or across multiple nodes, it can meet the ACID characteristics to ensure data integrity and accuracy.

[0109] For wide tables, differential tables, and reports in distributed relational data, a more fine-grained permission control system can be designed based on regulatory requirements and user roles. Staff of different regulatory agencies and personnel from different departments within an enterprise are given different access permissions to wide table fields according to their responsibilities and compliance requirements. For example, regulatory agency A can only view fields related to its own regulatory requirements, and the enterprise's finance department can view and modify finance-related fields but cannot access sensitive fields related to business. In this way, the secure use and compliance of data are ensured.

[0110] The embodiment of the present invention also proposes a multi-source data processing device based on the reporting output requirements of multiple regulatory requirements. Its principle is similar to the multi-source data processing method based on the reporting output requirements of multiple regulatory requirements, and will not be elaborated here.

[0111] Figure 6 It is a schematic diagram of the multi-source data processing device based on the reporting output requirements of multiple regulatory requirements in the embodiment of the present invention, including:

[0112] A wide table field determination module 601, configured to perform semantic association analysis on different regulatory requirements to determine wide table fields, where the wide table fields include fields storing the same data;

[0113] The difference table field determination module 602 is configured to, for each regulatory caliber, determine the difference table fields of the regulatory caliber according to the requirements of the regulatory caliber and the wide table fields, and determine the difference field processing rules for calculating the difference table fields from the wide table fields based on the difference table fields and the wide table fields, and store the difference field processing rules in a rule table;

[0114] The rule determination module 603 is configured to determine the report processing rules for calculating the regulatory caliber reports from the wide table fields and the difference table fields, and store them in a rule table;

[0115] The source data processing module 604 is configured to process the data of multiple data sources according to the wide table fields and store them in the wide table;

[0116] The wide table data processing module 605 is configured to query the rule table to obtain the difference field processing rules of the regulatory caliber, read the data from the wide table, process them according to the queried difference field processing rules, and store them in the difference table fields corresponding to the regulatory caliber;

[0117] The report generation module 606 is configured to, after receiving a submission request for a target regulatory caliber, query the rule table to obtain the report processing rules of the target regulatory caliber, read the data from the wide table, and read the data in the difference table corresponding to the target regulatory caliber, process them according to the queried report processing rules, and store them in the report, and output the report to the target regulatory caliber.

[0118] In one embodiment, the wide table field determination module 601 is configured to:

[0119] Perform semantic analysis on the requirements of different regulatory calibers, and obtain the fields required for each regulatory caliber requirement according to the smallest disassembly granularity;

[0120] Perform semantic association analysis on the non-common fields required for all regulatory calibers. For multiple fields representing the same data, determine the main field for storing the data from the multiple fields, and use the other fields except the main field among the multiple fields as secondary fields;

[0121] Establish a mapping relationship between the secondary fields and the main fields, store the mapping relationship in the secondary fields, and represent the mapping relationship using the hash value of the main field;

[0122] Use all the main fields as the wide table fields;

[0123] Store all the secondary fields in a secondary field registration form.

[0124] In one embodiment, the source data processing module 604 is configured to:

[0125] After obtaining the data of each data source, use a machine learning model to perform standard preprocessing on the data of each data source;

[0126] Parallelly store the standardized data of each data source in the distributed database nodes corresponding to each data source, and use the blockchain to record the source and transfer process of the data; wherein, when the distributed database nodes perform data storage, they are partitioned according to customers.

[0127] Perform correlation and dependency verification on the standardized data of each data source. If the verification fails, generate a reminder message.

[0128] After meeting the data processing trigger condition, process the data in the distributed database nodes according to the wide table fields and store it in the wide table.

[0129] In one embodiment, the device further includes an encryption algorithm determination module for:

[0130] After determining the wide table fields, determine the encryption requirements for different wide table fields.

[0131] Determine the encryption algorithm for the encryption requirements.

[0132] The source data processing module 604 is further used for:

[0133] After processing the data in the distributed database nodes according to the wide table fields, encrypt it using the corresponding encryption algorithm according to the encryption requirements of the wide table fields, and then store it in the wide table.

[0134] In one embodiment, the source data processing module 604 is used for:

[0135] Form a trigger event according to the data processing trigger condition.

[0136] Generate a job for each to-be-processed wide table field of each regulatory caliber.

[0137] Determine the dependency relationship and fault tolerance mechanism between jobs.

[0138] Adopt a job execution prediction model to predict the execution time and resource requirements required for each job. The job execution prediction model is obtained by training a machine learning model based on historical job execution data and the current system state.

[0139] When the trigger event is met, perform job scheduling and execution based on the execution time and resource requirements required for all jobs.

[0140] In one embodiment, the wide table fields further include regulatory caliber fields.

[0141] The wide table data processing module 605 is used for: Read the data corresponding to the regulatory caliber from the wide table according to the regulatory caliber field.

[0142] The regulatory caliber field is used to record which regulatory agencies each transaction belongs to, facilitating subsequent calculation of statistical reports for different regulatory calibers.

[0143] In one embodiment, the wide table, difference table, rule table, and report are stored in a distributed relational database;

[0144] The wide table data processing module 605 is used to: utilize the distributed computing framework of the distributed relational database to distribute the processing tasks of different difference field processing rules to multiple nodes for parallel execution;

[0145] The report generation module 506 is used to:

[0146] Utilize the distributed computing framework of the distributed relational database to distribute the processing tasks of different report processing rules to multiple nodes for parallel execution, and process the data read from the wide table and the data in the difference table corresponding to the target regulatory caliber read.

[0147] In summary, the method and device proposed in the embodiments of the present invention have the following beneficial effects:

[0148] Perform semantic association analysis on different regulatory caliber requirements to determine the wide table fields, where the wide table fields include fields storing the same data; for each regulatory caliber, determine the difference table fields of the regulatory caliber according to the requirements of the regulatory caliber and the wide table fields, and determine the difference field processing rules for calculating the difference table fields from the wide table fields based on the difference table fields and the wide table fields, and store the difference field processing rules in the rule table; determine the report processing rules for calculating the regulatory caliber reports from the wide table fields and the difference table fields, and store them in the rule table; process the data from multiple data sources according to the wide table fields and store them in the wide table; query the rule table to obtain the difference field processing rules of the regulatory caliber, read the data from the wide table, process them according to the queried difference field processing rules, and store them in the difference table fields corresponding to the regulatory caliber; after receiving the reporting request for the target regulatory caliber, query the rule table to obtain the report processing rules of the target regulatory caliber, read the data from the wide table, and read the data in the difference table corresponding to the target regulatory caliber, process them according to the queried report processing rules, and store them in the report, and output the report to the target regulatory caliber. In the above process, for the calculations of different calibers, the user only needs to maintain the first-layer wide table well, and for the changes in the rules, only the processing rules of the second and third layers of each regulatory caliber need to be maintained well. This enables efficient completion of the processing of multi-regulatory caliber reporting data in the face of multi-associated transaction data sources, reduces the storage space, effectively reduces the user's workload, improves the response speed to changes in regulatory rules, and effectively reduces the associated impact between the calculations of each caliber, reducing the impact of a change in the rules of one regulatory caliber on the processing of other regulatory calibers, and facilitating maintenance.

[0149] An embodiment of the present invention also provides a computer device. Figure 7 It is a schematic diagram of the computer device in the embodiment of the present invention. The computer device 700 includes a memory 710, a processor 720, and a computer program 730 stored on the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 730, the multi-source data processing method based on the multi-regulatory-caliber report output requirement is implemented.

[0150] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-source data processing method based on the multi-regulatory-caliber report output requirement is implemented.

[0151] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the multi-source data processing method based on the multi-regulatory-caliber report output requirement is implemented.

[0152] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0154] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one or more flows and / or blocks Figure 1The functions specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in Figure 1 one process or more processes and / or boxes Figure 1 the functions specified in one box or more boxes.

[0156] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-source data processing method based on the reporting output requirements of multiple regulatory standards, characterized in that, Including: Performing semantic association analysis on different regulatory caliber requirements to determine wide table fields, where the wide table fields include fields storing the same data; For each regulatory caliber, determining the differential table fields of this regulatory caliber according to the requirements of this regulatory caliber and the wide table fields, and determining the differential field processing rules for calculating the differential table fields from the wide table fields, and storing the differential field processing rules in a rule table; Determining the report processing rules for calculating regulatory caliber reports from the wide table fields and the differential table fields, and storing them in the rule table; Processing the data of multiple data sources according to the wide table fields and storing them in the wide table; Querying the rule table to obtain the differential field processing rules of the regulatory caliber, reading data from the wide table, processing according to the queried differential field processing rules, and storing them in the differential table fields corresponding to the regulatory caliber; After receiving a submission request for a target regulatory caliber, querying the rule table to obtain the report processing rules of the target regulatory caliber, reading data from the wide table, reading the data in the differential table corresponding to the target regulatory caliber, processing according to the queried report processing rules, and storing them in the report, and outputting the report to the target regulatory caliber.

2. The method according to claim 1, characterized in that, Performing semantic association analysis on different regulatory caliber requirements to determine wide table fields, including: Performing semantic analysis on different regulatory caliber requirements and obtaining the fields required for each regulatory caliber according to the smallest decomposition granularity; Taking the common fields required for all regulatory calibers as wide table fields; Performing semantic association analysis on the non-common fields required for all regulatory calibers. For multiple fields representing the same data, determining the main field for storing the data from the multiple fields, and taking the other fields except the main field in the multiple fields as secondary fields; Establishing a mapping relationship between the secondary fields and the main field, and storing the mapping relationship in the secondary fields, where the mapping relationship is represented by the hash value of the main field; Taking all the main fields as wide table fields; Storing all the secondary fields in a secondary field registration form.

3. The method according to claim 1, characterized in that, Processing the data of multiple data sources according to the wide table fields and storing them in the wide table, including: After obtaining the data of each data source, using a machine learning model to perform standard preprocessing on the data of each data source; Storing the standardized data of each data source in parallel in the distributed database nodes corresponding to each data source, and using a blockchain to record the source and flow process of the data; among them, the distributed database nodes perform data storage by partitioning according to customers; Performing an association dependency verification on the standardized data of each data source. If the verification fails, generating a reminder message; After meeting the data processing trigger condition, processing the data in the distributed database nodes according to the wide table fields and storing them in the wide table.

4. The method according to claim 3, wherein Also including: After determining the wide table fields, determining the encryption requirements for different wide table fields; Determining the encryption algorithm for the encryption requirements; Processing the data in the distributed database nodes according to the wide table fields and storing them in the wide table, including: Processing the data in the distributed database nodes according to the wide table fields, encrypting them using the corresponding encryption algorithm according to the encryption requirements of the wide table fields, and then storing them in the wide table.

5. The method according to claim 3, wherein After the data processing trigger condition is met, process the data in the distributed database nodes according to the wide table fields and store it in the wide table, including: Form a trigger event according to the data processing trigger condition; Generate a job for each wide table field to be processed under each regulatory caliber; Determine the dependency relationship and fault tolerance mechanism between jobs; Adopt a job execution prediction model to predict the execution time and resource requirements required for each job. The job execution prediction model is obtained by training a machine learning model based on historical job execution data and the current system state; When the trigger event is met, perform job scheduling and execution based on the execution time and resource requirements required for all jobs.

6. The method according to claim 1, characterized in that The wide table fields further include regulatory caliber fields; Read data from the wide table, including: read the data corresponding to the regulatory caliber from the wide table according to the regulatory caliber field.

7. The method according to claim 1, wherein The wide table, difference table, rule table, and report are stored in a distributed relational database; Read data from the wide table and process it according to the difference field processing rules queried, including: using the distributed computing framework of the distributed relational database, distribute the processing tasks of different difference field processing rules to multiple nodes for parallel execution; Read data from the wide table and read the data in the difference table corresponding to the target regulatory caliber, and process it according to the report processing rules queried, including: Using the distributed computing framework of the distributed relational database, distribute the processing tasks of different report processing rules to multiple nodes for parallel execution, and process the data read from the wide table and the data in the difference table corresponding to the target regulatory caliber read.

8. A multi-source data processing device based on the reporting output requirements of multiple regulatory calibers, characterized in that, Including: A wide table field determination module for performing semantic association analysis on different regulatory caliber requirements to determine wide table fields, where the wide table fields include fields storing the same data; A difference table field determination module for, for each regulatory caliber, determine the difference table fields of the regulatory caliber according to the requirements of the regulatory caliber and the wide table fields, and determine the difference field processing rules for calculating the difference table fields from the wide table fields according to the difference table fields and the wide table fields, and store the difference field processing rules in the rule table; A rule determination module for determining the report processing rules for calculating the regulatory caliber reports from the wide table fields and the difference table fields and storing them in the rule table; A source data processing module for processing the data of multiple data sources according to the wide table fields and storing it in the wide table; A wide table data processing module for querying the rule table to obtain the difference field processing rules of the regulatory caliber, reading data from the wide table, processing it according to the queried difference field processing rules, and storing it in the difference table fields corresponding to the regulatory caliber; A report generation module for, after receiving a reporting request for the target regulatory caliber, querying the rule table to obtain the report processing rules of the target regulatory caliber, reading data from the wide table, reading the data in the difference table corresponding to the target regulatory caliber, processing it according to the queried report processing rules, and storing it in the report, and outputting the report to the target regulatory caliber.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images