Data processing method, processing system, electronic equipment, storage medium and product

Abstract

The invention provides a data processing method and system, electronic equipment, a storage medium and a product, and relates to the technical field of information. The data processing method comprises the following steps: collecting original reconciliation data of each business department, taking a distributed file system as bottom storage, constructing a data warehouse framework and a distributed column-oriented open source database on the basis, and managing the original reconciliation data. Based on a user query instruction, target account checking data of a current checking time period is obtained from an open source database, and the data relates to more than two business departments. And then target reconciliation data are verified, and data of different departments are checked in a cross manner. According to the scheme, a large amount of data is efficiently stored through the distributed file system, ordered management and efficient analysis are achieved through an upper-layer framework and a database, quick and accurate response query can be achieved, the data accuracy and reliability are improved, potential business problems can be perceived in time, and business promotion and financial accounting accuracy are guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and particularly to a data processing method, a processing system, an electronic device, a storage medium, and a product. Background Art

[0002] Most of the traditional system reconciliations use a pairwise interaction system reconciliation mode. In this mode, each system needs to have an independent reconciliation processing program internally, and the systems also need to transfer reconciliation data and process reconciliation results through file interaction between each other. Figure 1 For the schematic diagram of the pairwise interaction system reconciliation mode in the prior art, as Figure 1 shown, if three systems A, B, and C need to perform transaction reconciliation, system A needs to generate a system A - B reconciliation file, and then wait for system B to complete the reconciliation and process the system A - B reconciliation result file generated by system B; system B needs to generate a system B - C reconciliation file, and then wait for system C to complete the reconciliation and process the system B - C reconciliation result file generated by system C. If more systems need to be involved in the reconciliation, it can be inferred in sequence. The advantage of this mode is that the implementation logic of a single system is relatively simple. From the perspective of a single system, for each newly added external system, only one reconciliation file needs to be generated and one reconciliation result file needs to be processed to complete the reconciliation function with the new system. However, from the enterprise level, if there are many internal systems in an enterprise and pairwise reconciliations are required between multiple systems, assuming the enterprise currently has N systems that need pairwise reconciliations, then for each newly added system, 2 * N new reconciliation files need to be added to perform the reconciliation operation, and the transformation cost is relatively high. Summary of the Invention

[0003] The technical problem to be solved by the present disclosure is to overcome the defect in the pairwise interaction system reconciliation mode in the prior art that when there are multiple reconciliation systems, a large number of reconciliation files need to be added for each newly added system to perform the reconciliation operation, and the transformation cost is relatively high, and to provide a data processing method, a processing system, an electronic device, a storage medium, and a product.

[0004] The present disclosure solves the above technical problem through the following technical solutions:

[0005] The present disclosure provides a data processing method, and the data processing method includes:

[0006] Collecting original reconciliation data, where the original reconciliation data is reconciliation data from different business departments;

[0007] Using a distributed file system as the underlying data storage medium, and building a data warehouse infrastructure and a distributed, column - oriented open - source database on the distributed file system to manage the original reconciliation data based on the data warehouse infrastructure and the open - source database;

[0008] Based on the user's query instruction, query the target reconciliation data corresponding to the current verification period of the current time from the original reconciliation data in the open-source database; the target reconciliation data includes reconciliation data of two or more different business departments.

[0009] Perform verification processing on the target reconciliation data to obtain a verification result.

[0010] In this solution, a distributed file system is used as the underlying data storage medium, which can efficiently store a large amount of original reconciliation data from different business departments. At the same time, the data warehouse infrastructure and the distributed, column-oriented open-source database built on it realize the orderly management and efficient analysis of data. When receiving a user's query instruction, it can quickly and accurately obtain the target reconciliation data corresponding to the current verification period of the current time from the open-source database. These data cover the reconciliation data of two or more different business departments. Then, through verification processing, cross-check the data of different departments, effectively improve the accuracy and reliability of the data, timely discover potential problems in the business process, and ensure the smooth progress of the business and the accuracy of financial accounting.

[0011] Optionally, the step of performing verification processing on the target reconciliation data to obtain a verification result includes:

[0012] Compare the reconciliation data of any two different business departments in the target reconciliation data and obtain a comparison result.

[0013] In response to the comparison result indicating that the reconciliation data between the two is inconsistent, determine that the verification result is an abnormal account.

[0014] In response to the comparison result indicating that the reconciliation data between the two is consistent, determine that the verification result is a normal account.

[0015] In this solution, the verification result is obtained by verifying and processing the target reconciliation data. In terms of ensuring data accuracy, the reconciliation data of different business departments are compared, enabling the timely detection of data differences, avoiding incorrect decisions and financial risks caused by inconsistent data, and ensuring that the enterprise conducts operation and management based on accurate data. At the level of optimizing business processes, the identification of abnormal accounts helps the enterprise deeply investigate possible loopholes, errors, or irregular operations in business processes, such as data entry errors and deviations in business process execution, and then make targeted improvements and optimizations to enhance the standardization and efficiency of business processes. From the perspective of enterprise internal control, this verification mechanism strengthens the mutual supervision and checks and balances of data between departments, forms an effective internal control system, prevents fraud and deception, and safeguards the safety of enterprise assets and financial health. Moreover, this standardized verification process is highly repeatable and can be continuously applied to subsequent reconciliation work, providing long-term and stable support for enterprise data management and business operations.

[0016] Optionally, after the step of verifying and processing the target reconciliation data to obtain a verification result, the data processing method further includes:

[0017] Tracking the verification process based on a monitoring tool;

[0018] And / or,

[0019] Tracking the verification result based on a monitoring tool, and generating an exception reminder message in response to the verification result being an abnormal account.

[0020] In this solution, in terms of tracking the verification process, relevant personnel can be enabled to grasp the progress and status of the verification work in real time, promptly discover possible problems such as delays and errors in the verification process, and ensure the smooth progress of the verification work. And tracking the verification result can ensure that once an abnormal account situation occurs, the system can quickly generate an exception reminder message, enabling the enterprise to promptly know the problem, gain valuable time for taking corrective measures and investigating the cause of the exception, avoid the further deterioration of the abnormal situation, safeguard the accuracy of enterprise financial data and the stability of business operations, and enhance the enterprise's ability and efficiency to respond to risks.

[0021] Optionally, the step of verifying and processing the target reconciliation data to obtain a verification result includes:

[0022] Obtaining historical reconciliation data corresponding to the previous verification period;

[0023] Determining the difference in reconciliation data based on the historical reconciliation data and the target reconciliation data;

[0024] If the difference in reconciliation data is inconsistent with the preset difference, determining the verification result as an abnormal account.

[0025] In this solution, potential risks can be warned in a timely manner. Once the difference in the reconciliation data does not match the preset value, financial risks, business mistakes, or external influencing factors can be quickly detected, facilitating the enterprise to respond in a timely manner and avoid the expansion of losses. At the same time, the data accuracy and reliability can be effectively improved. By using the regularity of historical data to test the current data, problems such as input errors and system failures can be identified and corrected, providing an accurate basis for decision-making.

[0026] Optionally, the step of using a distributed file system as the underlying data storage medium and constructing a data warehouse infrastructure and a distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data based on the data warehouse infrastructure and the open-source database includes:

[0027] Use the distributed file system as the underlying data storage medium, analyze through the mapping function programming mode, and construct the data warehouse infrastructure and the distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data.

[0028] In this solution, in terms of storage, the distributed file system has strong scalability, can handle a large amount of original reconciliation data, and enhances data reliability through redundant storage; in terms of processing, the mapping function programming mode supports parallel computing, has high processing efficiency, and can flexibly design the processing logic according to business requirements; in terms of data management, the data warehouse and the column-oriented open-source database build a unified management platform, improving data standardization, and the column-oriented database can perform efficient query analysis; in terms of cost, ordinary commercial hardware can be used to reduce hardware costs, and open-source software does not require high licensing fees and can be customized and extended as needed to save software costs.

[0029] Optionally, after the step of collecting the original reconciliation data, the data processing method further includes:

[0030] Preprocess the original reconciliation data according to different business requirements; the business requirements include the need for real-time transmission and the absence of the need for real-time transmission;

[0031] The step of preprocessing the original reconciliation data according to different business requirements specifically includes:

[0032] In response to the business requirement being the need for real-time transmission, use a streaming processing method to process the original reconciliation data;

[0033] In response to the business requirement being the absence of the need for real-time transmission, use a batch processing method to process the original reconciliation data;

[0034] and / or,

[0035] Before the step of querying, based on the user query instruction, the target reconciliation data corresponding to the current verification period of the current time from the open source database, the data processing method further includes:

[0036] Verify the identity of the user and generate a verification result;

[0037] Based on the verification result, determine whether to allow the user to access;

[0038] The verification result includes a success message and a failure message. The step of determining whether to allow the user to access based on the verification result includes:

[0039] In response to the verification result being the success message, allow the user to access;

[0040] In response to the verification result being the failure message, reject the user's access;

[0041] And / or,

[0042] After the step of verifying and processing the target reconciliation data to obtain a verification result, the data processing method further includes:

[0043] Use a preset display method to display the target reconciliation data and / or the verification process of the target reconciliation data.

[0044] In this solution, for the business requirements that need real-time transmission, the streaming processing method can process the original reconciliation data in a timely manner, ensuring that the data can be quickly analyzed and transmitted after generation, meeting the business scenarios with high timeliness requirements, such as real-time monitoring and instant decision-making, and ensuring the real-time and efficient operation of the business. For the business requirements that do not require real-time transmission, the batch processing method can centrally process a large amount of data, enabling more comprehensive and in-depth analysis and optimization of the data, improving resource utilization, reducing processing costs, and ensuring the accuracy and stability of data processing. It realizes flexible data processing according to the actual business needs, enhancing the overall efficiency and adaptability of business processing;

[0045] Verifying the user identity can prevent unauthorized access, avoid data leakage, protect data integrity, and prevent unauthorized operations from damaging the data. In terms of compliance, it can help enterprises meet industry regulations and standards requirements, avoid legal risks, and enhance industry reputation and competitiveness. In terms of stability, rejecting users who fail the verification can reduce the system load, avoid invalid requests from occupying resources, and can also reasonably allocate resources according to user permissions, ensuring the operation efficiency and stability of the system, and ensuring the effective utilization of resources and the smooth operation of the system.

[0046] By clearly and intuitively presenting the target reconciliation data, relevant personnel can quickly understand the overall data picture and key information, improve the efficiency of information acquisition, and facilitate quick decision-making.

[0047] The present disclosure also provides a data processing system, which includes:

[0048] A data collection module for collecting original reconciliation data, where the original reconciliation data is reconciliation data from different business departments;

[0049] A data management module for using a distributed file system as the underlying data storage medium, and building a data warehouse infrastructure and a distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data based on the data warehouse infrastructure and the open-source database;

[0050] A data query module for querying, based on a user query instruction, target reconciliation data corresponding to the current verification period of the current time from the original reconciliation data in the open-source database; the target reconciliation data includes reconciliation data of two or more different business departments;

[0051] A data verification module for performing verification processing on the target reconciliation data to obtain a verification result.

[0052] Optionally, the data verification module includes a data comparison unit, an account anomaly determination unit, and an account normal determination unit;

[0053] The data comparison unit is used to compare the reconciliation data of any two different business departments in the target reconciliation data and obtain a comparison result;

[0054] The account anomaly determination unit is used to determine that the verification result is an account anomaly in response to the comparison result indicating that the reconciliation data between the two is inconsistent;

[0055] The account normal determination unit is used to determine that the verification result is an account normal in response to the comparison result indicating that the reconciliation data between the two is consistent.

[0056] Optionally, the data processing system further includes: a verification process tracking module and / or an anomaly prompt generation module;

[0057] The verification process tracking module is used to track the verification process based on a monitoring tool;

[0058] The anomaly prompt generation module is used to track the verification result based on a monitoring tool, and generate an anomaly reminder message in response to the verification result being the account anomaly.

[0059] Optionally, the data verification module includes a historical reconciliation data acquisition unit, a reconciliation data difference determination unit, and an account anomaly determination unit;

[0060] The historical reconciliation data acquisition unit is configured to acquire historical reconciliation data corresponding to the previous verification period;

[0061] The reconciliation data difference determination unit is configured to determine a reconciliation data difference based on the historical reconciliation data and the target reconciliation data;

[0062] The account anomaly determination unit is configured to determine that the verification result is an account anomaly in response to the reconciliation data difference not being consistent with a preset difference.

[0063] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and configured to run on the processor. When the processor executes the computer program, the data processing method as described above is implemented.

[0064] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data processing method as described above is implemented.

[0065] The present disclosure also provides a computer program product, including a computer program. When the computer program is executed by a processor, the data processing method as described above is implemented.

[0066] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred examples of the present disclosure.

[0067] The positive and progressive effects of the present disclosure are as follows:

[0068] In this solution, a distributed file system is used as the underlying data storage medium, which can efficiently store a large amount of original reconciliation data from different business departments. At the same time, the data warehouse infrastructure and the distributed, column-oriented open-source database built thereon realize the orderly management and efficient analysis of data. When a user query instruction is received, the target reconciliation data corresponding to the current verification period of the current time can be accurately obtained from the open-source database quickly. These data cover the reconciliation data of two or more different business departments. Then, through verification processing, cross-checking of data from different departments is carried out, effectively improving the accuracy and reliability of data, timely discovering potential problems in the business process, and ensuring the smooth progress of the business and the accuracy of financial accounting;

[0069] The data processing method proposed in this paper combines the basic framework of distributed big data technology with distributed real-time database technology, standardizes the reconciliation function access process through data collection and preprocessing processes and a flexibly configurable data verification program; through a massive data processing engine, visual display, and monitoring alarm and response functions, it improves the discovery and processing efficiency of data anomaly problems and enhances the consistency of cross-domain system data. In summary, this method can simplify the development difficulty, improve the development efficiency, and reduce the technology investment cost during the iterative upgrade of enterprise system clusters, and has significant technological innovation and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the reconciliation mode of pairwise interaction systems in the prior art;

[0071] Figure 2 It is a schematic flowchart of a data processing method provided in Embodiment 1 of the present disclosure;

[0072] Figure 3 It is a schematic structural diagram of a module corresponding to the data processing method provided in Embodiment 1 of the present disclosure;

[0073] Figure 4 It is a schematic diagram of a module of a data processing system provided in Embodiment 2 of the present disclosure;

[0074] Figure 5 It is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0075] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0076] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in this embodiment, the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0077] In addition, if there are descriptions involving "first", "second", etc. in this embodiment, the descriptions of "first", "second", etc. are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this disclosure.

[0078] In the embodiments of the present disclosure, the processing of collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information complies with the provisions of relevant laws and regulations and does not violate public order and good customs.

[0079] Embodiment 1

[0080] Please refer to Figure 2 , this embodiment provides a data processing method. Figure 2 As shown in the flowchart of a data processing method provided in Embodiment 1 of the present disclosure, this specification provides the method operation steps such as in the embodiment or flowchart, but based on routine or non-creative labor, it may include more or fewer operation steps. The order of steps listed in the embodiment is only one of the execution orders of numerous steps and does not represent the only execution order. When the actual system or server product executes, it can be executed in the order shown in the embodiment or the drawing or executed in parallel (for example, in an environment of parallel processors or multi-threaded processing). Specifically, as Figure 2 shown, the data processing method includes:

[0081] S201: Collect original reconciliation data, where the original reconciliation data is the reconciliation data from different business departments;

[0082] S202: Use a distributed file system as the underlying data storage medium, and build a data warehouse infrastructure and a distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data based on the data warehouse infrastructure and the open-source database;

[0083] S203: Based on the user query instruction, query the target reconciliation data corresponding to the current verification period of the current time from the original reconciliation data in the open-source database; the target reconciliation data includes the reconciliation data of two or more different business departments;

[0084] S204: Perform verification processing on the target reconciliation data to obtain a verification result.

[0085] By adopting a distributed file system as the underlying data storage medium, a large amount of original reconciliation data from different business departments can be efficiently stored. Meanwhile, the data warehouse infrastructure and the distributed, column-oriented open-source database built on it enable the orderly management and efficient analysis of data. When a user query instruction is received, the target reconciliation data corresponding to the current verification period of the current time can be accurately obtained from the open-source database quickly. These data cover the reconciliation data of more than two different business departments. Then, through verification processing, cross-checking of data from different departments is carried out, effectively improving the accuracy and reliability of data, timely discovering potential problems in the business process, and ensuring the smooth progress of business and the accuracy of financial accounting.

[0086] Specifically, after the step of collecting the original reconciliation data, the data processing method further includes:

[0087] Preprocess the original reconciliation data according to different business requirements; the business requirements include the need for real-time transmission and the need for non-real-time transmission. The steps of preprocessing the original reconciliation data according to different business requirements specifically include: in response to the business requirement for real-time transmission, adopt a streaming processing method to process the original reconciliation data; in response to the business requirement for non-real-time transmission, adopt a batch processing method to process the original reconciliation data. This solution, for the business requirement of real-time transmission, can process the original reconciliation data in a timely manner by using the streaming processing method, ensuring that the data can be analyzed and transmitted quickly after being generated, meeting business scenarios with high requirements for timeliness, such as real-time monitoring and instant decision-making, and ensuring the real-time and efficient nature of the business. For the business requirement of non-real-time transmission, the batch processing method can centrally process a large amount of data, enabling more comprehensive and in-depth analysis and optimization of the data, improving resource utilization rate, reducing processing costs, and at the same time ensuring the accuracy and stability of data processing. It realizes flexible data processing according to the actual business needs, enhancing the efficiency and adaptability of the overall business processing.

[0088] Adopt a hybrid data collection scheme according to different business requirements and data characteristics. For example, for data with high real-time requirements, use a streaming processing method for collection and processing, while for data with low real-time requirements, use a batch processing method for collection and processing.

[0089] Among them, for batch data collection, Hadoop MapReduce (Hadoop distributed computing framework) and the data migration tool Sqoop are used to batch extract data from relational databases, and then the processed data is stored in the Hadoop Distributed File System (HDFS), Hadoop Database (HBase), or the data warehouse tool (Hive) built on top of Hadoop. After that, Hive can be further used for data analysis, or HBase can be used for fast query. Specifically, the data extraction technical tools: Batch extract data from relational databases with the help of Hadoop MapReduce and Sqoop. Hadoop MapReduce is a distributed computing model suitable for parallel processing of large-scale data sets. Sqoop is a tool for efficiently transferring data between Hadoop and relational databases, and it can import data from relational databases into the Hadoop ecosystem. Operation process: Utilize the functions of Sqoop to batch read data from relational databases (such as Structured Query Language MySQL, database Oracle, etc.) according to the set rules. For example, extract all customer data regularly (such as every early morning) from a relational database table containing customer information. In this process, Sqoop can utilize the distributed computing power of Hadoop MapReduce to parallel process the data reading operation and improve the data extraction efficiency.

[0090] Data storage: The extracted and processed data will be stored in HDFS, HBase, or Hive. HDFS (Hadoop Distributed File System) provides highly reliable and highly scalable storage, suitable for storing massive amounts of raw data; HBase is a distributed columnar storage system based on Hadoop, good at handling real-time read and write operations; Hive is a data warehouse tool based on Hadoop, allowing users to query data stored in HDFS using a SQL-like language, facilitating data analysis. Select a suitable storage method according to the characteristics of the data and subsequent usage requirements. If the data is mainly used for long-term storage and subsequent large-scale data analysis, HDFS may be preferred; if the data needs to support real-time query and fast read and write, HBase may be a better choice; if the data is to be used for complex data analysis and report generation, it is more appropriate to store it in Hive to utilize its SQL query function.

[0091] Data Analysis: After data storage is completed, Hive can be further used for data analysis. Hive converts the SQL query statements written by users into a programming model and a computing framework for processing big data, namely MapReduce jobs, which are executed on the Hadoop cluster, facilitating data analysts to perform statistical analysis, data mining, and other operations on the data stored in HDFS or Hive tables. For example, analyzing customer purchase behavior data, and counting the purchase frequencies and consumption amounts of customers in different regions and age groups, etc.

[0092] Fast Query: If fast query of data is required, the high-performance real-time read and write capabilities of HBase come into play. For example, in scenarios such as real-time querying of the latest order status of customers, HBase can quickly respond to query requests and provide the required data.

[0093] Streaming Data Processing: Data collection uses change data capture tools such as the database change data capture tool Debezium (CDC tools) in combination with the open-source distributed stream processing platform Apache Kafka to receive data streams from various data sources in real time, and transfer the real-time data to stream processing frameworks such as the fast and general cluster computing, components in the ecosystem for processing real-time data streams (Spark Streaming), the open-source, stream-batch integrated distributed computing framework (Flink), etc. for real-time processing and analysis, and store the processed data in HBase or Hive for subsequent querying and analysis.

[0094] Data Reception: For streaming data processing, data collection uses CDC (Change Data Capture) tools such as the database change data capture tool Debezium in combination with Apache Kafka to receive data streams from various data sources in real time. Debezium can monitor changes in the database, capture data insert, delete, and update operations in real time, and send these changes in the form of event streams. Apache Kafka is a high-throughput distributed publish-subscribe messaging system that can act as a buffer for data, receive and temporarily store data streams from Debezium, ensuring reliable data transmission.

[0095] Data Sources and Operations: Data sources can be various databases, log files, sensors, and other systems that generate data in real time. The stream processing framework passes the real-time data in the open-source distributed stream processing platform Kafka to stream processing frameworks such as Spark Streaming and Flink for real-time processing and analysis. Spark Streaming is an extended application programming interface of the Spark core application programming interface for processing real-time data streams. It slices the data stream by time slices (such as seconds) and converts it into a series of RDDs (Resilient Distributed Datasets) for processing. Flink is a distributed stream-batch integrated computing framework with the characteristics of low latency and high throughput in stream processing, and it can perform complex state calculations and window operations on real-time data streams.

[0096] Processing and Analysis Content: In the stream processing framework, operations such as real-time calculation, filtering, and aggregation can be performed on real-time data. For example, in the scenario of real-time e-commerce data analysis, the number of orders per minute can be counted in real time, and abnormal orders can be monitored in real time. Through these real-time processing and analysis, problems and trends in the business can be discovered in a timely manner, providing real-time support for decision-making.

[0097] Data Storage Selection: The processed data will be stored in HBase or Hive. Similar to batch processing, the storage method is selected according to the usage scenario of the data. If real-time query of the processed data is required, HBase is a better choice; if further data analysis and report generation are to be carried out, Hive is more suitable. The stored data can be used for subsequent queries and analysis. For example, through HBase, the real-time sales data of a certain product can be queried in real time to provide real-time information for inventory management; using Hive to deeply analyze the sales data over a period of time to generate sales reports, providing a basis for formulating market strategies.

[0098] In this embodiment, a distributed file system is used as the underlying data storage medium, and a data warehouse infrastructure, as well as a distributed, column-oriented open-source database, are built on top of the distributed file system. The steps for managing the original reconciliation data based on the data warehouse infrastructure and the open-source database include: using the distributed file system as the underlying data storage medium, analyzing through the mapping function programming mode, and building a data warehouse infrastructure and a distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data.

[0099] This solution has significant beneficial effects in many aspects such as storage, processing, management, and cost. In terms of storage, the distributed file system has strong scalability, can handle massive amounts of raw reconciliation data, and enhances data reliability through redundant storage; in terms of processing, the mapping function programming model supports parallel computing, has high processing efficiency, and can flexibly design processing logic according to business requirements; in terms of data management, the data warehouse and the column-oriented open-source database build a unified management platform, improving data standardization, and the column-oriented database can perform efficient query and analysis; in terms of cost, ordinary commercial hardware can be used to reduce hardware costs, and open-source software does not require high licensing fees and can be customized and extended as needed to save software costs.

[0100] Specifically, Hadoop, as the distributed system infrastructure, provides HDFS (Distributed File System) and MapReduce (Distributed Computing Model), which can handle large-scale data sets. HDFS divides data into multiple blocks and distributes them across multiple nodes in the cluster, providing high reliability and high scalability. MapReduce realizes the parallelization of complex data processing and improves processing efficiency.

[0101] HBase: A distributed columnar storage system based on Hadoop, HBase provides high-performance real-time read and write capabilities, especially suitable for processing unstructured and semi-structured data. Its columnar storage and indexing mechanism can accelerate the query process and meet the requirements of real-time data processing.

[0102] Hive: A data warehouse tool based on Hadoop, Hive allows users to query data in HDFS using SQL. Hive converts SQL queries into MapReduce jobs and executes them on the Hadoop cluster, simplifying the process of data query and analysis.

[0103] In this embodiment, the distributed relational database OceanBase can also be adopted. As a high-performance and scalable distributed relational database, OceanBase supports the Structured Query Language standard (SQL standard) and provides a data storage solution with strong consistency and high availability. It combines the advantages of traditional relational databases and non-relational databases (NoSQL databases) and is suitable for a variety of business scenarios.

[0104] The advantages of combining Hadoop, HBase, Hive, and OceanBase lie in aspects such as comprehensive data processing capabilities, flexibility and scalability, performance optimization, and wide applicability of scenarios. This combination method has become an important trend in the field of big data processing and analysis and provides strong support for various business needs.

[0105] In this embodiment, the original reconciliation data is stored in HDFS, and a large amount of original reconciliation data is imported into the distributed file system HDFS of Hadoop. HDFS divides the data into multiple data blocks and distributes them for storage on multiple nodes in the cluster, ensuring high reliability and high scalability of the data. For example, massive amounts of original data such as log files and sensor data can be first stored in HDFS. MapReduce preprocessing: If the original reconciliation data needs to be subjected to preliminary distributed computing and processing, the MapReduce distributed computing model of Hadoop can be utilized. MapReduce parallelizes complex data processing tasks and decomposes them into two stages: mapping (Map) and reduction (Reduce), processing the data simultaneously on multiple nodes in the cluster to improve processing efficiency. For example, preliminary processing such as format conversion and data cleaning of the original data is performed. HBase real-time reading and writing: For unstructured and semi-structured data that requires real-time reading and writing, the relevant data is extracted from HDFS and stored in HBase. HBase is built on top of Hadoop and provides high-performance real-time reading and writing capabilities. For example, in a real-time monitoring system, the real-time data generated by sensors can be written into HBase in real time, and fast query is also supported to meet the requirements of real-time data processing. Columnar storage and query acceleration: The columnar storage structure and index mechanism of HBase can accelerate the specific query process. When a query involves data in a specific column, HBase does not need to read the entire row of data, greatly improving the query efficiency. Hive SQL query: If users need to perform queries and analyses on the data stored in HDFS similar to those of traditional relational databases, they can rely on Hive. Users write SQL statements, and Hive converts them into MapReduce jobs for execution on the Hadoop cluster. For example, analysts want to analyze user behavior data. Through Hive SQL, they can conveniently perform complex queries on the user behavior log data stored in HDFS, such as counting the user activity levels in different time periods. OceanBase data storage: After preliminary processing and analysis, for data storage that requires strong consistency and high availability, as well as business scenarios suitable for the traditional relational database model, the data is stored in OceanBase. For example, the core business data of an enterprise, such as order data and user account information, has extremely high requirements for data consistency and availability and is suitable for storage in OceanBase. OceanBase supports the SQL standard, and enterprises can carry out various types of business development based on this, leveraging its advantages of combining traditional relational databases and NoSQL databases to meet the needs of multiple business scenarios, such as online transaction processing, data analysis reports, and other businesses.

[0106] In this embodiment, before the step of querying the target reconciliation data corresponding to the current verification period of the current time from the open-source database based on the user query instruction, the data processing method further includes: verifying the identity of the user to generate a verification result, and determining whether to allow the user to access based on the verification result; the verification result includes a success message and a failure message, and the step of determining whether to allow the user to access based on the verification result includes: in response to the verification result being a success message, allowing the user to access; in response to the verification result being a failure message, rejecting the user from accessing.

[0107] In terms of security, the above solution can prevent unauthorized access by verifying the user's identity, avoid data leakage, protect data integrity, and prevent unauthorized operations from damaging data. In terms of compliance, it can help enterprises meet industry regulations and standard requirements, avoid legal risks, and enhance industry reputation and competitiveness. In terms of stability, rejecting users who fail the verification can reduce the system burden, avoid invalid requests from occupying resources, and can also reasonably allocate resources according to user permissions, ensuring the system operation efficiency and stability, and ensuring the effective utilization of resources and the smooth operation of the system.

[0108] In this embodiment, the data mart implements user identity verification through the Kerberos protocol, which is used for secure identity verification in a distributed environment to ensure that only authorized users can access the cluster. At the same time, it supports mechanisms such as Role - Based Access Control (RBAC) and Access Control List (ACL). RBAC allows administrators to assign permissions according to the user's role, while ACL is used for more fine-grained permission control, such as read, write, and execute permissions for files and directories.

[0109] Specifically, the process of verifying the identity of the user can be achieved with the help of the Kerberos protocol. In a distributed environment, Kerberos is a widely used network authentication protocol that issues tickets through the Key Distribution Center (KDC) to verify the user's identity. The Kerberos protocol implements user identity verification, which is used for secure identity verification in a distributed environment to ensure that only authorized users can access the cluster.

[0110] Role - Based Access Control (RBAC): The data mart supports the RBAC mechanism, where administrators assign permissions based on the roles of users in the organization or system. For example, in an enterprise data mart, the role of data analysts may be granted query permissions to certain data sets, while the data administrator role has higher permissions such as adding, deleting, and modifying data. Through this role - based permission assignment method, the permissions of a large number of users can be efficiently managed, and when a user's role changes, the permission adjustment is more convenient.

[0111] Access Control List (ACL): In addition to RBAC, the data mart also uses ACL for more fine - grained permission control. ACL sets permissions for specific files and directories, specifying in detail which users or user groups have read, write, and execute permissions for specific files or directories. For example, for an important data file, only the data administrator group may be allowed write permissions, while ordinary analysts only have read permissions, further ensuring the security and integrity of the data.

[0112] In this embodiment, the data mart supports encrypting static data at the HDFS file system level. This usually involves using a key management service (such as Apache Key Management Service, KMS) to manage encryption keys and provides a transparent encryption function that allows users to encrypt data in HDFS without modifying the application code.

[0113] Specifically, encryption at the HDFS file system level: The data mart implements encryption protection for static data stored in the HDFS file system. In the big data storage scenario, HDFS is a commonly used distributed file system where data is stored in files for a long time. To prevent data from being stolen or tampered with in the static state, the data mart uses encryption means.

[0114] Key Management Service (KMS): During the encryption process, a key management service (such as Apache KMS) is used to properly manage encryption keys. Keys are like the "keys" to unlock encrypted data, and their security is crucial. KMS is responsible for operations such as key generation, storage, distribution, and update to ensure the confidentiality and availability of keys.

[0115] Transparent encryption function: The transparent encryption function provided by the data mart is very convenient. Users can encrypt data in HDFS without modifying the application code. This means that whether it is the original data processing program or a newly developed application, the data is automatically encrypted when written to HDFS and automatically decrypted when read. The whole process is almost transparent to users and application programs, greatly reducing the impact of encryption operations on the business.

[0116] In this embodiment, the data mart uses the Secure Sockets Layer (SSL) and Transport Layer Security (TLS), namely the SSL / TLS protocol, to encrypt the network communication between nodes in the cluster, ensuring the security of data during transmission. In addition, specific network traffic, such as remote management operations or data transfer, can also be encrypted through Secure Shell (SSH) tunnels.

[0117] Specifically, the SSL / TLS protocol: To ensure the security of data during transmission between cluster nodes, the data mart uses the SSL / TLS protocol. These are two widely used security protocols in network communication. By establishing an encrypted channel between the communicating parties, the data being transmitted is encrypted to prevent it from being stolen, tampered with, or eavesdropped on during transmission. For example, when data is transmitted from a computing node to a storage node, the data will be transmitted through the SSL / TLS encrypted channel.

[0118] SSH tunnels: In addition to the SSL / TLS protocol, the data mart can also encrypt specific network traffic, such as remote management operations or data transfer, through SSH tunnels. SSH (Secure Shell) is a network protocol commonly used in scenarios such as remote login and file transfer. By creating an SSH tunnel, specific network traffic can be encapsulated within an SSH connection and encrypted using the SSH encryption mechanism to further ensure the security of data transmission in specific business scenarios.

[0119] In this embodiment, the steps of validating the target reconciliation data to obtain a validation result include: comparing the reconciliation data of any two different business departments in the target reconciliation data and obtaining a comparison result; in response to the comparison result indicating that the reconciliation data between the two is inconsistent, determining that the validation result is an abnormal account; in response to the comparison result indicating that the reconciliation data between the two is consistent, determining that the validation result is a normal account.

[0120] The verification result is obtained by verifying the target reconciliation data. In terms of ensuring data accuracy, comparing the reconciliation data of different business departments can promptly detect data differences, avoid incorrect decisions and financial risks caused by inconsistent data, and ensure that the enterprise conducts operation and management based on accurate data. At the level of optimizing business processes, the identification of abnormal accounts helps the enterprise deeply investigate possible loopholes, errors or irregular operations in business processes, such as data entry errors and deviations in business process execution, and then make targeted improvements and optimizations to enhance the standardization and efficiency of business processes. From the perspective of enterprise internal control, this verification mechanism strengthens the mutual supervision and checks and balances of data among departments, forms an effective internal control system, prevents fraud and deception behaviors, and safeguards the safety of enterprise assets and financial health. Moreover, this standardized verification process is highly repeatable and can be continuously applied to subsequent reconciliation work, providing long-term and stable support for enterprise data management and business operations.

[0121] Specifically, professional data quality management tools (such as Informatica for data integration and management, and the open-source big data integration platform Talend, etc.) can be used to automatically perform tasks such as data inspection, cleaning, and verification. Tools like Informatica and Talend are widely used in the field of data quality management. They are comprehensive data integration and management platforms designed to help enterprises handle complex data management tasks. These tools can automatically execute key tasks such as data inspection, cleaning, and verification. In terms of data inspection, the integrity, accuracy, consistency, etc. of data can be checked according to preset rules; in the data cleaning process, problems such as duplicate data, incorrect data, and missing data can be identified and processed; the verification process ensures that the data complies with specific business rules and standards. Through automated operations, the efficiency of data quality management is greatly improved, and errors that may be caused by manual intervention are reduced.

[0122] Data quality verification tools: Such as the data quality verification tool Great Expectations and the open-source data quality monitoring tool Soda Core (formerly SodaSQL), these tools provide easily configurable and integrable data quality verification functions, can automate existing processes, and reduce the cost of large-scale implementation of data verification components. These tools provide easily configurable and integrable data quality verification functions. Users can quickly configure verification rules according to their own data characteristics and business requirements without complex programming and system setup. At the same time, these tools can be well integrated with existing data processing processes and can seamlessly integrate into large data platforms, data warehouses, or other data environments. By automating existing processes, the cost of large-scale implementation of data verification components is reduced, enabling enterprises to effectively control resource investment while ensuring data quality.

[0123] Data Extraction, Transformation, Loading Tools (ETL Tools): In the data extraction, transformation, and loading (ETL) process, ETL tools (such as web interface-based visual data integration tools (Apache NiFi), open-source workflow management platforms (Apache Airflow), etc.) are used to ensure data accuracy and consistency. The ETL process is an important link in the data processing flow, responsible for extracting data from data sources, transforming it to meet the requirements of the target system, and then loading it into the target system. In the ETL process, these tools play a key role in ensuring data accuracy and consistency. Apache NiFi can easily design and manage the data extraction, transformation, and loading process through its visual data flow design interface. In the data transformation link, various data processing rules can be set to ensure that the data meets the target format and quality requirements. Apache Airflow, with its powerful workflow scheduling function, ensures that ETL tasks are executed accurately according to the scheduled time and order, avoiding chaos and errors in the data processing process, and guaranteeing data accuracy and consistency at the process level.

[0124] Furthermore, users can also write custom scripts, such as scripts in the Python programming language and Shell scripts for the interface program between the operating system and users according to specific needs, to check and verify specific data. The Python language has rich data analysis and processing libraries, such as the open-source data analysis and processing library Pandas, the basic scientific computing library (Numerical Python, NumPy), etc., which can easily read, analyze, process, and verify data. Shell scripts are commonly used for the automated execution of system commands and simple processing of data files in an environment based on an open-source operating system (Linux system). When facing specific data formats, business rules, or personalized data quality inspection requirements, general tools may not be sufficient, and writing custom scripts becomes a flexible and effective solution. For example, for data files with special formats or specific business logics, accurate data checks and verifications can be achieved by writing Python scripts to ensure that the data quality meets specific requirements.

[0125] In this embodiment, the steps of validating the target reconciliation data to obtain a validation result include: obtaining historical reconciliation data corresponding to the previous verification period, determining the reconciliation data difference based on the historical reconciliation data and the target reconciliation data, and if the reconciliation data difference is inconsistent with the preset difference, determining that the validation result is an abnormal account.

[0126] Through the above method, potential risks can be warned in a timely manner. Once the difference in the reconciliation data does not match the preset value, financial risks, business mistakes or external influencing factors can be quickly detected, facilitating the enterprise to respond in a timely manner and avoid the expansion of losses. At the same time, the data accuracy and reliability can be effectively improved. By using the regularity of historical data to verify the current data, problems such as input errors and system failures can be identified and corrected, providing accurate basis for decision-making.

[0127] During the entire data processing process, processing logs can be generated. For example, the set of various information recorded during the operation of the Hive Server 2 component (HiveServer2 logs). HiveServer2 can be configured to record the SQL operation logs of users, including query statements, execution times, user information, etc., which can be used for data access auditing. For example, for auditing tables, create a dedicated auditing table in Hive, and through triggers or the Hooks mechanism of Hive's programming mechanism, automatically record operation information into the auditing table when data operations occur. Hive Metadata Storage Service (Hive Metastore): HiveMetastore stores the metadata of the Hive database, including table structures, partition information, etc. These metadata can be used to track data changes and sources. Further, regular backups can be performed, regularly backing up the data of Hadoop, HBase, Hive, and OceanBase to ensure rapid recovery in case of data loss or damage.

[0128] In this solution, after the step of validating the target reconciliation data to obtain a validation result, the data processing method further includes: tracking the validation process based on a monitoring tool; tracking the validation result based on the monitoring tool, and generating an exception reminder message in response to the validation result being an account anomaly.

[0129] In terms of tracking the validation process in this solution, relevant personnel can be enabled to grasp the progress and status of the validation work in real time, and timely discover problems such as jams and errors that may occur during the validation process, ensuring the smooth progress of the validation work. The tracking of the validation result can ensure that once an account anomaly occurs, the system can quickly generate an exception reminder message, enabling the enterprise to promptly know the problem, gain valuable time for taking corrective measures in a timely manner and investigating the cause of the anomaly, avoiding the further deterioration of the anomaly, ensuring the accuracy of the enterprise's financial data and the stability of business operations, and enhancing the enterprise's ability and efficiency to respond to risks.

[0130] Specifically, the viewing and distribution of verification reports can be implemented through Hive. Hive supports SQL queries, can generate reports and export them to common report tools for viewing. With the monitoring tools of HBase and the integration of third-party monitoring software: the open-source distributed monitoring system (Ganglia) and the open-source network monitoring system (Nagios), the verification process and results can be tracked in real time. At the same time, reasonable alarm thresholds are set. When the verification results are abnormal, relevant personnel are notified in a timely manner by means of emails, text messages, etc., to ensure the accuracy and timeliness of data verification and ensure that problems can be processed in a timely manner.

[0131] In this embodiment, after the step of validating the target reconciliation data to obtain a validation result, the data processing method further includes: presenting the target reconciliation data and / or the validation process of the target reconciliation data in a preset presentation manner.

[0132] In this solution, by clearly and intuitively presenting the target reconciliation data, relevant personnel can quickly understand the overall situation and key information of the data, improve the information acquisition efficiency, and facilitate making decisions quickly.

[0133] Specifically, by using data visualization tools (Tableau), business analysis tools (PowerBI), open-source data visualization libraries (Data-Driven Documents JavaScript library, D3.js), etc., these data visualization tools support multiple data sources and provide rich chart types and interactive interfaces to help users better understand and analyze data. Import the processed data into the selected visualization tool. Present the data in ways such as charts, graphs, dashboards, etc., including bar charts, line charts, scatter plots, pie charts, etc., to intuitively display the trends, distributions, and correlation relationships of the data. Users can be allowed to freely explore and operate the data through the interactive interface to achieve dynamic presentation and in-depth analysis. Throughout the process, attention needs to be paid to the security and privacy protection of data. Ensure the security of data through measures such as access control, data encryption, and data desensitization. Avoid exposing sensitive data in public places to prevent the risks of information leakage and privacy leakage.

[0134] The following will combine Figure 3 to sort out the entire data processing process. Figure 3 FIG. 16 is a schematic structural diagram of a module corresponding to a data processing method provided in Embodiment 1 of the present disclosure. Suppose there are three different departments, and the corresponding systems of different departments are A, B, and C respectively. First, obtain the original reconciliation data of different departments based on the data extraction program, store it in the data mart, and query the data from the data mart based on the user query instruction to obtain the target reconciliation data. Then start the reconciliation processing program to validate the target reconciliation data to obtain a validation result.

[0135] Embodiment 2

[0136] Corresponding to the foregoing embodiments of the data processing method, the present disclosure also provides an embodiment of a data processing system.

[0137] Figure 4 It is a schematic diagram of modules of a data processing system provided in Embodiment 2 of the present disclosure. The data processing system includes:

[0138] A data acquisition module 1 for acquiring original reconciliation data, where the original reconciliation data is reconciliation data from different business departments;

[0139] A data management module 2 for using a distributed file system as the underlying data storage medium, and building a data warehouse infrastructure and a distributed, column-oriented open-source database on top of the distributed file system to manage the original reconciliation data based on the data warehouse infrastructure and the open-source database;

[0140] A data query module 3 for querying, based on a user query instruction, target reconciliation data corresponding to the current verification period of the current time from the original reconciliation data in the open-source database; the target reconciliation data includes reconciliation data of two or more different business departments;

[0141] A data verification module 4 for performing verification processing on the target reconciliation data to obtain a verification result.

[0142] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present disclosure.

[0143] Embodiment 3

[0144] Figure 5 It is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored on the memory and used to run on the processor. When the processor executes the computer program, it implements the data processing method described in any of the foregoing embodiments. Figure 5 The displayed electronic device 50 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0145] Such as Figure 5As shown, the electronic device 50 may be presented in the form of a general-purpose computing device. For example, it may be a server device. The components of the electronic device 50 may include, but are not limited to: the at least one processor 51 described above, the at least one memory 52 described above, and a bus 53 that connects different system components (including the memory 52 and the processor 51).

[0146] The bus 53 includes a data bus, an address bus, and a control bus.

[0147] The memory 52 may include volatile memory, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0148] The memory 52 may also include a program tool 525 (or utility) having a set (at least one) of program modules 524. Such program modules 524 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0149] The processor 51 executes various functional applications and data processing by running computer programs stored in the memory 52, such as the data processing method provided in any of the above embodiments.

[0150] The electronic device 50 may also communicate with one or more external devices 54 (such as a keyboard, a pointing device, etc.). Such communication may be carried out through an input / output (I / O) interface 55. And, the electronic device 50 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 56. As shown in the figure, the network adapter 56 communicates with other modules of the electronic device 50 through the bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 50, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0151] It should be noted that, although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / modules. Conversely, the features and functions of one unit / modules described above may be further divided and embodied by multiple units / modules.

[0152] Embodiment 4

[0153] Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the data processing method provided in any of the above embodiments is implemented.

[0154] Among them, more specifically, the readable storage medium may include but is not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0155] Embodiment 5

[0156] Embodiments of the present disclosure also provide a computer program product, including a computer program, and when the computer program is executed by a processor, the data processing method described in any of the above is implemented.

[0157] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0158] Although the specific embodiments of the present disclosure have been described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Without departing from the principles and essence of the present disclosure, those skilled in the art can make various changes or modifications to these embodiments, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Collecting original reconciliation data, where the original reconciliation data is reconciliation data from different business departments; A distributed file system is used as the underlying data storage medium, and a data warehouse infrastructure and a distributed, column-oriented open source database are constructed on the distributed file system, so as to manage the original reconciliation data based on the data warehouse infrastructure and the open source database; Based on the user query instruction, query the original reconciliation data of the open source database to obtain target reconciliation data corresponding to the current verification period; the target reconciliation data includes reconciliation data of two or more different business departments; The target reconciliation data is verified to obtain a verification result.

2. The data processing method according to claim 1, characterized in that: The step of verifying the target reconciliation data to obtain a verification result includes: Comparing the reconciliation data of any two different business departments in the target reconciliation data, and obtaining a comparison result; In response to the comparison result indicating that the reconciliation data between the two are inconsistent, determining that the verification result is an account abnormality; In response to the comparison result indicating that the reconciliation data between the two are consistent, it is determined that the verification result is that the account is normal.

3. The data processing method according to claim 2, characterized in that: After the step of verifying the target reconciliation data to obtain a verification result, the data processing method further includes: Track verification progress based on monitoring tools; and / or, Based on the monitoring tool tracking verification results, in response to the verification result being that the account is abnormal, abnormal reminder information is generated.

4. The data processing method according to claim 1, characterized in that: The step of verifying the target reconciliation data to obtain a verification result includes: Obtain historical reconciliation data corresponding to the previous verification period; Determining a reconciliation data difference based on the historical reconciliation data and the target reconciliation data; In response to the reconciliation data difference being inconsistent with a preset difference, the verification result is determined to be an account abnormality.

5. The data processing method according to claim 1, characterized in that: The step of using a distributed file system as an underlying data storage medium, and building a data warehouse infrastructure framework and a distributed, column-oriented open source database on the distributed file system, so as to manage the original reconciliation data based on the data warehouse infrastructure framework and the open source database, includes: The distributed file system is used as the underlying data storage medium, and analysis is performed through a mapping function programming model. The data warehouse infrastructure and the distributed, column-oriented open source database are built on top of the distributed file system to manage the original reconciliation data.

6. The data processing method according to any one of claims 1 to 5, characterized in that: After the step of collecting the original reconciliation data, the data processing method further includes: Preprocessing the original reconciliation data according to different business requirements; the business requirements include requiring real-time transmission or not requiring real-time transmission; The step of preprocessing the original reconciliation data according to different business requirements specifically includes: In response to the business requirement that real-time transmission is required, a streaming processing method is used to process the original reconciliation data; In response to the business requirement that no real-time transmission is required, a batch processing method is used to process the original reconciliation data; and / or, Before the step of querying and obtaining target reconciliation data corresponding to the current verification period from the open source database based on the user query instruction, the data processing method further includes: Verify the user's identity and generate verification results; Determining whether to allow the user to access based on the verification result; The verification result includes success information and failure information, and the step of determining whether to allow the user to access based on the verification result includes: In response to the verification result being the success information, allowing the user to access; In response to the verification result being the failure information, denying the user access; and / or, After the step of verifying the target reconciliation data to obtain a verification result, the data processing method further includes: The target reconciliation data and / or the verification process of the target reconciliation data are displayed in a preset display manner.

7. A data processing system, characterized in that: The data processing system comprises: A data collection module is used to collect original reconciliation data, where the original reconciliation data is reconciliation data from different business departments; A data management module, configured to use a distributed file system as an underlying data storage medium, and to construct a data warehouse infrastructure framework and a distributed, column-oriented open source database on the distributed file system, so as to manage the original reconciliation data based on the data warehouse infrastructure framework and the open source database; A data query module, configured to query the original reconciliation data of the open source database based on a user query instruction to obtain target reconciliation data corresponding to the current verification period; the target reconciliation data includes reconciliation data of two or more different business departments; The data verification module is used to verify the target reconciliation data to obtain a verification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images