Data processing method and device, computer device, and storage medium

Abstract

The application belongs to the fields of big data and financial technology, and relates to a data processing method, comprising the following steps: obtaining incremental data of a target data table in a business main database based on a target processing link and sending the incremental data to a first Kafka cluster; loading the incremental data to a first data table of a first big data platform; obtaining burying point data corresponding to the target data table and writing the burying point data to a second Kafka cluster; loading the burying point data to a second data table of a second big data platform; performing data verification processing on the incremental data and the burying point data to generate a data verification result; and executing a corresponding data synchronization business processing flow based on the data verification result. The application also provides a data processing device, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the incremental data can be stored in a blockchain. The application can be applied to a business data verification scene in the financial field, and effectively improves the processing efficiency of data synchronization processing of the data table.

Description

Technical Field

[0001] This application relates to the fields of big data technology and fintech, and in particular to data processing methods, apparatus, computer equipment and storage media. Background Technology

[0002] Fintech companies, such as insurance companies and banks, typically use Oracle databases to store business data. They often need to extract business data from large tables in Oracle databases, synchronize it to a big data platform, and then use this data for data analysis and reporting. However, current fintech companies usually synchronize all business data from these large Oracle tables using a full-scale synchronization strategy. This approach is inefficient due to the sheer volume of data in these tables, leading to slow processing times and frequent synchronization failures, ultimately causing delays or even rendering data analysis unusable. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, apparatus, computer equipment, and storage medium to solve the technical problem of low processing efficiency in the data synchronization methods used by existing financial technology companies for business data of large data tables in Oracle databases.

[0004] To address the aforementioned technical problems, this application provides a data processing method, employing the following technical solution:

[0005] Based on the target processing link corresponding to the preset business master database, the incremental data corresponding to the target data table in the business master database is obtained, and the incremental data is sent to the preset first Kafka cluster;

[0006] The incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform;

[0007] Obtain the tracking data corresponding to the target data table in the main business database, and write the tracking data to the preset second Kafka cluster;

[0008] Load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform;

[0009] Perform data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data.

[0010] Based on the data verification results, the corresponding data synchronization business processing flow is executed.

[0011] Furthermore, the step of performing data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data specifically includes:

[0012] The incremental data and the embedded data are verified based on a preset data count verification method.

[0013] If the number of records passes the verification, the incremental data and the data tracking points are verified using a preset field verification method.

[0014] If the field validation passes, a first data validation result is generated, showing that the incremental data and the data point data have passed the data validation.

[0015] If the field validation fails, a second data validation result is generated, indicating that the incremental data and the data corresponding to the embedded data have failed validation.

[0016] Furthermore, the step of performing field validation on the incremental data and the tracking data based on a preset field validation method specifically includes:

[0017] Based on a preset data parallel processing strategy, the incremental data and the embedded data are subjected to numerical matching of all fields to obtain the corresponding numerical matching results.

[0018] Determine whether all the numerical matching results are successful matches;

[0019] If so, the field verification result between the incremental data and the embedded data is determined to be that the field verification passed;

[0020] If not, the field validation result between the incremental data and the embedded data is determined to be that the field validation failed.

[0021] Furthermore, the step of executing the corresponding data synchronization business processing flow based on the data verification result specifically includes:

[0022] If the data verification result is that the data verification is successful, a notification message indicating that the data synchronization corresponding to the incremental data is successful will be generated.

[0023] Obtain the communication information of the target business personnel;

[0024] Based on the communication information, the reminder message is sent to the target business personnel.

[0025] Furthermore, the step of executing the corresponding data synchronization business processing flow based on the data verification result specifically includes:

[0026] If the data verification result is that the data verification fails, then the difference data between the incremental data and the embedded data is obtained;

[0027] Based on the difference data, the incremental data is complemented to obtain the processed target incremental data;

[0028] The incremental data is replaced using the target incremental data.

[0029] Furthermore, the step of loading the incremental data in the first Kafka cluster into the first data table in the preset first big data platform specifically includes:

[0030] Invoke the preset distributed computing engine;

[0031] Obtain the address information of the first big data platform;

[0032] Based on the address information, the distributed computing engine is used to consume the incremental data in the first Kafka cluster, and the consumed incremental data is loaded into the first data table of the first big data platform.

[0033] Furthermore, before the step of obtaining the tracking data corresponding to the target data table in the business master database and writing the tracking data to the preset second Kafka cluster, the method further includes:

[0034] Receive the input data tracking requirements;

[0035] Construct corresponding tracking code based on the tracking requirement information;

[0036] The tracking code is used to perform tracking processing on the source system corresponding to the main business database.

[0037] To address the aforementioned technical problems, this application also provides a data processing apparatus, which employs the following technical solution:

[0038] The first processing module is used to obtain incremental data corresponding to the target data table in the business master database based on the target processing link corresponding to the preset business master database, and send the incremental data to the preset first Kafka cluster.

[0039] The first loading module is used to load the incremental data in the first Kafka cluster into the first data table in the preset first big data platform;

[0040] The second processing module is used to obtain the tracking data corresponding to the target data table in the business master database, and write the tracking data to the preset second Kafka cluster;

[0041] The second loading module is used to load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform;

[0042] The verification module is used to perform data verification processing on the incremental data in the first data table and the embedded data in the second data table, and generate a data verification result between the incremental data and the embedded data.

[0043] The execution module is used to execute the corresponding data synchronization business processing flow based on the data verification result.

[0044] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0045] Based on the target processing link corresponding to the preset business master database, the incremental data corresponding to the target data table in the business master database is obtained, and the incremental data is sent to the preset first Kafka cluster;

[0046] The incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform;

[0047] Obtain the tracking data corresponding to the target data table in the main business database, and write the tracking data to the preset second Kafka cluster;

[0048] Load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform;

[0049] Perform data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data.

[0050] Based on the data verification results, the corresponding data synchronization business processing flow is executed.

[0051] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0052] Based on the target processing link corresponding to the preset business master database, the incremental data corresponding to the target data table in the business master database is obtained, and the incremental data is sent to the preset first Kafka cluster;

[0053] The incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform;

[0054] Obtain the tracking data corresponding to the target data table in the main business database, and write the tracking data to the preset second Kafka cluster;

[0055] Load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform;

[0056] Perform data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data.

[0057] Based on the data verification results, the corresponding data synchronization business processing flow is executed.

[0058] Compared with the prior art, the embodiments of this application have the following main advantages:

[0059] This application embodiment first obtains incremental data corresponding to the target data table in the business master database based on the target processing link corresponding to the preset business master database, and sends the incremental data to the preset first Kafka cluster; then, the incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform; next, the tracking data corresponding to the target data table in the business master database is obtained and written into the preset second Kafka cluster; the tracking data in the second Kafka cluster is loaded into the second data table in the preset second big data platform; subsequently, data verification processing is performed on the incremental data in the first data table and the tracking data in the second data table to generate a data verification result between the incremental data and the tracking data; finally, the corresponding data synchronization business processing flow is executed based on the data verification result. This application embodiment obtains incremental data corresponding to the target data table in the business master database by using the target processing link corresponding to the business master database, and then performs data synchronization processing on the target data table by incremental synchronization based on the incremental data, thereby avoiding full table synchronization of the target data table and effectively improving the processing efficiency of data table synchronization. In addition, the incremental data corresponding to the target data table in the main business database will be used to verify the incremental data corresponding to the target data table. Based on the data verification results, the corresponding data synchronization business process will be executed to ensure the accuracy of the data synchronization process corresponding to the target data table. This helps to avoid synchronization failures, which could lead to data analysis delays or unusable data. Attached Figure Description

[0060] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0062] Figure 2 A flowchart of an embodiment of the data processing method according to this application;

[0063] Figure 3 This is a schematic diagram of the structure of an embodiment of the data processing apparatus according to this application;

[0064] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0066] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0068] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0069] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0070] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0071] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0072] It should be noted that the data processing method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the data processing device is generally located in the server / terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0074] Continue to refer to Figure 2 A flowchart illustrating an embodiment of the data processing method according to this application is shown. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different needs. The data processing method provided in this application embodiment can be applied to any scenario requiring business data verification, and thus can be applied to products in these scenarios, such as business data verification in the financial insurance field. The data processing method includes the following steps:

[0075] Step S201: Obtain incremental data corresponding to the target data table in the business master database based on the target processing link corresponding to the preset business master database, and send the incremental data to the preset first Kafka cluster.

[0076] In this embodiment, the data processing method runs on an electronic device (e.g., Figure 1 The server / terminal device shown can acquire incremental data corresponding to the target data table from the main business database via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods. The aforementioned main business database is the primary database used to store the business data of the target business system, and this primary database can specifically be an Oracle database. The aforementioned target processing link can specifically be an Oracle GG link. Oracle GoldenGate (OGG) is a high-performance, high-availability heterogeneous data replication technology provided by Oracle Corporation, which can realize data synchronization between different databases. The Oracle GG link captures changed data in the source database and transmits it to the target database, enabling real-time or periodic data synchronization, achieving real-time data backup, data migration, and data sharing operations. The aforementioned target data table refers to a large data table in the Oracle database used to store business data. In the business data verification scenario within fintech, the aforementioned target business systems may include insurance systems, banking systems, transaction systems, and order systems, while the business data may include business data, transaction data, payment data, and so on. Furthermore, the aforementioned first Kafka cluster is a Kafka cluster used to cache the incremental data.

[0077] Step S202: Load the incremental data in the first Kafka cluster into the first data table in the preset first big data platform.

[0078] In this embodiment, the aforementioned first big data platform may specifically adopt the HDFS distributed file system, and the aforementioned first data table is Hive table A in the HDFS distributed file system. The specific implementation process of loading the incremental data from the first Kafka cluster into the first data table of the preset first big data platform will be described in further detail in subsequent embodiments of this application, and will not be elaborated upon here.

[0079] Step S203: Obtain the tracking data corresponding to the target data table in the business master database, and write the tracking data to the preset second Kafka cluster.

[0080] In this embodiment, preset tracking codes can be configured in the main business database to automatically collect tracking data corresponding to the target data table in the main business database. The second Kafka cluster mentioned above is a Kafka cluster used to cache the tracking data.

[0081] Step S204: Load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform.

[0082] In this embodiment, the aforementioned second big data platform can specifically adopt the HDFS distributed file system, and the aforementioned first data table is Hive table B in the HDFS distributed file system. The process of loading the data points from the second Kafka cluster into the second data table of the preset second big data platform can be referred to as the process of loading the incremental data from the first Kafka cluster into the first data table of the preset first big data platform, and will not be elaborated further here.

[0083] Step S205: Perform data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data.

[0084] In this embodiment, the incremental data in the first data table and the embedded data in the second data table are subjected to data verification processing to generate a data verification result between the incremental data and the embedded data. The specific implementation process will be further described in detail in subsequent embodiments of this application, and will not be elaborated upon here.

[0085] Step S206: Execute the corresponding data synchronization business processing flow based on the data verification result.

[0086] In this embodiment, the specific implementation process of executing the corresponding data synchronization business process based on the data verification result will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0087] This application first obtains incremental data corresponding to a target data table in the business master database based on the target processing link corresponding to the preset business master database, and sends the incremental data to a preset first Kafka cluster; then, the incremental data in the first Kafka cluster is loaded into a first data table in a preset first big data platform; next, the tracking data corresponding to the target data table in the business master database is obtained and written into a preset second Kafka cluster; the tracking data in the second Kafka cluster is loaded into a second data table in a preset second big data platform; subsequently, data verification processing is performed on the incremental data in the first data table and the tracking data in the second data table to generate a data verification result between the incremental data and the tracking data; finally, the corresponding data synchronization business processing flow is executed based on the data verification result. This application obtains incremental data corresponding to a target data table in the business master database by using the target processing link corresponding to the business master database, and then performs data synchronization processing on the target data table by incremental synchronization based on the incremental data, thereby avoiding full table synchronization of the target data table and effectively improving the processing efficiency of data table synchronization. In addition, the incremental data corresponding to the target data table in the main business database will be used to verify the incremental data corresponding to the target data table. Based on the data verification results, the corresponding data synchronization business process will be executed to ensure the accuracy of the data synchronization process corresponding to the target data table. This helps to avoid synchronization failures, which could lead to data analysis delays or unusable data.

[0088] In some alternative implementations, step S205 includes the following steps:

[0089] The incremental data and the embedded data are verified based on a preset data count verification method.

[0090] In this embodiment, the above-mentioned preset record count verification method includes the following process for verifying the record count of the incremental data and the embedded data: obtaining the first total record count of the incremental data in the first data table and obtaining the second total record count of the embedded data in the second data table; then comparing whether the first total record count and the second record count are the same; if they are the same, the record count verification is determined to be passed; otherwise, the record count verification is determined to be failed.

[0091] If the number of records passes the verification, the incremental data and the embedded data are verified using a preset field verification method.

[0092] In this embodiment, the specific implementation process of performing field verification on the incremental data and the embedded data based on the preset field verification method will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0093] If the field verification passes, a first data verification result is generated, showing that the incremental data and the data points corresponding to the data verification have passed.

[0094] In this embodiment, if the incremental data in the first data table and the tracking data in the second data table pass the count verification and field verification, it will be determined that the incremental data in the main business database of the main link and the tracking data corresponding to the target data table in the source client are completely consistent, and then a first data verification result will be generated in which the data verification of the incremental data and the tracking data passes.

[0095] If the field validation fails, a second data validation result is generated, indicating that the incremental data and the data corresponding to the embedded data have failed validation.

[0096] In this embodiment, if the incremental data in the first data table and the tracking data in the second data table fail the count verification or the field verification, it will be determined that the incremental data in the main business database of the main link is not completely consistent with the tracking data corresponding to the target data table in the source client, and a second data verification result will be generated indicating that the data verification of the incremental data and the tracking data has failed.

[0097] This application verifies the incremental data and the tracking data based on a preset count verification method. If the count verification passes, it then verifies the fields of the incremental data and the tracking data based on a preset field verification method. If the field verification passes, a first data verification result is generated indicating that the data verification of the incremental data and the tracking data has passed. If the field verification fails, a second data verification result is generated indicating that the data verification of the incremental data and the tracking data has failed. This application verifies the incremental data in the first data table and the tracking data in the second data table based on preset count and field verification methods. This allows for the rapid and accurate generation of data verification results between the incremental data and the tracking data, improving the efficiency of data verification result generation and ensuring the accuracy of the generated data verification results.

[0098] In some optional implementations of this embodiment, the field verification of the incremental data and the embedded data based on a preset field verification method includes the following steps:

[0099] Based on a preset data parallel processing strategy, the incremental data and the embedded data are subjected to numerical matching of all fields to obtain the corresponding numerical matching results.

[0100] In this embodiment, the aforementioned parallel data processing strategy refers to parallel data computation using a SIMD model. Specifically, this can be achieved by performing data alignment operations on the incremental data and the embedded data to ensure the correspondence between the fields of the incremental data and the fields of the embedded data. Then, using multi-core and multi-threaded processing, parallel numerical matching processing of all fields is performed on the data-aligned incremental data and the embedded data to obtain the corresponding numerical matching results.

[0101] Determine whether all the numerical matching results are successful.

[0102] In this embodiment, content analysis is performed on each of the numerical matching results to identify whether all the numerical matching results are successful. The content of the numerical matching results includes whether the match is successful or unsuccessful.

[0103] If so, the field verification result between the incremental data and the embedded data is determined to be that the field verification passed.

[0104] If not, the field validation result between the incremental data and the embedded data is determined to be that the field validation failed.

[0105] This application performs numerical matching of all fields between the incremental data and the tracking data based on a preset data parallel processing strategy to obtain corresponding numerical matching results. Then, it determines whether all numerical matching results are successful. If so, the field verification result between the incremental data and the tracking data is determined to be successful; otherwise, it is determined to be unsuccessful. By using a data parallel processing strategy to perform numerical matching of all fields between the incremental data and the tracking data, and then performing content analysis on the obtained numerical matching results, this application can quickly and accurately generate field verification results between the incremental data and the tracking data, improving the efficiency of field verification result generation and ensuring the accuracy of the generated field verification results.

[0106] In some alternative implementations, step S206 includes the following steps:

[0107] If the data verification result is that the data verification is successful, a notification message indicating that the data synchronization corresponding to the incremental data is successful will be generated.

[0108] In this embodiment, the aforementioned reminder information is a message generated by the system that indicates successful data synchronization of incremental data related to the target data table in the business master database.

[0109] Obtain the contact information of the target business personnel.

[0110] In this embodiment, the aforementioned target business personnel may refer to the operation and maintenance personnel of the main business database. The aforementioned communication information may include email addresses or telephone numbers.

[0111] Based on the communication information, the reminder message is sent to the target business personnel.

[0112] In this embodiment, the reminder information can be sent to the communication terminal corresponding to the target business personnel based on the communication information.

[0113] If this application detects that the data verification result is successful, it generates a notification message indicating successful data synchronization corresponding to the incremental data. Then, it obtains the communication information of the target business personnel. Subsequently, based on the communication information, it sends the notification message to the target business personnel. When this application detects that the data verification result is successful, it intelligently generates a notification message indicating successful data synchronization corresponding to the incremental data and sends the notification message to the relevant target business personnel. This allows the target business personnel to promptly understand the successful data synchronization information by checking the notification message, enabling them to take subsequent processing measures for the incremental data, thereby improving the work efficiency and experience of the target business personnel.

[0114] In some alternative implementations, step S206 includes the following steps:

[0115] If the data verification result is that the data verification fails, then the difference data between the incremental data and the embedded data is obtained.

[0116] In this embodiment, if the data verification result is that the data verification fails, the difference between the incremental data and the embedded data will be recorded simultaneously during the data verification process of the incremental data in the first data table and the embedded data in the second data table.

[0117] The incremental data is complemented based on the difference data to obtain the processed target incremental data.

[0118] In this embodiment, the aforementioned difference data may include a specified field indicating a difference between the incremental data and the tracking data, a first field data in the incremental data corresponding to the specified field, and a second field data in the tracking data corresponding to the specified field. Then, the second field data in the tracking data is used to complement the first field data in the incremental data to obtain the complemented incremental data, i.e., the aforementioned target incremental data.

[0119] The incremental data is replaced using the target incremental data.

[0120] In this embodiment, the incremental data in the first data table of the first big data platform can be replaced by using the target incremental data to ensure that the synchronized incremental data is the same as the corresponding source data, that is, the tracking data corresponding to the target data table in the business master database.

[0121] If this application detects that the data verification result is a data verification failure, it obtains the difference data between the incremental data and the tracking data; then, based on the difference data, it performs complement processing on the incremental data to obtain the processed target incremental data; subsequently, it uses the target incremental data to perform data replacement processing on the incremental data. When this application detects that the data verification result is a data verification failure, it also intelligently performs complement processing on the incremental data based on the difference data between the incremental data and the tracking data, and uses the processed target incremental data to perform data replacement processing on the incremental data. This ensures that the synchronized incremental data is the same as the corresponding source data, i.e., the tracking data corresponding to the target data table in the business master database, thus guaranteeing the accuracy of data synchronization processing for the target data table.

[0122] In some optional implementations of this embodiment, step S202 includes the following steps:

[0123] Invoke the preset distributed computing engine.

[0124] In this embodiment, the distributed computing engine can specifically be Spark or Flink.

[0125] Obtain the address information of the first big data platform.

[0126] In this embodiment, the aforementioned address information may refer to the communication address information of the first big data platform, such as a URL address.

[0127] Based on the address information, the distributed computing engine is used to consume the incremental data in the first Kafka cluster, and the consumed incremental data is loaded into the first data table of the first big data platform.

[0128] In this embodiment, the incremental data in the first Kafka cluster can be consumed using Spark / Flink, and then the incremental data can be stored in the first big data platform and loaded into the first data table of the first big data platform.

[0129] This application invokes a preset distributed computing engine; then obtains the address information of the first big data platform; subsequently, based on the address information, it uses the distributed computing engine to consume the incremental data in the first Kafka cluster, and loads the consumed incremental data into the first data table of the first big data platform. This application utilizes a distributed computing engine to consume the incremental data in the first Kafka cluster, and then loads the consumed incremental data into the first data table of the first big data platform, thereby achieving rapid completion of incremental data loading processing and ensuring the smooth execution of incremental data loading.

[0130] In some optional implementations of this embodiment, before step S203, the electronic device may further perform the following steps:

[0131] Receive the input information regarding the required data points.

[0132] In this embodiment, the aforementioned data collection requirement information is generated by the relevant user based on the business requirements of data collection points for incremental data corresponding to big data tables in the main business database.

[0133] Based on the aforementioned tracking point requirements, construct the corresponding tracking point code.

[0134] In this embodiment, the corresponding tracking code can be constructed by filling the tracking requirement information into a preset tracking code template. The tracking code template is a code template constructed based on the actual tracking requirements.

[0135] The tracking code is used to perform tracking processing on the source system corresponding to the main business database.

[0136] In this embodiment, by using the tracking code to perform tracking processing on the source system corresponding to the business master database, the tracking data of incremental data corresponding to the big data table in the business master database can be automatically collected subsequently.

[0137] This application receives input tracking point requirements; then constructs corresponding tracking point code based on the requirements; and subsequently performs tracking point processing on the source system corresponding to the main business database based on the tracking point code. By utilizing the input tracking point requirements to construct corresponding tracking point code, and then performing tracking point processing on the source system corresponding to the main business database based on the code, this application can effectively and conveniently collect tracking point data corresponding to incremental data in the large data tables of the main business database. Furthermore, based on the obtained tracking point data, accurate data verification processing of the incremental data corresponding to the large data tables in the main business database can be achieved, thus improving the efficiency of data verification processing.

[0138] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0139] It should be emphasized that, to further ensure the privacy and security of the aforementioned incremental data, the incremental data can also be stored in a node of a blockchain.

[0140] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0141] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0142] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0144] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0145] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0146] like Figure 3 As shown, the data processing device 300 described in this embodiment includes: a first processing module 301, a first loading module 302, a second processing module 303, a second loading module 304, a verification module 305, and an execution module 306. Wherein:

[0147] The first processing module 301 is used to obtain incremental data corresponding to the target data table in the business master database based on the target processing link corresponding to the preset business master database, and send the incremental data to the preset first Kafka cluster.

[0148] The first loading module 302 is used to load the incremental data in the first Kafka cluster into the first data table in the preset first big data platform;

[0149] The second processing module 303 is used to obtain the tracking data corresponding to the target data table in the business master database, and write the tracking data to the preset second Kafka cluster;

[0150] The second loading module 304 is used to load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform;

[0151] Verification module 305 is used to perform data verification processing on the incremental data in the first data table and the embedded data in the second data table, and generate a data verification result between the incremental data and the embedded data.

[0152] The execution module 306 is used to execute the corresponding data synchronization business processing flow based on the data verification result.

[0153] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0154] In some optional implementations of this embodiment, the verification module 305 includes:

[0155] The first verification submodule is used to verify the number of records of the incremental data and the embedded data based on a preset record count verification method.

[0156] The second verification submodule is used to perform field verification on the incremental data and the embedded data based on a preset field verification method if the number of entries verification passes.

[0157] The first generation submodule is used to generate a first data verification result if the field verification passes, and the data verification of the incremental data and the embedded data passes.

[0158] The second generation submodule is used to generate a second data verification result if the field verification fails, indicating that the incremental data and the data corresponding to the embedded data have failed data verification.

[0159] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0160] In some optional implementations of this embodiment, the second verification submodule includes:

[0161] The matching unit is used to perform numerical matching processing on all fields of the incremental data and the embedded data based on a preset data parallel processing strategy to obtain the corresponding numerical matching result.

[0162] The judgment unit is used to determine whether all the numerical matching results are successful matches.

[0163] The first determination unit is used to determine, if yes, that the field verification result between the incremental data and the embedded data is that the field verification is passed.

[0164] The second determination unit is used to determine, if not, that the field verification result between the incremental data and the embedded data is that the field verification failed.

[0165] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0166] In some optional implementations of this embodiment, the execution module 306 includes:

[0167] The third generation submodule is used to generate a notification message indicating successful data synchronization for the incremental data if the data verification result is that the data verification is passed.

[0168] The first acquisition submodule is used to acquire the communication information of the target business personnel;

[0169] The sending submodule is used to send the reminder information to the target business personnel based on the communication information.

[0170] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0171] In some optional implementations of this embodiment, the execution module 306 includes:

[0172] The second acquisition submodule is used to acquire the difference data between the incremental data and the embedded data if the data verification result is that the data verification fails.

[0173] The first processing submodule is used to perform complementation on the incremental data based on the difference data to obtain the processed target incremental data.

[0174] The second processing submodule is used to perform data replacement processing on the incremental data using the target incremental data.

[0175] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0176] In some optional implementations of this embodiment, the first loading module 302 includes:

[0177] Calling submodules is used to invoke the preset distributed computing engine;

[0178] The third acquisition submodule is used to acquire the address information of the first big data platform;

[0179] The loading submodule is used to consume the incremental data in the first Kafka cluster based on the address information using the distributed computing engine, and load the consumed incremental data into the first data table of the first big data platform.

[0180] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0181] In some optional implementations of this embodiment, the data processing apparatus further includes:

[0182] The receiving module is used to receive the input data tracking requirements.

[0183] The building module is used to build the corresponding tracking code based on the tracking requirement information;

[0184] The event tracking module is used to perform event tracking processing on the source system corresponding to the main business database based on the event tracking code.

[0185] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the data processing method in the aforementioned embodiments, and will not be repeated here.

[0186] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0187] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0188] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0189] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0190] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the data processing method.

[0191] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0192] Compared with the prior art, the embodiments of this application have the following main advantages:

[0193] In this embodiment, firstly, incremental data corresponding to the target data table in the business master database is obtained based on the target processing link corresponding to the preset business master database, and the incremental data is sent to the preset first Kafka cluster; then, the incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform; next, the tracking data corresponding to the target data table in the business master database is obtained and written into the preset second Kafka cluster; and the tracking data in the second Kafka cluster is loaded into the second data table in the preset second big data platform; subsequently, data verification processing is performed on the incremental data in the first data table and the tracking data in the second data table to generate a data verification result between the incremental data and the tracking data; finally, the corresponding data synchronization business processing flow is executed based on the data verification result. This embodiment uses the target processing link corresponding to the business master database to obtain incremental data corresponding to the target data table in the business master database, and then performs incremental synchronization processing on the target data table based on this incremental data, thereby avoiding full table synchronization of the target data table and effectively improving the processing efficiency of data table synchronization. In addition, the incremental data corresponding to the target data table in the main business database will be used to verify the incremental data corresponding to the target data table. Based on the data verification results, the corresponding data synchronization business process will be executed to ensure the accuracy of the data synchronization process corresponding to the target data table. This helps to avoid synchronization failures, which could lead to data analysis delays or unusable data.

[0194] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the data processing method described above.

[0195] Compared with the prior art, the embodiments of this application have the following main advantages:

[0196] In this embodiment, firstly, incremental data corresponding to the target data table in the business master database is obtained based on the target processing link corresponding to the preset business master database, and the incremental data is sent to the preset first Kafka cluster; then, the incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform; next, the tracking data corresponding to the target data table in the business master database is obtained and written into the preset second Kafka cluster; and the tracking data in the second Kafka cluster is loaded into the second data table in the preset second big data platform; subsequently, data verification processing is performed on the incremental data in the first data table and the tracking data in the second data table to generate a data verification result between the incremental data and the tracking data; finally, the corresponding data synchronization business processing flow is executed based on the data verification result. This embodiment uses the target processing link corresponding to the business master database to obtain incremental data corresponding to the target data table in the business master database, and then performs incremental synchronization processing on the target data table based on this incremental data, thereby avoiding full table synchronization of the target data table and effectively improving the processing efficiency of data table synchronization. In addition, the incremental data corresponding to the target data table in the main business database will be used to verify the incremental data corresponding to the target data table. Based on the data verification results, the corresponding data synchronization business process will be executed to ensure the accuracy of the data synchronization process corresponding to the target data table. This helps to avoid synchronization failures, which could lead to data analysis delays or unusable data.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0198] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data processing method, characterized in that, Includes the following steps: Based on the target processing link corresponding to the preset business master database, the incremental data corresponding to the target data table in the business master database is obtained, and the incremental data is sent to the preset first Kafka cluster; The incremental data in the first Kafka cluster is loaded into the first data table in the preset first big data platform; Obtain the tracking data corresponding to the target data table in the main business database, and write the tracking data to the preset second Kafka cluster; Load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform; Perform data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data. Based on the data verification results, the corresponding data synchronization business processing flow is executed.

2. The data processing method according to claim 1, characterized in that, The step of performing data verification processing on the incremental data in the first data table and the embedded data in the second data table to generate a data verification result between the incremental data and the embedded data specifically includes: The incremental data and the embedded data are verified based on a preset data count verification method. If the number of records passes the verification, the incremental data and the data tracking points are verified using a preset field verification method. If the field validation passes, a first data validation result is generated, showing that the incremental data and the data point data have passed the data validation. If the field validation fails, a second data validation result is generated, indicating that the incremental data and the data corresponding to the embedded data have failed validation.

3. The data processing method according to claim 2, characterized in that, The step of performing field validation on the incremental data and the tracking data based on a preset field validation method specifically includes: Based on a preset data parallel processing strategy, the incremental data and the embedded data are subjected to numerical matching of all fields to obtain the corresponding numerical matching results. Determine whether all the numerical matching results are successful matches; If so, the field verification result between the incremental data and the embedded data is determined to be that the field verification passed; If not, the field validation result between the incremental data and the embedded data is determined to be that the field validation failed.

4. The data processing method according to claim 1, characterized in that, The steps of executing the corresponding data synchronization business process based on the data verification result specifically include: If the data verification result is that the data verification is successful, a notification message indicating that the data synchronization corresponding to the incremental data is successful will be generated. Obtain the communication information of the target business personnel; Based on the communication information, the reminder message is sent to the target business personnel.

5. The data processing method according to claim 1, characterized in that, The steps of executing the corresponding data synchronization business process based on the data verification result specifically include: If the data verification result is that the data verification fails, then the difference data between the incremental data and the embedded data is obtained; Based on the difference data, the incremental data is complemented to obtain the processed target incremental data; The incremental data is replaced using the target incremental data.

6. The data processing method according to claim 1, characterized in that, The step of loading the incremental data in the first Kafka cluster into the first data table in the preset first big data platform specifically includes: Invoke the preset distributed computing engine; Obtain the address information of the first big data platform; Based on the address information, the distributed computing engine is used to consume the incremental data in the first Kafka cluster, and the consumed incremental data is loaded into the first data table of the first big data platform.

7. The data processing method according to claim 1, characterized in that, Before the step of obtaining the tracking data corresponding to the target data table in the business master database and writing the tracking data to the preset second Kafka cluster, the method further includes: Receive the input data tracking requirements; Construct corresponding tracking code based on the tracking requirement information; The tracking code is used to perform tracking processing on the source system corresponding to the main business database.

8. A data processing apparatus, characterized in that, include: The first processing module is used to obtain incremental data corresponding to the target data table in the business master database based on the target processing link corresponding to the preset business master database, and send the incremental data to the preset first Kafka cluster; The first loading module is used to load the incremental data in the first Kafka cluster into the first data table in the preset first big data platform; The second processing module is used to obtain the tracking data corresponding to the target data table in the business master database, and write the tracking data to the preset second Kafka cluster; The second loading module is used to load the embedded data in the second Kafka cluster into the second data table in the preset second big data platform; The verification module is used to perform data verification processing on the incremental data in the first data table and the embedded data in the second data table, and generate a data verification result between the incremental data and the embedded data. The execution module is used to execute the corresponding data synchronization business processing flow based on the data verification result.

9. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 7.

Images