Data processing system, method and device and storage medium

Through the combined architecture of monitoring components and computing engine, automatic repairs in the data warehouse are realized, solving the problems of low efficiency and poor accuracy of manual repairs, and improving the maintainability of the data processing system.

CN118939638APending Publication Date: 2024-11-12MASHANG CONSUMER FINANCE CO LTD

Patent Information

Application Number
CN202310520088.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing data warehouse storage usage scheme cannot automatically correct data, resulting in low efficiency, high cost and error-prone manual repairs.

Method used

Adopt the architecture of monitoring components + message middleware + computing engine. The monitoring components obtain and parse data change logs of data sources in real time. The computing engine labels data based on data change operation information and writes it to the original data table of the data warehouse to automatically Revise the number.

Benefits of technology

It improves the number of repairs, reduces the probability of errors, ensures the accuracy of data changes, and improves the maintainability of the data processing system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939638A_ABST
    Figure CN118939638A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing system, method and device and a storage medium. The data processing system comprises a monitoring component, message middleware, a calculation engine and a data warehouse, the monitoring component is used for acquiring and analyzing a data change log of a data source in real time to obtain data operation information for the data source, and sending the data operation information to the message-oriented middleware, the data operation information comprises first target data and data change operation information corresponding to the first target data, and the data change operation information corresponds to the first target data; the first target data is changed data in the data source; the calculation engine is used for acquiring the data operation information from the message middleware, marking a corresponding operation label for first target data based on data operation change information, and writing the first target data and the operation label into an original data table of the data warehouse; and the data warehouse is used for analyzing and processing the data in the original data table based on the operation label of the data in the original data table.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system, method, device and storage medium. Background Art

[0002] The construction of a big data warehouse is a very important link in the field of big data. Maintaining the quality of data in the data warehouse is even more important. Otherwise, low-quality data will lose its value in the data warehouse.

[0003] Since the data warehouse comes from data sources such as business systems, data errors and duplications in the data warehouse are often caused by program problems of the data source. Since the current common data warehouse storage and use solutions all use the Hive+Hadoop architecture to form a big data query solution that can be used to implement structured query statements (Structured Query Language, SQL). This data warehouse storage and use solution cannot realize automatic correction of data on Hive. In this way, relevant personnel are required to manually check the correctness of the data in the data warehouse and manually repair the data in the data warehouse. However, this manual repair method has the problems of high labor cost, complicated operation, and low repair efficiency, and it is difficult to ensure that there are no problems in each repair, so there is a greater risk of error in repair. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data processing system, method, device and storage medium for realizing automatic data update of a data warehouse without human intervention, thereby improving the update efficiency and reducing the probability of errors in the update process.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a data processing system, including: a monitoring component, a message middleware, a computing engine, and a data warehouse;

[0007] The monitoring component is used to obtain and parse the data change log of the data source in real time, obtain data operation information for the data source, and send the data operation information to the message middleware, wherein the data operation information includes the first target data and the data change operation information corresponding to the first target data, and the first target data includes the data changed in the data source;

[0008] The computing engine is used to obtain the data operation information from the message middleware, and is also used to mark the first target data with a corresponding operation tag based on the data change operation information, and write the first target data and the operation tag of the first target data into an original data table of the data warehouse, wherein the original data table stores data synchronized from the data source, and the operation tag is used to indicate a data change operation performed on the first target data in the data source;

[0009] The data warehouse is used to analyze and process the data in the original data table based on the operation tags of the data in the original data table.

[0010] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a computing engine, and the method includes:

[0011] Acquire data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information is acquired by a monitoring component in real time and parsed after the data change log of the data source is obtained and sent to the message middleware;

[0012] Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source;

[0013] The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0014] In a third aspect, an embodiment of the present application provides a data processing device, which is applied to a computing engine, and includes:

[0015] an acquisition unit, configured to acquire data operation information for a data source from a message middleware, wherein the data operation information includes first target data and data change operation information corresponding to the first target data, wherein the first target data includes data changed in the data source, and the data operation information is acquired by a monitoring component in real time and after parsing a data change log of the data source, and then sent to the message middleware;

[0016] a marking unit, configured to mark a corresponding operation tag for the first target data based on the data change operation information, wherein the operation tag is used to indicate the data change operation performed on the first target data in the data source;

[0017] The access unit is used to write the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a computing engine, the computing engine is enabled to execute the data processing method as described in the second aspect.

[0019] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the ETL layer under the original data warehouse storage and use architecture is optimized and improved to the architecture of monitoring component + message middleware + computing engine, and the monitoring component obtains and parses the data change log of the data source in real time, obtains data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation tag for the data based on the data change operation information to indicate the data change operation performed on the first target data in the data source, and writes the data and its operation tag into the original data table of the data warehouse; since the process of sending data operation information from the monitoring component to the message middleware is real-time, the process of consuming data operation information from the message middleware and marking the data by the computing engine is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume the data operation information from the message middleware. After the data operation information is processed, it falls into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label; in this way, the data warehouse can timely and intuitively obtain the changes that have occurred in the data synchronized from the data source in the data source based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in the data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. In addition, the change of the data in the original data table is determined according to the data change operation on the data source, rather than the active change and determination of the data warehouse, so as to ensure the accuracy of the data change situation and reduce the probability of errors in the automatic adjustment process; in addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream has done on the data, the downstream can track and record according to the data operation label, so that it can accurately check and locate the problem when data processing errors occur, and improve the maintainability of the entire data processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A schematic diagram of a data modification scheme under the current data warehouse storage usage architecture;

[0022] Figure 2 A schematic diagram of the structure of a data processing system provided for one embodiment of the present application;

[0023] Figure 3 A structural schematic diagram of a data processing system provided for another embodiment of the present application;

[0024] Figure 4 A flowchart of a data processing method provided for one embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of a data processing device provided for one embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0028] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] Some concept descriptions:

[0030] Apache Spark: Spark is a fast and general computing engine designed for large-scale data processing. Spark is a general engine that can be used to perform a variety of operations, including SQL queries, text processing, machine learning, etc.

[0031] Apache Flink: Flink for short, it executes any streaming data program in a data-parallel and pipelined manner. When Flink's pipeline runs, the system can execute batch and stream processing programs.

[0032] Data warehouse: The purpose of a data warehouse is to build an integrated data environment for analysis and provide decision support for enterprises. It is created for analytical reporting and decision support purposes. The data warehouse itself does not "produce" any data, nor does it need to "consume" any data. The data comes from the outside and is open to external applications.

[0033] Apache Hive: Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table and provide SQL query functions, which can convert SQL statements into MapReduce tasks for execution.

[0034] Apache Hoodie: Apache Hudi brings core warehouse and database functionality directly into the data lake. Hudi provides tables, transactions, efficient append / delete, advanced indexing, streaming ingestion services, data clustering / compression optimizations, and concurrency, while keeping the data in an open source file format.

[0035] Apache Hudi can be easily used on any cloud storage platform. Hudi's advanced performance optimizations make analytical workloads faster for any popular query engine, including Apache Spark, Flink, Presto, Trino, Hive, and more.

[0036] Kafka: Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action streams of consumers on the website. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern network. This data is usually solved by processing logs and log aggregation due to throughput requirements.

[0037] MaxWell: Maxwell is a MySQL change data capture software written in Java. It monitors the data change operations (including insert, update, delete) of the MySQL database in real time, and sends the change data in JSON format to stream data processing platforms such as Kafka and Kinesi. Maxwell works by reading the binary log (Binlog) of the MySQL database in real time, obtaining the change data from it, and then sending the change data in JSON format to stream processing platforms such as Kafka.

[0038] ODS: Operation Data Store data preparation area, also known as the source layer. The data table of the source system of the data warehouse is usually stored intact, which is called the ODS layer and is the source of the subsequent data warehouse processing data.

[0039] DWD: data warehouse details, which is the isolation layer between the business layer and the data warehouse. It mainly performs some data cleaning and normalization operations on the ODS data layer.

[0040] DWS: Data warehouse service, based on the basic data on DWB, integrates and summarizes the service data layer for analyzing a certain subject domain, usually a wide table. It is used to provide subsequent business queries, OLAP analysis, data distribution, etc.

[0041] ETL: Extract-Transform-Load, data loading layer.

[0042] DIM: Dimension, public dimension layer.

[0043] ADS: Application Data Store, application data layer. Personalized dimension summary layer. Statistical dimension data that is not particularly common will be placed in this layer. Dimensions and indicators that only the business itself will pay attention to are calculated here.

[0044] JSON: It is a lightweight data exchange format. It is based on a subset of ECMAScript (js specification developed by the European Computer Association) and uses a text format that is completely independent of programming languages ​​to store and represent data.

[0045] Apache Doris: Abbreviated as Doris, it is a distributed database based on the Massively Parallel Processing (MMP) architecture, which has the advantages of low overhead and high performance.

[0046] Apache Sqoop: Sqoop for short, is a data extraction tool that can synchronize data from a relational database to HDFS (Hadoop's underlying storage system), and can also export HDFS data and store it back into a relational database.

[0047] As described in the background technology, the current common data warehouse storage and usage solutions all use the Hive+Hadoop architecture to form a big data query solution that can be used to implement SQL for structured query statements. This data warehouse storage and usage solution cannot achieve automatic correction of data on Hive. In this way, relevant personnel are required to manually check the correctness of the data in the data warehouse and manually repair the data in the data warehouse.

[0048] For example, Figure 1 As shown in the figure, under the current data warehouse storage and usage architecture, data is regularly obtained from the data source through the ETL layer, and the data is stored in the data warehouse after being extracted, transformed, and loaded. In order to ensure the synchronization of data in the data warehouse and the data source, and to ensure the reliability and accuracy of the subsequent data application process, it is often necessary to manually check the correctness of the data in the data warehouse with the help of Hive and manually correct the data.

[0049] However, this manual method of number correction has the problems of high labor cost, complicated operation, and low number correction efficiency. In addition, it is difficult to ensure that there will be no problems every time the number is corrected, so there is a greater risk of error in the number correction.

[0050] In view of this, an embodiment of the present application aims to provide a data processing system, which optimizes the ETL layer under the original data warehouse storage and usage architecture, and improves it to an architecture of monitoring component + message middleware + computing engine, wherein the monitoring component obtains and parses the data change log of the data source in real time, obtains data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation tag for the data based on the data change operation information to indicate the data change operation performed on the first target data in the data source, and writes the data and its operation tag into the original data table of the data warehouse; since the process of sending data operation information from the monitoring component to the message middleware is real-time, the process of consuming data operation information from the message middleware and marking the data by the computing engine is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume data from the message middleware. After the operation information is processed, it falls into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label; in this way, the data warehouse can timely and intuitively obtain what changes have occurred in the data synchronized from the data source in the data source based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in the data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. In addition, the change of the data in the original data table is determined according to the data change operation on the data source, rather than the active change and determination of the data warehouse, so as to ensure the accuracy of the data change situation and reduce the probability of errors in the automatic adjustment process; in addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream has done on the data, the downstream can track and record according to the data operation label, so that it can accurately check and locate the problem when data processing errors occur, and improve the maintainability of the entire data processing system.

[0051] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0052] See also Figure 2 , is a structural diagram of a data processing system provided in one embodiment of the present application, the data processing system includes: a monitoring component, a message middleware, a computing engine and a data warehouse.

[0053] Among them, message middleware is a middleware used for publishing and subscribing messages. It can receive messages published by producers and provide consumers with subscribed messages.

[0054] The monitoring component is a component that can monitor the data changes of the data source in real time. In an embodiment of the present application, the monitoring component can obtain and parse the data change log of the data source in real time to obtain data operation information for the data source, wherein the data operation information includes the first target data and the data change operation information corresponding to the first target data, the first target data includes the data that has changed in the data source, and the data change operation information corresponding to the first target data is used to describe the data change operation performed on the first target data in the data source. In addition, the monitoring component can also act as a producer to send the data operation information parsed from the data change log obtained in real time to the message middleware.

[0055] The computing engine is an engine for data computing, especially suitable for big data computing, and has parallel batch processing and stream processing capabilities. In an embodiment of the present application, the computing engine can act as a consumer and subscribe to data operation information for a data source from the message middleware. After obtaining the data operation information from the message middleware, the computing engine can mark the first target data with a corresponding operation tag according to the data change operation information contained therein. The operation tag of the first target data is used to indicate the data change operation performed on the first target data in the data source. Furthermore, the computing engine also writes the first target data and the operation tag of the first target data into the original data table of the data warehouse, wherein the original data table records the data synchronized from the data source by the data warehouse.

[0056] For example, the data change operation includes at least one of a deletion operation, an update operation, and a new addition operation. If the data change operation is a deletion operation, the first target data deleted by the data change operation may be marked with a corresponding deletion label (such as DELETE); if the data change operation is a new addition operation, the first target data added by the data change operation may be marked with a corresponding addition label (such as INSERT); if the data change operation is an update operation, the first target data updated by the data change operation may be marked with a corresponding update label (such as UPDATE). In this way, the operation label of the data can intuitively reflect what kind of data change operation is performed on the data in the data source.

[0057] The data warehouse is used to analyze and process the data in the original data table based on the operation tags of the data in the original data table.

[0058] It can be seen that the data processing system provided in the embodiment of the present application optimizes the ETL layer under the original data warehouse storage and usage architecture, and improves it to an architecture of monitoring component + message middleware + computing engine, in which the monitoring component obtains and parses the data change log of the data source in real time, obtains data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation tag for the data based on the data change operation information to indicate the data change operation performed on the first target data in the data source, and writes the data and its operation tag into the original data table of the data warehouse; since the process of sending data operation information from the monitoring component to the message middleware is real-time, the process of consuming data operation information from the message middleware and marking the data by the computing engine is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume data operation information from the message middleware. After the information is processed, it falls into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label; in this way, the data warehouse can timely and intuitively obtain what changes have occurred in the data synchronized from the data source in the data source based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in the data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. In addition, the change of data in the original data table is determined according to the data change operation on the data source, rather than the active change and determination of the data warehouse, so as to ensure the accuracy of data change situation and reduce the probability of errors in the automatic adjustment process; in addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream has done on the data, the downstream can track and record according to the data operation label, so that it can accurately check and locate the problem when data processing errors occur, and improve the maintainability of the entire data processing system.

[0059] In order for those skilled in the art to better understand the data processing system provided in the embodiments of the present application, each component of the data processing system is described in detail below.

[0060] The data processing system of the embodiment of the present application can access various data sources, such as Figure 3The business system shown. A relational database is deployed in the business system, and the data processing system can access the relational database, and the monitoring component obtains data operation information for structured data in the relational database in real time. Of course, a non-relational database can also be deployed in the business system, and the data processing system can also access the non-relational database, and the monitoring component obtains data operation information for unstructured data in the non-relational database in real time.

[0061] In the embodiment of the present application, the message middleware can adopt any appropriate publish-subscribe message system with high-throughput data transmission capability, such as Kafka, etc. The specific selection can be made according to actual needs, and the embodiment of the present application does not limit this.

[0062] In an embodiment of the present application, the monitoring component can adopt any appropriate component with data monitoring function. Optionally, the monitoring component can include a data capture component such as MaxWell. MaxWell can obtain the binary log (BINLOG) of the data source in real time, that is, the data change log. The binary log records the data operation information for the data source. MaxWell can obtain the data operation information by parsing the binary log and send the data operation information to the message middleware in JSON format.

[0063] In the embodiment of the present application, the computing engine can adopt any appropriate engine with parallel batch processing and streaming processing capabilities, such as Flink, Spark, etc., which can be selected according to actual needs, and the embodiment of the present application does not limit this. In a more preferred solution, since Spark's parallel batch processing and streaming data processing capabilities are more mature and more suitable for actual business development needs, the computing engine can adopt Spark. In addition, another advantage of using Spark as a computing engine is that the original data table in the data warehouse is usually a Hoodie table. Since Hoodie has a corresponding API to connect with Spark, Spark only needs to import the Jar package corresponding to Hooide to use it in the program, and the first target data and its operation label can be dropped into the Hoodie table, which is simple to implement. In the embodiment of the present application, the computing engine can write the first target data and its operation label into the original data table in various appropriate ways, which can be selected according to actual needs, and the embodiment of the present application does not limit this.

[0064] Optionally, in order to enable the data warehouse to more intuitively and accurately understand what changes have occurred in the data in the original data table in the data source, and thus make more accurate analysis and processing decisions, the original data table may include data fields and status fields, wherein the data field is used to record the data synchronized from the data source by the data warehouse, and the status field is used to record the status of the data synchronized from the data source in the data source, such as whether the data is new data in the data source, whether the data has been deleted from the data source, and whether the data has been updated in the data source.

[0065] Correspondingly, in the process of writing the first target data and the operation tag of the first target data into the original data table of the data warehouse, the computing engine is specifically used to: update the field content of the data field based on the first target data, and update the field content of the status field based on the operation tag of the first target data.

[0066] Specifically, if the operation tag of the first target data is a delete tag, it indicates that a delete operation has been performed on the first target data in the data source, and then the field content corresponding to the first target data under the status field can be updated to a delete tag to indicate that the first target data has been deleted from the data source. For example, Table 1 shows an example of an original data table, which includes a data field "data" and a status field "status". If a delete operation is performed on a in the data source, then the operation tag of a is a delete tag "DELETE", and the field content corresponding to a under the status field can be updated to a delete tag to indicate that a has been deleted from the data source. The updated original data table is shown in Table 2.

[0067] Table 1

[0068] data state a NULL b NULL …… ……

[0069] Table 2

[0070] data state a DELETE b NULL …… ……

[0071] It is understandable that when a deletion operation is performed on a piece of data in a data source, the field content corresponding to the data under the status field is updated to a deletion label, which is equivalent to logically deleting the data in the original data table instead of physically deleting it. This makes it easier to trace the source and locate problems when errors occur in subsequent data analysis and data applications.

[0072] If the operation tag of the first target data is an update tag, it indicates that an update operation has been performed on the first target data in the data source, and then the first target data can be overwritten with the data obtained after the first target data is updated under the data field, and the field content corresponding to the first target data under the status field is updated to the update tag "UPDATE" to indicate that the first target data has been updated in the data source. For example, still taking the original data table shown in Table 1 as an example, if an update operation is performed on b in the data source (update b to b'), then the operation tag of b is the update tag "UPDATE", then b under the data field can be updated to b', and the field content corresponding to b under the status field is updated to the update tag to indicate that b' is the updated data. Thus, the updated original data table obtained is shown in Table 3.

[0073] Table 3

[0074] data state a NULL b’ UPDATE …… ……

[0075] If the operation tag of the first target data is a new tag, it indicates that a new operation has been performed on the first target data in the data source, and the first target data and its new tag can be inserted into the original data table as a new record, wherein the first target data is the field content of the data field, and the new tag is the field content of the status field. For example, still taking the original data table shown in Table 1 as an example, if data c is newly added in the data source, then the operation tag of c is the new tag "INSERT", then c and its new tag "INSERT" can be inserted into the original data table as a new record. The updated original data table is shown in Table 4:

[0076] Table 4

[0077] data state a NULL b NULL c INSERT …… ……

[0078] It is worth mentioning that in actual applications, in order to facilitate more efficient and accurate tracing and problem location in subsequent data processing and application processes, the original data table can also include more fields, such as operation time and other fields. The operation time field is used to record the time when the data change operation is performed on each data in the data source.

[0079] In the embodiment of the present application, the data warehouse can use various appropriate methods to analyze and process the data in the original data table, and the embodiment of the present application is not limited to this.

[0080] Optionally, the data warehouse is specifically used to: determine the task type of the target analysis and processing task to be executed and the target operation label that matches the task type; obtain the template operation statement corresponding to the task type, the template operation statement is used to execute the analysis and processing task corresponding to the task type; based on the target operation label, generate a filter condition statement, the filter condition statement is used to filter out the data with the target operation label in the original data table; splice the template operation statement and the filter condition statement to obtain the target operation statement for processing the target analysis and processing task, and execute the target operation statement, when the target operation statement is executed, it executes the target analysis and processing task on the data in the original data table.

[0081] Among them, the template operation statement corresponding to a certain task type refers to the general operation statement used to execute tasks of this task type. The template operation statement can be written in any appropriate syntax. For example, taking SQL statement as an example, if the task type is a sum task, then the template operation statement can be: SELET SUM(Column_name)FROM table_name, which means to sum the field values ​​of a certain field in the data table, where table_name represents the table name of the data table, and Column_name represents the column name (i.e., field name) in the data table. For the filtering condition statement, if you want to filter out the data with the delete label "DELETE" in the original data table, then the filtering condition statement can be: where field<>'DELETE'.

[0082] Assuming that the target analysis task is to perform a sum operation on the field content of field 1 in the original data table, and the target operation label that matches the task is the delete label "DELETE", then the template operation statement "SELET SUM (field name of field 1) FROM table name of the original data table" and the filter condition statement "WHERE field name of field 1 <> 'DELETE" can be obtained; further, by splicing the two, the target operation statement "SELET SUM (field name of field 1) FROM table name of the original data table WHERE field name of field 1 <> 'DELETE" for executing the target analysis task can be obtained. Therefore, by executing the target operation statement, the data in field 1 of the original data table that has not been deleted from the data source can be summed.

[0083] It can be understood that in the above implementation, the data warehouse itself does not update the data that has been changed in the data source. Instead, when executing the target analysis and processing task, it adds corresponding filter condition statements to filter out the corresponding data based on the operation labels of each data in the original data table. In this way, the data warehouse is equivalent to making logical changes to the data in the original data table based on the data changes in the data source, rather than physical changes, which is conducive to tracing and locating problems when errors occur in subsequent data analysis and processing and data applications.

[0084] Optionally, in order to ensure that the data used in the subsequent data application process is accurate and reliable, the target analysis and processing task may include a data cleaning task, which is used to clean the data in the original data table to filter out the data that does not meet the requirements in the original data table. Specifically, the data that has been deleted in the data source is a typical non-compliant data, which seriously affects the accuracy and reliability of subsequent data analysis and application. In this case, the target operation tag corresponding to the data cleaning task may include a deletion tag, and the deletion tag of the data is used to indicate that the data change operation performed on the data in the data source is a deletion operation.

[0085] In the embodiment of the present application, the data warehouse can have any appropriate architecture, which can be set according to actual needs, and the embodiment of the present application does not limit this. Optionally, in order to make the blood relationship between the data in the data warehouse clearer, improve the efficiency of data analysis and processing, and reduce the complexity of data analysis and processing, the data warehouse can adopt a layered architecture.

[0086] Specifically, Figure 3 As shown, the data warehouse may include an on-source (ODS) layer and a detail data (DWD) layer. The data in the ODS layer is the source of the subsequent data warehouse processing data. Here, the ODS layer can be used to store the original data table; the DWD layer, as an isolation layer between the business and the original data table, is usually used to perform some data cleaning and normalization operations on the data in the original data table. In this case, the DWD layer can be used to perform the above-mentioned steps of determining the task type of the target analysis and processing task to be executed and the target operation label matching the task type to execute the target operation statement, and obtain the cleaned data table. The cleaned data table stores the cleaned and qualified data in the original data table, which can provide strong and reliable data support for subsequent data applications.

[0087] In actual applications, the DWD layer can retrieve data from the ODS layer on a daily basis by pulling snapshots, which realizes the offline data pulling in the data warehouse.

[0088] Optionally, the data warehouse can also include more processing layers to better adapt to business needs. Specifically, Figure 3 As shown, the data warehouse may include at least one of the following processing layers: a data service (DWS) layer, an application data (ADS) layer, and a common dimension (DIM) layer, etc. The DWS layer is used to summarize the data in the cleaned data table according to the template wide table corresponding to the preset subject, and obtain the subject wide table corresponding to the preset subject; the ADS layer is used to perform statistical analysis on the data in the subject wide table generated by the DWS layer according to the business statistical dimension related to the target business, and obtain the business dimension data table corresponding to the business statistical dimension; the DIM layer is used to perform statistical analysis on the data in the subject wide table generated by the DWS layer and the data in the business dimension data table generated by the ADS layer according to the preset common statistical dimension, and obtain the common dimension data table corresponding to the common statistical dimension.

[0089] In actual applications, the preset subject, business statistical dimension and general statistical dimension can be set according to actual needs, and the embodiment of the present application does not limit this. For example, the preset subject can be a loan product, the business statistical dimension can include the collection rate, etc., and the general statistical dimension can include the date, etc.

[0090] It is worth noting that the specific implementation methods of the data analysis and processing of the DWS layer, the ADS layer and the DIM layer can be achieved through various existing processing technologies and will not be elaborated herein.

[0091] It can be understood that in the above embodiment, since the DWS layer, ADS layer and DIM layer are all downstream of the DWD layer, the DWD layer has taken into account the data changes in the data source when performing data analysis and processing tasks, and combined with the operation labels of each data in the original data table, the data that does not meet the requirements in the original data table is filtered out. Therefore, when performing data analysis and processing, the DWS layer, ADS layer and DIM layer no longer need to consider the identification of each data, thereby simplifying the data analysis and processing process and improving the data analysis and processing efficiency.

[0092] In another embodiment of the present application, Figure 3 As shown, in order to better utilize various business processing requirements, the above data processing system may further include an application component. The application component is used to respond to the received business processing request, obtain the second target data from the data warehouse based on the data type of the second target data required by the target business requested to be processed by the business processing request, and process the target business based on the second target data. The data type includes offline data and real-time data.

[0093] In practical applications, such as Figure 3As shown, the target business may include, for example, but is not limited to, generating business intelligence (BI) reports, data analysis and mining, data query, and large-screen display.

[0094] In an optional implementation, the application component is specifically used to: if the data type of the second target data is real-time data, obtain the second target data from the original data table stored in the ODS layer, and process the target business based on the second target data and the operation tag of the second target data; if the data type of the second target data is offline data, obtain the second target data from the business dimension data table generated by the ADS layer and / or the general dimension data table generated by the DIM layer, and process the target business based on the second target data.

[0095] For example, if the target business is to analyze and mine transaction data within a specified historical time period, then the second target data is the offline transaction data generated within the specified historical time period. The application component can then obtain the offline transaction data generated within the specified historical time period from the general dimension data table obtained by daily statistics at the DIM layer, and then perform analysis and mining based on these offline transaction data.

[0096] For example, if the target business is to display the real-time transaction data in the data warehouse on a large screen, then the second target data is the real-time transaction data in the original data table. The application component can then obtain the real-time transaction data of the original data table from the ODS layer and send it to the large-screen display terminal for display.

[0097] It is understandable that due to the diversity of data application scenarios in the downstream, offline data can only solve part of the business, and some businesses require real-time data. The computing engine processes data operation information in real time and writes the data and its operation tags into the original data table of the ODS layer. The application component obtains real-time data from the ODS layer, and offline data from the ADS layer and / or DIM layer. This is equivalent to realizing offline and real-time data branches in the ODS layer, which can better meet the data application needs in diverse application scenarios.

[0098] In another embodiment of the present application, the data processing system provided in the embodiment of the present application may further include a storage component (not shown in the figure).

[0099] The storage component is used to obtain data with an operation tag of a delete tag from the original data table as the third target data, and to perform cold standby processing on the third target data, and / or to write the third target data to the disk. In practical applications, the cold standby processing of data can be implemented by various cold standby processing technologies commonly used in the art, which will not be described in detail.

[0100] It is understandable that, since the data with a deletion tag is data that has been deleted from the data source, this data may be core data, and the deletion operation on this data may be caused by accidental deletion or a program error in the data source. Once this data is deleted, it is a clear sign for the enterprise; for this reason, by cold-backup processing of the data with deletion tags in the original data table and / or storing the data with deletion tags in the original data table to disk, it can play a role in backing up these data, even if these data are deleted by mistake, they can continue to meet business needs. In addition, since the cost of disks is relatively low, using disks to store data with deletion tags can reduce data backup costs.

[0101] It is worth noting that, in practical applications, the data processing system provided in the embodiments of the present application may also adopt a layered architecture. Figure 3 As shown, the data source can be a part of the data processing system. In addition, the data processing system can include a data processing area, a data warehouse and application components, among which the listening component, the message middleware and the computing engine belong to the data processor, and the ODS layer, the DWD layer, the DWS layer, the ADS layer and the DIM layer belong to the data warehouse.

[0102] The above embodiments introduce the data processing system provided by the embodiments of the present application, and the data processing system can be applied to a variety of business scenarios. Here, the construction of the consumer finance data warehouse system is taken as an example. The core is the account, and the account will also be divided into multiple accounts. Similar to the credit card business, the business will include account tables, credit plan tables, repayment tables, installment plan tables, authentication tables, loan notes tables, etc. These are core data and need to be permanently preserved. The accuracy of the data and the timeliness of the data are of great importance to both accountants and data warehouse personnel. However, due to historical data iterations, changes in product usage scenarios, some errors in accounting procedures, and other factors, the data warehouse revision process is an inevitable process, and it is currently a regular job. The existing manual revision method has the problems of high labor costs, complicated operations, and low revision efficiency, and it is difficult to ensure that there are no problems with each revision, so there is a large risk of revision errors, which is a very stressful job for the relevant personnel.

[0103] The data processing system provided by the embodiment of the present application optimizes the ETL layer of the original consumer finance data warehouse system and improves it to the architecture of monitoring component + message middleware + computing engine. The monitoring component obtains and parses the data change log of the data source in real time, obtains data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, marks the corresponding operation tag for the data based on the data change operation information to indicate the data change operation performed on the first target data in the data source, and writes the data and its operation tag into the original data table of the data warehouse; since the process of sending data operation information from the monitoring component to the message middleware is real-time, the process of consuming data operation information from the message middleware and marking the data by the computing engine is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume the data operation information from the message middleware. After processing, it falls into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label; in this way, the data warehouse can timely and intuitively obtain what changes have occurred in the data synchronized from the data source in the data source based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in the data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. In addition, the change of data in the original data table is determined according to the data change operation on the data source, rather than the active change and determination of the data warehouse, so as to ensure the accuracy of data change situation and reduce the probability of errors in the automatic adjustment process; in addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream has done on the data, the downstream can track and record according to the data operation label, so that it can accurately check and locate the problem when data processing errors occur, and improve the maintainability of the entire data processing system.

[0104] based on Figure 2 and Figure 3 The data processing system disclosed in the embodiment shown in the figure, the embodiment of the present application also provides a data processing method, which can apply a computing engine, such as Figure 2 or Figure 3 The computing engine in the data processing system disclosed in the illustrated embodiment. Figure 4 , is a flow chart of a data processing method provided by an embodiment of the present application, the method comprising the following steps:

[0105] S402: Obtain data operation information for the data source from the message middleware.

[0106] The data operation information includes the first target data and the data change operation information corresponding to the first target data. The first target data includes the data that has changed in the data source, and the data change operation information corresponding to the first target data is used to describe the data change operation performed on the first target data in the data source, such as the type of data change operation, the data before and after the data change operation is performed, etc. The data operation information is obtained by the monitoring component after real-time acquisition and parsing of the data change log of the data source and sent to the message middleware.

[0107] S404: Based on the data change operation information, mark the first target data with a corresponding operation tag.

[0108] The operation tag of the first target data is used to indicate the data change operation performed on the first target data in the data source.

[0109] S406, writing the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0110] It should be noted that the specific implementation method of each step in the data processing method provided in the embodiment of the present application can refer to the specific implementation method of the corresponding function in the computing engine in the above-mentioned data processing system. Since the principle is the same, it will not be repeated here.

[0111] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] In addition, Figure 4 Corresponding to the data processing method shown, the embodiment of the present application also provides a data processing device, which can be applied to a computing engine, such as Figure 2 or Figure 3 The computing engine in the data processing system disclosed in the illustrated embodiment. Figure 5 , is a schematic diagram of a data processing device 500 provided in one embodiment of the present application, the device 500 includes:

[0113] The acquisition unit 510 is used to acquire data operation information for a data source from the message middleware, wherein the data operation information includes first target data and data change operation information corresponding to the first target data, wherein the first target data includes data changed in the data source, and the data operation information is acquired by the monitoring component in real time and the data change log of the data source is parsed and then sent to the message middleware;

[0114] A marking unit 520, configured to mark a corresponding operation label for the first target data based on the data change operation information, wherein the operation label is used to indicate the data change operation performed on the first target data in the data source;

[0115] The access unit 530 is used to write the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0116] Obviously, the data processing device provided in the embodiment of the present application can be used as the above Figure 4 The execution subject of the data processing method shown in FIG. Figure 4 The functions realized are as follows. Since the principles are the same, they will not be described here.

[0117] Figure 6 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present application, on which a computing engine is running. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0118] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0119] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0120] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0121] Acquire data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information is acquired by a monitoring component in real time and parsed after the data change log of the data source is obtained and sent to the message middleware;

[0122] Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source;

[0123] The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0124] The above application Figure 4The method performed by the data processing device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0125] The electronic device may also perform Figure 4 The method and the data processing device are implemented in Figure 4 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0126] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0127] Acquire data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information is acquired by a monitoring component in real time and parsed after the data change log of the data source is obtained and sent to the message middleware;

[0128] Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source;

[0129] The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

[0130] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0131] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0132] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0133] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A data processing system, characterized in that: include: Listening components, message middleware, computing engine, and data warehouse; The monitoring component is used to obtain and parse the data change log of the data source in real time, obtain data operation information for the data source, and send the data operation information to the message middleware, wherein the data operation information includes first target data and data change operation information corresponding to the first target data, and the first target data includes data that has changed in the data source; The computing engine is used to obtain the data operation information from the message middleware, and is also used to mark a corresponding operation tag for the first target data based on the data change operation information, and write the first target data and the operation tag of the first target data into an original data table of the data warehouse, wherein the original data table stores data synchronized from the data source, and the operation tag is used to indicate a data change operation performed on the first target data in the data source; The data warehouse is used to analyze and process the data in the original data table based on the operation tags of the data in the original data table.

2. The data processing system according to claim 1, characterized in that The data warehouse is specifically used for: Determine the task type of the target analysis and processing task to be executed and the target operation tag matching the task type; Acquire a template operation statement corresponding to the task type, wherein the template operation statement is used to execute the analysis and processing task corresponding to the task type; Based on the target operation tag, generating a filtering condition statement, wherein the filtering condition statement is used to filter out the data in the original data table having the target operation tag; The template operation statement and the filter condition statement are concatenated to obtain a target operation statement for processing the target analysis and processing task, and the target operation statement is executed. When the target operation statement is executed, the target analysis and processing task is performed on the data in the original data table.

3. The data processing system according to claim 2, characterized in that: The target analysis and processing task includes a data cleaning task, which is used to clean the data in the original data table. The target operation tag corresponding to the data cleaning task includes a delete tag, which is used to indicate that the data change operation performed on the corresponding data in the data source is a delete operation. The data warehouse includes a source layer and a detail data layer, the source layer is used to store the original data table, and the detail data layer is used to execute the steps of determining the task type of the target analysis and processing task to be executed and the target operation tag matching the task type to executing the target operation statement to obtain a cleaned data table.

4. The data processing system according to claim 3, characterized in that: The data warehouse also includes at least one of the following processing layers: The data service layer is used to summarize the data in the cleaned data table according to the template wide table corresponding to the preset theme, so as to obtain the theme wide table corresponding to the preset theme; The application data layer is used to perform statistical analysis on the data in the subject wide table according to the business statistical dimension related to the target business, and obtain the business dimension data table corresponding to the business statistical dimension; The common dimension layer is used to perform statistical analysis on the data in the subject wide table and the data in the business dimension data table according to a preset common statistical dimension, and obtain a common dimension data table corresponding to the common statistical dimension.

5. The data processing system according to claim 4, characterized in that: The data processing system further comprises: The application component is used to respond to the received business processing request, obtain the second target data from the data warehouse based on the data type of the second target data required by the target business requested to be processed by the business processing request, and process the target business based on the second target data.

6. The data processing system according to claim 5, characterized in that: The application component is specifically used for: If the data type of the second target data is real-time data, acquiring the second target data from the original data table stored in the source layer, and processing the target business based on the second target data and the operation tag of the second target data; If the data type of the second target data is offline data, the second target data is obtained from the business dimension data table generated by the application data layer and / or the common dimension data table generated by the common dimension layer, and the target business is processed based on the second target data.

7. The data processing system according to claim 1, characterized in that: The original data table includes a data field and a status field, the data field is used to record the data synchronized from the data source, and the status field is used to record the status of the data synchronized from the data source in the data source; In the process of writing the first target data and the operation tag of the first target data into the original data table of the data warehouse, the computing engine is specifically used to: update the field content of the data field based on the first target data, and update the field content of the status field based on the operation tag of the first target data.

8. The data processing system according to any one of claims 1 to 7, characterized in that: The data processing system further comprises: A storage component is used to obtain data with an operation tag of a delete tag from the original data table as third target data, and to perform cold standby processing on the third target data, and / or to write the third target data to a disk, wherein the delete tag is used to indicate in the data source that the data change operation performed on the corresponding data is a delete operation.

9. A data processing method, characterized in that: The method is applied to a computing engine, and comprises: Acquire data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information is acquired by a monitoring component in real time and parsed after the data change log of the data source is obtained and sent to the message middleware; Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source; The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

10. A data processing device, characterized in that: The device is applied to a computing engine, and comprises: an acquisition unit, configured to acquire data operation information for a data source from a message middleware, wherein the data operation information includes first target data and data change operation information corresponding to the first target data, wherein the first target data includes data changed in the data source, and the data operation information is acquired by a monitoring component in real time and after parsing a data change log of the data source, and then sent to the message middleware; a marking unit, configured to mark a corresponding operation tag for the first target data based on the data change operation information, wherein the operation tag is used to indicate the data change operation performed on the first target data in the data source; The access unit is used to write the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.

11. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a computing engine, the computing engine is enabled to execute the data processing method as claimed in claim 9.

Citation Information

Patent Citations

  • Service message sending method and device

    CN105528378A

  • Big data synchronization method and device based on Binlog, HBase and Hive

    CN112286941A

  • Data processing system, method, equipment and medium

    CN115168145A

Cited By

  • Stream batch collaborative super-large-scale user risk real-time assessment system and method, electronic equipment and computer program product

    CN121788158A