Data processing method, electronic device, storage medium, and apparatus
Through the combination of listening components and message middleware, automatic correction of data warehouses is achieved, solving the problems of high cost and low efficiency of manual repair in the prior art, ensuring data accuracy and system maintainability.
Patent Information
- Application Number
- PCT/CN2024/091725
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-14
AI Technical Summary
The existing data warehouse storage usage scheme cannot achieve automatic data correction, resulting in high cost, low efficiency, and error-proneness in manual verification and repair.
The monitoring component is used to obtain the data change log of the data source in real time, parse and send it to the message middleware, and the calculation engine obtains data operation information from the middleware, marks operation labels and writes them to the original data table, realizing automatic data correction and analysis processing of data.
It improves data correction efficiency, ensures the accuracy of data changes, reduces the probability of errors, and improves the maintainability of the data processing system.
Smart Images

Figure CN2024091725_14112024_PF_FP_ABST
Abstract
Description
Data processing method, electronic device, storage medium and device
[0001] Cross-references
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 9, 2023, with application number 202310520088.9, and invention name “Data processing system, method, device and storage medium”, all contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of data processing technology, and in particular to a data processing method, electronic equipment, storage medium and device. Background Art
[0004] The construction of a data warehouse for big data is a very important link in the field of big data. Maintaining the quality of data in the data warehouse is even more important, otherwise low-quality data will lose its value in the data warehouse.
[0005] Because data warehouses are derived from data sources such as business systems, data errors and duplications are common due to programming issues within the data sources. Currently, common data warehouse storage solutions utilize a Hive + Hadoop architecture, creating a big data query solution that can be used with Structured Query Language (SQL). This data warehouse storage solution cannot automatically correct data on Hive. Consequently, personnel must manually verify the accuracy of data in the data warehouse and perform data repairs.
[0006] Summary of the Invention
[0007] The present application provides a data processing method, comprising: obtaining data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, the data operation information being obtained by obtaining a data change log of the data source and parsing the data change log; marking a corresponding operation tag for the first target data based on the data change operation information, the operation tag being used to indicate a data change operation performed on the first target data in the data source; and writing the first target data and the operation tag of the first target data into an original data table to analyze and process the data in the original data table.
[0008] The present application also provides an electronic device, comprising: a memory; and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the following steps: obtaining data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, the data operation information being obtained by obtaining a data change log of the data source and parsing the data change log; marking a corresponding operation tag for the first target data based on the data change operation information, the operation tag being used to indicate a data change operation performed on the first target data in the data source; and writing the first target data and the operation tag of the first target data into an original data table to analyze and process the data in the original data table.
[0009] The present application also provides a computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the following steps: obtaining data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, the data operation information being obtained by obtaining a data change log of the data source and parsing the data change log; marking a corresponding operation tag for the first target data based on the data change operation information, the operation tag being used to indicate a data change operation performed on the first target data in the data source; and writing the first target data and the operation tag of the first target data into an original data table to analyze and process the data in the original data table.
[0010] The present application also provides a data processing device, including: an acquisition unit, used to acquire data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, and the data operation information is obtained by acquiring a data change log of the data source and parsing the data change log; a marking unit, used to mark a corresponding operation label for the first target data based on the data change operation information, the operation label being used to indicate a data change operation performed on the first target data in the data source; an access unit, used to write the first target data and the operation label of the first target data into an original data table to analyze and process the data in the original data table. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0012] FIG1 is a schematic diagram of a data modification solution under the current data warehouse storage and usage architecture;
[0013] FIG2 is a schematic diagram of the structure of a data processing system provided by one embodiment of the present application;
[0014] FIG3 is a schematic structural diagram of a data processing system provided by another embodiment of the present application;
[0015] FIG4 is a flow chart of a data processing method provided by an embodiment of the present application;
[0016] FIG5 is a schematic structural diagram of a data processing device provided by an embodiment of the present application;
[0017] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0018] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0019] In this specification and claims, terms such as "first," "second," and the like are used to distinguish similar objects and are not intended to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein. In addition, in this specification and claims, "and / or" refers to at least one of the connected objects, and the character " / " generally indicates that the connected objects are in an "or" relationship.
[0020] Some concept descriptions:
[0021] Apache Spark: Spark is a fast and general computing engine designed for large-scale data processing. Spark is a general engine that can be used to perform a variety of operations, including SQL queries, text processing, machine learning, etc.
[0022] Apache Flink: Flink for short, executes arbitrary streaming programs in a data-parallel, pipelined manner. When Flink's pipeline runs, the system can execute both batch and stream processing programs.
[0023] Data warehouse: The purpose of a data warehouse is to build an integrated, analytical data environment to provide decision support for enterprises. Created for analytical reporting and decision support, the data warehouse itself does not "produce" or "consume" any data. Data originates externally and is open to external applications.
[0024] Apache Hive: Hive is a data warehouse tool based on Hadoop that is used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large amounts of data stored in Hadoop. The Hive data warehouse tool maps structured data files into database tables and provides SQL query capabilities, converting SQL statements into MapReduce tasks for execution.
[0025] Apache Hoodie: Apache Hudi brings core warehouse and database functionality directly to the data lake. Hudi provides tables, transactions, efficient append / delete, advanced indexing, streaming ingestion services, data clustering / compression optimizations, and concurrency, all while keeping the data in an open source file format.
[0026] Apache Hudi can be easily used on any cloud storage platform. Hudi's advanced performance optimizations make analytical workloads faster on any popular query engine, including Apache Spark, Flink, Presto, Trino, Hive, and more.
[0027] Kafka: Kafka is a high-throughput distributed publish-subscribe messaging system that handles streaming data from all consumer actions on a website. This action (web browsing, searching, and other user actions) is a key factor in many social functions on the modern web. This data is often processed through log processing and log aggregation due to throughput requirements.
[0028] MaxWell: Maxwell is MySQL change data capture software written in Java. It monitors MySQL database data change operations (including insert, update, and delete) in real time and sends this change data in JSON format to stream data processing platforms such as Kafka and Kinesi. Maxwell works by reading the MySQL database binary log in real time, extracting change data from it, and then sending this change data in JSON format to stream processing platforms such as Kafka.
[0029] ODS: Operation Data Store (ODS), also known as the source layer. Data tables in the source system of the data warehouse are usually stored intact. This layer, called the ODS, is the source of data for subsequent data warehouse processing.
[0030] DWD: Data warehouse details layer, which isolates the business layer from the data warehouse. It primarily performs data cleansing and normalization operations on the ODS data layer.
[0031] DWS: Data warehouse service (DWS) is a data service layer that aggregates and summarizes basic data from the DWB to analyze a specific subject domain. It is typically a wide table and is used for subsequent business queries, OLAP analysis, and data distribution.
[0032] ETL: Extract-Transform-Load, data loading layer.
[0033] DIM: Dimension, public dimension layer.
[0034] ADS: Application Data Store, application data layer. Personalized dimension aggregation layer. Less commonly used statistical dimension data is stored in this layer. Dimensions and indicators that are specific to your business are calculated here.
[0035] JSON: A lightweight data exchange format based on a subset of ECMAScript (a JS specification developed by the European Computer Society) that uses a text format that is completely independent of programming languages to store and represent data.
[0036] Apache Doris: abbreviated as Doris, is a distributed database based on the Massively Parallel Processing (MMP) architecture, which has the advantages of low overhead and high performance.
[0037] Apache Sqoop: Sqoop for short, is a data extraction tool that can synchronize data from relational databases to HDFS (Hadoop's underlying storage system), and can also export HDFS data and store it back into relational databases.
[0038] As described in the background section, current common data warehouse storage solutions utilize a Hive + Hadoop architecture, creating a large-scale query solution that can be used to convert structured query statements into SQL. This data warehouse storage solution cannot automatically correct data in Hive. Consequently, personnel must manually verify the accuracy of data in the data warehouse and perform data repairs.
[0039] For example, as shown in Figure 1, in the current data warehouse storage architecture, data is regularly retrieved from the data source through the ETL layer. After extraction, transformation, and loading, the data is stored in the data warehouse. To ensure data synchronization between the data warehouse and the data source, and to ensure the reliability and accuracy of subsequent data applications, manual verification of data in the data warehouse and manual correction of data are often required using Hive.
[0040] However, this manual method of number adjustment has the problems of high labor cost, complicated operation, and low number adjustment efficiency. It is also difficult to ensure that there will be no problems every time the number is adjusted, so there is a greater risk of number adjustment errors.
[0041] In view of this, an embodiment of the present application aims to provide a data processing system, which optimizes the ETL layer under the original data warehouse storage and usage architecture, and improves it to an architecture of monitoring component + message middleware + computing engine, wherein the monitoring component obtains and parses the data change log of the data source in real time, obtains data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation label for the data based on the data change operation information to indicate the data change operation performed on the data in the data source, and writes the data and its operation label into the original data table of the data warehouse; since the process of the monitoring component sending data operation information to the message middleware is real-time, the process of the computing engine consuming data operation information from the message middleware and marking the data is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume the data operation information from the message middleware and process it into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation Tags; In this way, the data warehouse can obtain the changes that have occurred in the data synchronized from the data source in a timely and intuitive manner based on the original data table, and then perform corresponding data analysis and processing based on the data changes. There is no need for manual intervention in data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. Moreover, the changes in the data in the original data table are determined according to the data change operations on the data source, rather than the active changes and determinations of the data warehouse. Therefore, the accuracy of the data changes can be ensured and the probability of errors in the automatic adjustment process can be reduced. In addition, since the data in the data warehouse has its own operation tags, no matter what operations the upstream performs on the data, the downstream can track and record them according to the data operation tags, so that when data processing errors occur, the problem can be accurately checked and located, thereby improving the maintainability of the entire data processing system.
[0042] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.
[0043] Please refer to Figure 2, which is a structural diagram of a data processing system provided in an embodiment of the present application. The data processing system includes: a monitoring component, a message middleware, a computing engine, and a data warehouse.
[0044] Among them, message middleware is a middleware used for publishing and subscribing to messages. It can receive messages published by producers and provide consumers with subscribed messages.
[0045] The monitoring component is a component that can monitor the data changes of the data source in real time. In an embodiment of the present application, the monitoring component can obtain the data change log of the data source in real time and parse the data change log to obtain data operation information for the data source, wherein the data operation information includes the first target data and the data change operation information corresponding to the first target data, the first target data includes the data that has changed in the data source, and the data change operation information corresponding to the first target data is used to describe the data change operation performed on the first target data in the data source. In addition, the monitoring component can also act as a producer to send the data operation information parsed from the data change log obtained in real time to the message middleware.
[0046] The computing engine is an engine for data computing, especially suitable for big data computing, and has parallel batch processing and stream processing capabilities. In an embodiment of the present application, the computing engine can act as a consumer and subscribe to data operation information for a data source from the message middleware. After obtaining the data operation information from the message middleware, the computing engine can mark the first target data with a corresponding operation tag based on the data change operation information contained therein. The operation tag of the first target data is used to represent the data change operation performed on the first target data in the data source. Furthermore, the computing engine also writes the first target data and the operation tag of the first target data into the original data table of the data warehouse, wherein the original data table records the data synchronized from the data source by the data warehouse.
[0047] For example, a data change operation includes at least one of a delete operation, an update operation, and a new operation. If the data change operation is a delete operation, the first target data deleted by the data change operation can be marked with a delete tag (such as DELETE); if the data change operation is an new operation, the first target data added by the data change operation can be marked with an add tag (such as INSERT); if the data change operation is an update operation, the first target data updated by the data change operation can be marked with an update tag (such as UPDATE). In this way, the data operation tag can intuitively reflect what data change operation was performed on the data in the data source.
[0048] The data warehouse is used to analyze and process the data in the original data table based on the operation tags of the data in the original data table.
[0049] It can be seen that the data processing system provided by the embodiment of the present application optimizes the ETL layer under the original data warehouse storage and usage architecture, and improves it to the architecture of monitoring component + message middleware + computing engine, in which the monitoring component obtains the data change log of the data source in real time and parses the data change log to obtain data operation information for the data source and sends it to the message middleware; since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation label for the data based on the data change operation information to indicate the data change operation performed on the data in the data source, and writes the data and its operation label into the original data table of the data warehouse; since the process of the monitoring component sending data operation information to the message middleware is real-time, the process of the computing engine consuming data operation information from the message middleware and marking the data is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume data from the message middleware. After data operation information is processed, it falls into the original data table in the data warehouse. The original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label. In this way, the data warehouse can obtain the changes that have occurred in the data synchronized from the data source in a timely and intuitive manner based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the efficiency of adjustment. Moreover, the changes in the data in the original data table are determined based on the data change operations on the data source, rather than actively changed and determined by the data warehouse. Therefore, the accuracy of the data change situation can be ensured and the probability of errors in the automatic adjustment process can be reduced. In addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream performs on the data, the downstream can track and record them based on the data operation label, so that when data processing errors occur, the problem can be accurately checked and located, thereby improving the maintainability of the entire data processing system.
[0050] In order for those skilled in the art to better understand the data processing system provided in the embodiments of the present application, each component of the data processing system is described in detail below.
[0051] The data processing system of the embodiment of the present application can access a variety of data sources, such as the business system shown in Figure 3. A relational database is deployed in the business system, and the data processing system can access the relational database, and the monitoring component can obtain data operation information for the structured data in the relational database in real time. Of course, a non-relational database can also be deployed in the business system, and the data processing system can also access the non-relational database, and the monitoring component can obtain data operation information for the unstructured data in the non-relational database in real time.
[0052] In the embodiment of the present application, the message middleware can adopt any appropriate publish-subscribe message system with high-throughput data transmission capabilities, such as Kafka, etc. The specific selection can be made according to actual needs, and the embodiment of the present application does not limit this.
[0053] In the embodiment of the present application, the monitoring component can adopt any appropriate component with data monitoring function. Optionally, the monitoring component can include a data capture component such as MaxWell. MaxWell can obtain the binary log (BINLOG) of the data source in real time, that is, the data change log. The binary log records the data operation information for the data source. MaxWell can obtain the data operation information by parsing the binary log and send the data operation information to the message middleware in JSON format.
[0054] In the embodiment of the present application, the computing engine can adopt any appropriate engine with parallel batch processing and streaming processing capabilities, such as Flink, Spark, etc., which can be selected according to actual needs, and the embodiment of the present application does not limit this. In a more preferred solution, since Spark's parallel batch processing and streaming data processing capabilities are more mature and more suitable for actual business development needs, the computing engine can adopt Spark. In addition, another advantage of using Spark as a computing engine is that the original data table in the data warehouse is usually a Hoodie table. Since Hoodie has a corresponding API to connect with Spark, Spark only needs to import the Jar package corresponding to Hooide to use it in the program, and the first target data and its operation label can be dropped into the Hoodie table, which is simple to implement. In the embodiment of the present application, the computing engine can write the first target data and its operation label into the original data table in various appropriate ways, which can be selected according to actual needs, and the embodiment of the present application does not limit this.
[0055] Optionally, in order to enable the data warehouse to more intuitively and accurately understand what changes have occurred to the data in the original data table in the data source, and thus make more accurate analysis and processing decisions, the original data table may include data fields and status fields, wherein the data field is used to record the data synchronized from the data source by the data warehouse, and the status field is used to record the status of the data synchronized from the data source in the data source, such as whether the data is new data in the data source, whether the data has been deleted from the data source, and whether the data has been updated in the data source.
[0056] Correspondingly, in the process of writing the first target data and the operation tag of the first target data into the original data table of the data warehouse, the computing engine is specifically used to: update the field content of the data field based on the first target data, and update the field content of the status field based on the operation tag of the first target data.
[0057] Specifically, if the operation tag of the first target data is a delete tag, this indicates that a delete operation has been performed on the first target data in the data source. Consequently, the content of the field corresponding to the first target data under the status field can be updated to a delete tag to indicate that the first target data has been deleted from the data source. For example, Table 1 shows an example of a raw data table, which includes a data field "data" and a status field "status." If a delete operation is performed on a in the data source, and the operation tag of a is the delete tag "DELETE," the content of the field corresponding to a under the status field can be updated to a delete tag to indicate that a has been deleted from the data source. The updated raw data table is shown in Table 2.
[0058] Table 1
[0059] Table 2
[0060] It's understandable that when a data source is deleted, the corresponding field under the Status field is updated with a Delete tag, effectively performing a logical deletion of the data in the original data table, rather than a physical deletion. This facilitates tracing and problem location when errors occur during subsequent data analysis, processing, and application.
[0061] If the operation tag of the first target data is an update tag, it indicates that an update operation has been performed on the first target data in the data source, and then the first target data can be overwritten with the data obtained after the first target data is updated under the data field, and the field content corresponding to the first target data under the status field is updated to the update tag "UPDATE" to indicate that the first target data has been updated in the data source. For example, still taking the original data table shown in Table 1 as an example, if an update operation is performed on b in the data source (update b to b'), then the operation tag of b is the update tag "UPDATE", then b under the data field can be updated to b', and the field content corresponding to b under the status field is updated to the update tag to indicate that b' is the updated data. Thus, the updated original data table obtained is shown in Table 3.
[0062] Table 3
[0063] If the operation tag of the first target data is a new tag, it indicates that a new operation has been performed on the first target data in the data source, and the first target data and its new tag can be inserted into the original data table as a new record, wherein the first target data serves as the field content of the data field, and the new tag serves as the field content of the status field. For example, still taking the original data table shown in Table 1 as an example, if data c is newly added to the data source, then the operation tag of c is the new tag "INSERT", then c and its new tag "INSERT" can be inserted into the original data table as a new record. The updated original data table is shown in Table 4:
[0064] Table 4
[0065] It is worth noting that in actual applications, in order to facilitate more efficient and accurate tracing and problem location in subsequent data processing and application processes, the original data table can also include more fields, such as operation time fields, etc. The operation time field is used to record the time when data change operations are performed on each data in the data source.
[0066] In the embodiment of the present application, the data warehouse can use various appropriate methods to analyze and process the data in the original data table, and the embodiment of the present application is not limited to this.
[0067] Optionally, the data warehouse is specifically used to: determine the task type of the target analysis and processing task to be executed and the target operation label that matches the task type; obtain the template operation statement corresponding to the task type, the template operation statement is used to execute the analysis and processing task corresponding to the task type; based on the target operation label, generate a filter condition statement, the filter condition statement is used to filter out the data with the target operation label in the original data table; splice the template operation statement and the filter condition statement to obtain the target operation statement for processing the target analysis and processing task, and execute the target analysis and processing task on the data in the original data table by executing the target operation statement.
[0068] Among them, the template operation statement corresponding to a certain task type refers to a general operation statement used to execute tasks of this task type. Template operation statements can be written in any appropriate syntax. For example, taking SQL statements as an example, if the task type is a sum task, then the template operation statement can be: SELET SUM(Column_name)FROM table_name. This template operation statement means summing the field values of a certain field in the data table, where table_name represents the table name of the data table, and Column_name represents the column name (i.e., field name) in the data table. For the filter condition statement, if you want to filter out the data with the delete label "DELETE" in the original data table, then the filter condition statement can be: where field <> 'DELETE'.
[0069] Assume that the target analysis task is to sum the contents of Field 1 in the original data table, and the target operation tag matching this task is the delete tag "DELETE." Then, we can obtain the template operation statement "SELET SUM (Field 1's field name) FROM Original Data Table's table name" and the filter condition statement "WHERE Field 1's field name <> 'DELETE'." Furthermore, by concatenating the two, we obtain the target operation statement "SELET SUM (Field 1's field name) FROM Original Data Table's table name WHERE Field 1's field name <> 'DELETE'" for executing the target analysis task. Therefore, by executing this target operation statement, we can sum the data in Field 1 of the original data table that has not been deleted from the data source.
[0070] It can be understood that in the above implementation, the data warehouse itself does not update the data that has changed in the data source. Instead, when executing the target analysis and processing task, it adds corresponding filter condition statements to filter out the corresponding data based on the operation labels of each data in the original data table. In this way, it is equivalent to the data warehouse making logical changes to the data in the original data table based on the data changes in the data source, rather than physical changes, which is conducive to tracing the source and locating problems when errors occur in subsequent data analysis and processing and data applications.
[0071] Optionally, to ensure the accuracy and reliability of the data used in subsequent data applications, the target analysis and processing task may include a data cleaning task. This task cleans the data in the original data table to filter out any data that does not meet the requirements. Specifically, data that has been deleted from the data source is a typical example of non-compliant data, which seriously affects the accuracy and reliability of subsequent data analysis, processing, and applications. In this case, the target operation tag corresponding to the data cleaning task may include a delete tag, which indicates that the data change operation performed on the data in the data source is a delete operation.
[0072] In the embodiments of the present application, the data warehouse can have any appropriate architecture, which can be set according to actual needs and is not limited in the embodiments of the present application. Optionally, in order to make the relationship between the data in the data warehouse clearer, improve the efficiency of data analysis and processing, and reduce the complexity of data analysis and processing, the data warehouse can adopt a layered architecture.
[0073] Specifically, as shown in Figure 3, the data warehouse can include an on-source (ODS) layer and a detail data (DWD) layer. Among them, the data in the ODS layer is the source of the data processed by the subsequent data warehouse. Here, the ODS layer can be used to store the original data table; the DWD layer serves as an isolation layer between the business and the original data table, and is usually used to perform some data cleaning and normalization operations on the data in the original data table. In this case, the DWD layer can be used to perform the above-mentioned steps of determining the task type of the target analysis and processing task to be executed and the target operation label that matches the task type to executing the target operation statement, and obtain a cleaned data table. Among them, the cleaned data table stores the cleaned data in the original data table that meets the requirements, which can provide strong and reliable data support for subsequent data applications.
[0074] In actual applications, the DWD layer can retrieve data from the ODS layer on a daily basis by pulling snapshots, which realizes offline data pulling in the data warehouse.
[0075] Optionally, the data warehouse can also include more processing layers to better adapt to business needs. Specifically, as shown in Figure 3, the data warehouse can include at least one of the following processing layers: a data service (DWS) layer, an application data (ADS) layer, and a common dimension (DIM) layer. Among them, the DWS layer is used to summarize the data in the cleaned data table according to the template wide table corresponding to the preset subject, and obtain the subject wide table corresponding to the preset subject; the ADS layer is used to perform statistical analysis on the data in the subject wide table generated by the DWS layer according to the business statistical dimension related to the target business, and obtain the business dimension data table corresponding to the business statistical dimension; the DIM layer is used to perform statistical analysis on the data in the subject wide table generated by the DWS layer and the data in the business dimension data table generated by the ADS layer according to the preset common statistical dimension, and obtain the common dimension data table corresponding to the common statistical dimension.
[0076] In actual applications, the preset topics, business statistical dimensions, and general statistical dimensions can all be set according to actual needs, and the embodiments of this application are not limited to this. For example, the preset topic can be a loan product, the business statistical dimensions can include the repayment rate, and the general statistical dimensions can include the date.
[0077] It is worth noting that the specific implementation methods of the data analysis and processing of the DWS layer, the ADS layer and the DIM layer can be achieved through various existing processing technologies and will not be described in detail.
[0078] It can be understood that in the above embodiment, since the DWS layer, ADS layer and DIM layer are all downstream of the DWD layer, the DWD layer has taken into account the data changes in the data source when performing data analysis and processing tasks, and combined with the operation labels of each data in the original data table, the data that does not meet the requirements in the original data table is filtered out. Therefore, when performing data analysis and processing, the DWS layer, ADS layer and DIM layer no longer need to consider the identification of each data, thereby simplifying the data analysis and processing process and improving data analysis and processing efficiency.
[0079] In another embodiment of the present application, as shown in FIG3 , to better meet various business processing requirements, the data processing system may further include an application component. The application component is configured to respond to a received business processing request, obtain second target data from a data warehouse based on the data type of the second target data required for the target business to be processed in the business processing request, and process the target business based on the second target data. The data types include offline data and real-time data.
[0080] In actual applications, as shown in FIG3 , target services may include, for example, but are not limited to, generating business intelligence (BI) reports, data analysis and mining, data query, and large-screen display.
[0081] In an optional implementation, the application component is specifically used to: if the data type of the second target data is real-time data, obtain the second target data from the original data table stored in the ODS layer, and process the target business based on the second target data and the operation tag of the second target data; if the data type of the second target data is offline data, obtain the second target data from the business dimension data table generated by the ADS layer and / or the general dimension data table generated by the DIM layer, and process the target business based on the second target data.
[0082] For example, if the target business is to analyze and mine transaction data within a specified historical time period, then the second target data is the offline transaction data generated within the specified historical time period. The application component can then obtain the offline transaction data generated within the specified historical time period from the general dimension data table obtained by daily statistics at the DIM layer, and then perform analysis and mining based on these offline transaction data.
[0083] For example, if the target business is to display the real-time transaction data in the data warehouse on a large screen, then the second target data is the real-time transaction data in the original data table. The application component can then obtain the real-time transaction data of the original data table from the ODS layer and send it to the large-screen display terminal for display.
[0084] It is understandable that due to the diversity of data application scenarios in the downstream, offline data can only solve part of the business, and some businesses require real-time data. The computing engine processes data operation information in real time and writes the data and its operation tags into the original data table of the ODS layer. The application components obtain real-time data from the ODS layer and offline data from the ADS layer and / or DIM layer. This is equivalent to realizing offline and real-time data branches in the ODS layer, which can better meet the data application needs in diverse application scenarios.
[0085] In another embodiment of the present application, the data processing system may further include a storage component (not shown in the figure).
[0086] The storage component is configured to retrieve data with a delete operation tag from the original data table, use the data with the delete operation tag as third target data, perform cold standby processing on the third target data, and / or write the third target data to a disk. In practical applications, cold standby processing of data can be implemented using various cold standby processing technologies commonly used in the art, which will not be described in detail here.
[0087] It is understandable that since data with deletion tags is data that has been deleted from the data source, this data may be core data, and the deletion operation on this data may be caused by accidental deletion or a program error in the data source. Once this data is deleted, it is fatal to the enterprise. For this reason, by performing cold backup processing on the data with deletion tags in the original data table and / or storing the data with deletion tags in the original data table to disk, this data can be backed up. Even if this data is accidentally deleted, it can still continue to meet business needs. In addition, due to the low cost of disks, using disks to store data with deletion tags can reduce data backup costs.
[0088] It is worth noting that, in practical applications, the data processing system provided in the embodiments of the present application may also adopt a layered architecture. For example, as shown in Figure 3, the data source may be part of the data processing system. In addition, the data processing system may include a data processing area, a data warehouse, and application components. Among them, the monitoring component, the message middleware, and the computing engine belong to the data processor, and the ODS layer, the DWD layer, the DWS layer, the ADS layer, and the DIM layer belong to the data warehouse.
[0089] The above embodiments introduce the data processing system provided by the embodiments of the present application, and the data processing system can be applied to a variety of business scenarios. Here, we take the construction of the consumer finance data warehouse system as an example. The core is the accounts, and the accounts can be divided into multiple accounts. Similar to the credit card business, the business will include account tables, credit plan tables, repayment tables, installment plan tables, authentication tables, loan note tables, etc. These are core data and need to be permanently preserved. The accuracy and timeliness of the data are very important to both accounting personnel and data warehouse personnel. However, due to historical data iterations, changes in product usage scenarios, some errors in accounting procedures and other factors, the data warehouse revision process is an inevitable process, and it is currently a routine job. The existing manual revision method has the problems of high labor costs, complicated operations, and low revision efficiency, and it is difficult to ensure that there are no problems in each revision. Therefore, there is a large risk of revision errors, which is a very stressful job for the relevant personnel.
[0090] The data processing system provided by the embodiment of the present application optimizes the ETL layer of the original consumer finance data warehouse system and improves it to the architecture of monitoring component + message middleware + computing engine. The monitoring component obtains the data change log of the data source in real time and parses the data change log to obtain data operation information for the data source and sends it to the message middleware. Since the data operation information includes the data that has changed in the data source and the data change operation information corresponding to the data, the computing engine obtains the data operation information from the message middleware, and marks the corresponding operation label for the data based on the data change operation information to indicate the data change operation performed on the data in the data source, and writes the data and its operation label into the original data table of the data warehouse. Since the process of the monitoring component sending data operation information to the message middleware is real-time, the process of the computing engine consuming data operation information from the message middleware and marking the data is also real-time, that is, as long as the monitoring component sends data operation information to the message middleware, the computing engine will consume data from the message middleware. After the operation information is obtained and processed, it falls into the original data table in the data warehouse, and then the original data table is equivalent to a quasi-real-time table, and the data in the original data table has its own operation label; in this way, the data warehouse can obtain the changes that have occurred in the data synchronized from the data source in a timely and intuitive manner based on the original data table, and then perform corresponding data analysis and processing based on the data change situation. There is no need for manual intervention in the data adjustment, but automatic adjustment is achieved in a "hands-off" manner, which can improve the adjustment efficiency. Moreover, the changes in the data in the original data table are determined based on the data change operations on the data source, rather than actively changed and determined by the data warehouse, so that the accuracy of the data changes can be ensured and the probability of errors in the automatic adjustment process can be reduced; in addition, since the data in the data warehouse has its own operation label, no matter what operations the upstream performs on the data, the downstream can track and record them based on the data operation label, so that when data processing errors occur, the problem can be accurately checked and located, thereby improving the maintainability of the entire data processing system.
[0091] Based on the data processing system disclosed in the embodiments shown in Figures 2 and 3, embodiments of the present application further provide a data processing method, which can be applied to a computing engine, such as the computing engine in the data processing system disclosed in the embodiments shown in Figures 2 or 3. Please refer to Figure 4, which is a flow chart of a data processing method provided in one embodiment of the present application. The method includes the following steps:
[0092] S402: Obtain data operation information for the data source from the message middleware.
[0093] The data operation information includes the first target data and the data change operation information corresponding to the first target data. The first target data includes data that has changed in the data source. The data change operation information corresponding to the first target data describes the data change operation performed on the first target data in the data source, such as the type of data change operation and the data before and after the data change operation. The data operation information is obtained by the monitoring component by acquiring and parsing the data change log of the data source in real time. The monitoring component then sends the data operation information to the message middleware.
[0094] S404: Based on the data change operation information, mark the first target data with a corresponding operation tag.
[0095] The operation tag of the first target data is used to indicate a data change operation performed on the first target data in the data source.
[0096] S406: Write the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.
[0097] In some embodiments of the present application, the data warehouse analyzes and processes the data in the original data table, including: determining the task type of the target analysis and processing task to be executed and the target operation label that matches the task type; obtaining a template operation statement corresponding to the task type, and the template operation statement is used to execute the analysis and processing task corresponding to the task type; based on the target operation label, generating a filter condition statement, and the filter condition statement is used to filter out the data in the original data table with the target operation label; splicing the template operation statement and the filter condition statement to obtain a target operation statement for processing the target analysis and processing task, and executing the target analysis and processing task on the data in the original data table by executing the target operation statement.
[0098] In some embodiments of the present application, the target analysis and processing task includes a data cleaning task, which is used to clean the data in the original data table. The target operation label corresponding to the data cleaning task includes a deletion label, which is used to indicate that the data change operation performed on the corresponding data in the data source is a deletion operation; the data warehouse includes a source layer and a detail data layer, the source layer is used to store the original data table, and the detail data layer is used to execute the steps of "determining the task type of the target analysis and processing task to be executed and the target operation label matching the task type" to "executing the target operation statement" to obtain a cleaned data table.
[0099] In some embodiments of the present application, the data warehouse also includes at least one of the following processing layers: a data service layer, which is used to summarize the data in the cleaned data table according to the template wide table corresponding to the preset topic, and obtain the topic wide table corresponding to the preset topic; an application data layer, which is used to perform statistical analysis on the data in the topic wide table according to the business statistical dimension related to the target business, and obtain the business dimension data table corresponding to the business statistical dimension; a common dimension layer, which is used to perform statistical analysis on the data in the topic wide table and the data in the business dimension data table according to the preset common statistical dimension, and obtain the common dimension data table corresponding to the common statistical dimension.
[0100] In some embodiments of the present application, the data processing method further includes: utilizing an application component to respond to a business processing request, obtaining the second target data from the data warehouse based on the data type of the second target data required for the target business requested to be processed by the business processing request, and processing the target business based on the second target data.
[0101] In some embodiments of the present application, the data processing method further includes: if the data type of the second target data is real-time data, using the application component to obtain the second target data from the original data table, and processing the target business based on the second target data and the operation tag of the second target data; if the data type of the second target data is offline data, using the application component to obtain the second target data from the business dimension data table and / or the general dimension data table, and processing the target business based on the second target data.
[0102] In some embodiments of the present application, the original data table includes a data field and a status field, the data field is used to record the data synchronized from the data source, and the status field is used to record the status of the data synchronized from the data source in the data source; the computing engine writes the first target data and the operation tag of the first target data into the original data table of the data warehouse, including: updating the field content of the data field based on the first target data, and updating the field content of the status field based on the operation tag of the first target data.
[0103] In some embodiments of the present application, the data processing method further includes: using a storage component to obtain data with an operation tag as a delete tag from the original data table, and using the data with the operation tag as the delete tag as third target data, and performing cold standby processing on the third target data, and / or writing the third target data to a disk, wherein the delete tag is used to indicate in the data source that the data change operation performed on the corresponding data is a delete operation.
[0104] It should be noted that the specific implementation method of each step in the data processing method provided in the embodiment of the present application can refer to the specific implementation method of the corresponding function in the computing engine in the above-mentioned data processing system. Since the principles are the same, they will not be repeated here.
[0105] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0106] In addition, corresponding to the data processing method shown in FIG. 4 , an embodiment of the present application further provides a data processing device, which can be applied to a computing engine, such as the computing engine in the data processing system disclosed in the embodiment shown in FIG. 2 or FIG. Referring to FIG. 5 , a schematic diagram of the structure of a data processing device 500 provided in one embodiment of the present application is shown. The device 500 includes:
[0107] An acquiring unit 510 is configured to acquire data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information being acquired by a monitoring component after acquiring a data change log of the data source in real time and parsing the data change log, and the monitoring component sending the data operation information to the message middleware;
[0108] A marking unit 520 is configured to mark the first target data with a corresponding operation label based on the data change operation information, where the operation label is used to indicate the data change operation performed on the first target data in the data source;
[0109] The access unit 530 is configured to write the first target data and the operation tag of the first target data into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.
[0110] Obviously, the data processing device provided in the embodiment of the present application can be used as the execution subject of the data processing method shown in Figure 4, and thus can realize the functions realized by the data processing method in Figure 4. Since the principles are the same, they will not be described in detail here.
[0111] FIG6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, on which a computing engine is running. Referring to FIG6 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.
[0112] The processor, network interface, and memory can be interconnected via an internal bus, such as an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. These buses can be classified as address buses, data buses, and control buses. For ease of illustration, FIG6 shows only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.
[0113] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.
[0114] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0115] Obtaining data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, the data operation information being obtained by a monitoring component acquiring a data change log of the data source in real time and parsing the data change log, and the monitoring component sending the data operation information to the message middleware;
[0116] Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source;
[0117] The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.
[0118] The method performed by the data processing device disclosed in the embodiment shown in Figure 4 of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.
[0119] The electronic device can also execute the method of FIG4 and realize the functions of the data processing device in the embodiment shown in FIG4 , and the embodiments of the present application will not be described in detail here.
[0120] Of course, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the processing flow of the data processing method is not limited to each logic unit, but can also be hardware or logic devices. The processing flow of the data processing method includes:
[0121] Obtaining data operation information for a data source from the message middleware, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data that has changed in the data source, the data operation information being obtained by a monitoring component acquiring a data change log of the data source in real time and parsing the data change log, and the monitoring component sending the data operation information to the message middleware;
[0122] Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source;
[0123] The first target data and the operation tag of the first target data are written into the original data table of the data warehouse, so that the data warehouse can analyze and process the data in the original data table.
[0124] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
[0125] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0126] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.
[0127] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0128] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information being obtained by acquiring a data change log of the data source and parsing the data change log; Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source; The first target data and the operation tag of the first target data are written into an original data table to analyze and process the data in the original data table.
2. The data processing method according to claim 1, characterized in that: The analyzing and processing of the data in the original data table comprises: Determine the task type of the target analysis and processing task to be executed and the target operation tag matching the task type; Acquire a template operation statement corresponding to the task type, wherein the template operation statement is used to execute the analysis and processing task corresponding to the task type; Based on the target operation tag, generating a filtering condition statement, wherein the filtering condition statement is used to filter out the data in the original data table having the target operation tag; The template operation statement and the filter condition statement are concatenated to obtain a target operation statement for processing the target analysis and processing task, and the target analysis and processing task is executed on the data in the original data table by executing the target operation statement.
3. The data processing method according to claim 2, characterized in that: The target analysis and processing task includes a data cleaning task, which is used to clean the data in the original data table. The target operation tag corresponding to the data cleaning task includes a delete tag, which is used to indicate that the data change operation performed on the corresponding data in the data source is a delete operation. The method also includes: storing the original data table; and executing the steps of "determining the task type of the target analysis and processing task to be executed and the target operation label matching the task type" to "executing the target operation statement" to obtain a cleaned data table.
4. The data processing method according to claim 3, characterized in that: The method further includes: According to the template wide table corresponding to the preset theme, the data in the cleaned data table is summarized to obtain the theme wide table corresponding to the preset theme; According to the business statistical dimension related to the target business, statistical analysis is performed on the data in the subject wide table to obtain a business dimension data table corresponding to the business statistical dimension; According to the preset common statistical dimension, statistical analysis is performed on the data in the subject wide table and the data in the business dimension data table to obtain a common dimension data table corresponding to the common statistical dimension.
5. The data processing method according to claim 4, characterized in that: The method further includes: In response to a business processing request, the second target data is acquired based on a data type of the second target data required for the target business processed by the business processing request, and the target business is processed based on the second target data.
6. The data processing method according to claim 5, characterized in that: The method further includes: If the data type of the second target data is real-time data, acquiring the second target data from the original data table, and processing the target business based on the second target data and the operation tag of the second target data; If the data type of the second target data is offline data, the second target data is obtained from the business dimension data table and / or the general dimension data table, and the target business is processed based on the second target data.
7. The data processing method according to claim 1, characterized in that: The original data table includes a data field and a status field, the data field is used to record the data synchronized from the data source, and the status field is used to record the status of the data synchronized from the data source in the data source; Writing the first target data and the operation tag of the first target data into the original data table includes: updating the field content of the data field based on the first target data, and updating the field content of the status field based on the operation tag of the first target data.
8. The data processing method according to any one of claims 1 to 7, characterized in that: The method further includes: From the original data table, data having an operation tag of a delete tag is obtained, and the data having an operation tag of a delete tag is used as third target data, and cold standby processing is performed on the third target data, and / or the third target data is written to a disk, wherein the delete tag is used to indicate in the data source that the data change operation performed on the corresponding data is a delete operation.
9. An electronic device, characterized in that: include: Memory; and processor, The memory stores a computer program, and the processor is used to execute the computer program to implement the following steps: Acquire data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information being obtained by acquiring a data change log of the data source and parsing the data change log; Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source; The first target data and the operation tag of the first target data are written into an original data table to analyze and process the data in the original data table.
10. The electronic device according to claim 9, characterized in that: The analyzing and processing of the data in the original data table comprises: Determine the task type of the target analysis and processing task to be executed and the target operation tag matching the task type; Acquire a template operation statement corresponding to the task type, wherein the template operation statement is used to execute the analysis and processing task corresponding to the task type; Based on the target operation tag, generating a filtering condition statement, wherein the filtering condition statement is used to filter out the data in the original data table having the target operation tag; The template operation statement and the filter condition statement are concatenated to obtain a target operation statement for processing the target analysis and processing task, and the target analysis and processing task is executed on the data in the original data table by executing the target operation statement.
11. The electronic device according to claim 10, characterized in that: The target analysis and processing task includes a data cleaning task, which is used to clean the data in the original data table. The target operation tag corresponding to the data cleaning task includes a delete tag, which is used to indicate that the data change operation performed on the corresponding data in the data source is a delete operation. The processor executes the computer program to further implement the following steps: storing the original data table; and executing the steps of "determining the task type of the target analysis and processing task to be executed and the target operation label matching the task type" to "executing the target operation statement" to obtain a cleaned data table.
12. The electronic device according to claim 11, characterized in that: The processor executes the computer program to further implement the following steps: According to the template wide table corresponding to the preset theme, the data in the cleaned data table is summarized to obtain the theme wide table corresponding to the preset theme; According to the business statistical dimension related to the target business, statistical analysis is performed on the data in the subject wide table to obtain a business dimension data table corresponding to the business statistical dimension; According to the preset common statistical dimension, statistical analysis is performed on the data in the subject wide table and the data in the business dimension data table to obtain a common dimension data table corresponding to the common statistical dimension.
13. The electronic device according to claim 12, characterized in that: The processor executes the computer program to further implement the following steps: In response to a business processing request, the second target data is acquired based on a data type of the second target data required for the target business processed by the business processing request, and the target business is processed based on the second target data.
14. The electronic device according to claim 13, characterized in that: The processor executes the computer program to further implement the following steps: If the data type of the second target data is real-time data, acquiring the second target data from the original data table, and processing the target business based on the second target data and the operation tag of the second target data; If the data type of the second target data is offline data, the second target data is obtained from the business dimension data table and / or the general dimension data table, and the target business is processed based on the second target data.
15. The electronic device according to claim 9, characterized in that: The original data table includes a data field and a status field, the data field is used to record the data synchronized from the data source, and the status field is used to record the status of the data synchronized from the data source in the data source; Writing the first target data and the operation tag of the first target data into the original data table includes: updating the field content of the data field based on the first target data, and updating the field content of the status field based on the operation tag of the first target data.
16. The electronic device according to any one of claims 9 to 15, characterized in that: The processor executes the computer program to further implement the following steps: From the original data table, data having an operation tag of a delete tag is obtained, and the data having an operation tag of a delete tag is used as third target data, and cold standby processing is performed on the third target data, and / or the third target data is written to a disk, wherein the delete tag is used to indicate in the data source that the data change operation performed on the corresponding data is a delete operation.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, wherein the instructions, when executed by the processor, implement the following steps: Acquire data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information being obtained by acquiring a data change log of the data source and parsing the data change log; Based on the data change operation information, marking a corresponding operation tag for the first target data, where the operation tag is used to indicate the data change operation performed on the first target data in the data source; The first target data and the operation tag of the first target data are written into an original data table to analyze and process the data in the original data table.
18. The computer-readable storage medium of claim 17, wherein: The analyzing and processing of the data in the original data table comprises: Determine the task type of the target analysis and processing task to be executed and the target operation tag matching the task type; Acquire a template operation statement corresponding to the task type, wherein the template operation statement is used to execute the analysis and processing task corresponding to the task type; Based on the target operation tag, generating a filtering condition statement, wherein the filtering condition statement is used to filter out the data in the original data table having the target operation tag; The template operation statement and the filter condition statement are concatenated to obtain a target operation statement for processing the target analysis and processing task, and the target analysis and processing task is executed on the data in the original data table by executing the target operation statement.
19. The computer-readable storage medium of claim 18, wherein: The target analysis and processing task includes a data cleaning task, which is used to clean the data in the original data table. The target operation tag corresponding to the data cleaning task includes a delete tag, which is used to indicate that the data change operation performed on the corresponding data in the data source is a delete operation. When the instructions are executed by the processor, the following steps are also implemented: storing the original data table; and executing the steps of "determining the task type of the target analysis and processing task to be executed and the target operation label matching the task type" to "executing the target operation statement" to obtain a cleaned data table.
20. A data processing device, characterized in that: The data processing device comprises: an acquiring unit, configured to acquire data operation information for a data source, the data operation information including first target data and data change operation information corresponding to the first target data, the first target data including data changed in the data source, the data operation information being acquired by acquiring a data change log of the data source and parsing the data change log; a marking unit, configured to mark a corresponding operation tag for the first target data based on the data change operation information, wherein the operation tag is used to indicate the data change operation performed on the first target data in the data source; The access unit is used to write the first target data and the operation tag of the first target data into an original data table to analyze and process the data in the original data table.
Citation Information
Patent Citations
Data change condition obtaining method and device, and electronic equipment
CN111797104A
Data synchronization method and device, electronic equipment and storage medium
CN114443599A
Data processing method and device, electronic equipment and storage medium
CN114610804A
Data processing method, device and equipment and computer readable storage medium
CN115130794A
Business data change recording method and device, computer equipment and medium
CN115858471A
Cited By
Data transmission method, device and equipment based on Apache Flink and medium
CN119603344A
An Apache Flink-based data transmission method, device, equipment and medium
CN119603344B
Data real-time analysis output method and system
CN120086256A
Multi-source data processing method and device, storage medium and electronic equipment
CN120353856A
Equipment data storage Dores method based on industrial internet
CN120386780A
