Data processing method and system, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and system, electronic equipment and a computer readable storage medium. The method comprises the steps that change information of a service database is acquired; determining an adjustment state of the data warehouse according to the change information and a mapping relationship between a source database table in the business database and a target database table in the data warehouse; and in response to the condition that the adjustment state is to be adjusted, adjusting metadata of the data warehouse and / or a data acquisition task of the data warehouse according to the change information. Therefore, the change of the business system database can be automatically perceived and responded by perceiving the change of the source database table in the business database and timely adjusting the metadata of the data warehouse and / or the data acquisition task according to the source database table.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data processing method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] In modern enterprise architecture, data warehouses play a crucial role, aggregating massive amounts of data from multiple internal business systems to provide management with in-depth analytical reports and strategic decision support. However, with the rapid development of business operations and the continuous evolution of the technological environment, various business systems often undergo frequent iterations and changes, including adjustments to data table structures and the addition or removal of fields. If these changes are not communicated to the data warehouse team in a timely and effective manner, it will directly lead to errors and omissions in the data collection process, thereby affecting the accuracy of the data within the data warehouse. Ultimately, this may distort the analytical reports and decision support information generated based on this data, causing unnecessary losses and risks to the enterprise.

[0003] Therefore, in order to meet this challenge, it is particularly important to build an efficient and agile data warehouse change awareness solution to be able to detect and respond to any iterative changes to the upstream business system database in a timely manner. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data processing method, system, electronic device, and computer-readable storage medium to automatically detect and respond to changes in the business system database.

[0005] In a first aspect, embodiments of the present invention aim to provide a data processing method, the method comprising:

[0006] Obtain change information from the business database, wherein the change information is used to indicate that the source database tables in the business database have been changed;

[0007] Based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse, the adjustment status of the data warehouse is determined.

[0008] In response to the adjustment status being pending, the metadata of the data warehouse and / or the data collection tasks of the data warehouse are adjusted according to the change information.

[0009] Further, determining the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse includes:

[0010] The change information is identified to determine the change source object and change data. The change source object is the source database table in the business database that has undergone the change, and the change data is used to characterize the metadata of the changed source database table.

[0011] The target object for change is determined from the data warehouse based on the mapping relationship;

[0012] The changed data is compared with the metadata of the target object to determine the comparison result, which is used to characterize the integrity of the target object.

[0013] In response to the comparison result indicating that the target object for the change is incomplete, the adjustment status of the data warehouse is determined to be pending adjustment.

[0014] Furthermore, the step of determining the data warehouse's adjustment status as "pending adjustment" in response to the comparison result indicating that the target object for change is incomplete includes:

[0015] In response to the comparison result indicating that the target object to be changed is incomplete, the data type of the changed data is validated.

[0016] In response to the fact that the data type of the changed data is a preset type, the adjustment status of the data warehouse is determined to be pending adjustment.

[0017] Furthermore, determining the target object for change from the data warehouse based on the mapping relationship includes:

[0018] In response to the fact that the source object of the change is an existing source database table, the target object of the change is determined according to the mapping relationship. The target object of the change is a target database table in the data warehouse that has a mapping relationship with the source object of the change.

[0019] Furthermore, determining the target object for change from the data warehouse based on the mapping relationship includes:

[0020] In response to the change source object being a newly added source database table, a reference source object corresponding to the change source object is determined, wherein the reference source object is a source database table in the business database that is related to the change source object;

[0021] The target object for change is determined based on the mapping relationship. The target object for change is a target database table in the data warehouse that has a mapping relationship with the reference source object.

[0022] Furthermore, obtaining change information from the business database includes:

[0023] Use the database console, data log-based change tracking, or set database triggers to detect change events in the business database;

[0024] In response to the detection of a change event, the change information of the business database is obtained.

[0025] Furthermore, the change event includes at least one of the following: adding or deleting a source database table in the business database, adding or deleting a field in the source database table, or modifying the field enumeration value in the source database table.

[0026] Furthermore, obtaining change information from the business database includes:

[0027] The target database tables in the data warehouse are examined to determine the data collection results;

[0028] In response to the data collection results not meeting expectations, change information in the business database is obtained.

[0029] Furthermore, the method also includes:

[0030] In response to the adjustment status being pending, a change warning message is sent to relevant personnel via a preset communication interface.

[0031] Secondly, embodiments of the present invention aim to provide a data processing system, the system comprising:

[0032] A sensing unit is used to acquire change information of the business database, the change information being used to indicate that the source database table in the business database has been changed;

[0033] A response unit is configured to determine the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database table in the business database and the target database table in the data warehouse; and in response to the adjustment status being pending adjustment, to adjust the metadata of the data warehouse and / or the data acquisition task of the data warehouse based on the change information.

[0034] Furthermore, the system also includes:

[0035] The early warning unit is used to send change warning information to relevant personnel based on a preset communication interface in response to the adjustment status being pending adjustment.

[0036] Thirdly, embodiments of the present invention aim to provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the method described in any of the preceding claims.

[0037] Fourthly, embodiments of the present invention aim to provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any of the preceding claims.

[0038] Fifthly, embodiments of the present invention aim to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0039] The technical solution of this invention obtains change information representing changes in the source database tables in the business database, determines the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse, and adjusts the metadata and / or data acquisition tasks of the data warehouse according to the change information when the adjustment status is pending. This enables timely detection of changes in the source database tables in the business database and timely adjustment of the metadata and / or data acquisition tasks of the data warehouse based on the source database tables, thereby achieving automated detection and response to changes in the business system database. Attached Figure Description

[0040] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0041] Figure 1 This is a schematic diagram of the data acquisition process according to an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of the data processing method according to an embodiment of the present invention;

[0043] Figure 3 This is a flowchart illustrating the process of obtaining change information according to an embodiment of the present invention;

[0044] Figure 4 This is a flowchart illustrating the process of determining the adjustment status of a data warehouse according to an embodiment of the present invention;

[0045] Figure 5 This is a flowchart illustrating the determination of the target database according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the processing procedure according to an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the data processing system according to an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0049] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0050] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0051] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0052] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0053] The solutions described in this specification and embodiments, if involving information acquisition, will collect data under legal and compliant conditions, ensuring the legality of the data source, and will take appropriate technical and management measures to ensure data security. If involving personal information processing, processing will be carried out under legal grounds (e.g., obtaining the consent of the personal information subject, or being necessary for contract performance), and will only be conducted within the prescribed or agreed scope. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0054] Data warehouses acquire necessary data from business databases within business systems by performing data acquisition tasks. During data acquisition, if data in the business system changes, a method is needed to ensure the accuracy of the data in the data warehouse, enabling it to promptly detect and respond to these changes. However, traditional data warehouse data acquisition heavily relies on manual communication to transmit change information and on manually adjusting the data within the warehouse. While this approach can improve data accuracy to some extent, it presents significant limitations and challenges in terms of data timeliness, operational costs, automation levels, and information integrity. Therefore, this invention aims to provide a data processing method that automates the detection and response to changes in business system databases, thereby improving the automation level, data timeliness, operational costs, and information integrity during the data acquisition process.

[0055] Figure 1 This is a schematic diagram of the data acquisition process according to an embodiment of the present invention. For example... Figure 1 As shown, the data acquisition process is completed based on the interaction between the business database 10 and the data warehouse 20. The business database 10 is a database that stores business data related to the business system. The business system is a complete information system supporting the daily operations and business processes of an enterprise, covering multiple business aspects such as procurement, inventory, production, finance, and human resources. The data carried or flowing in the business database (which can be structured or unstructured data) serves as the data source for data acquisition from the data warehouse. The data warehouse 20 (Data Warehouse, abbreviated as DW or DWH) is a crucial foundational measure in the era of big data. It is a subject-oriented, integrated, relatively stable dataset that reflects historical changes. It is primarily used to support the enterprise's decision-making process, providing the data needed for analytical reporting and decision support.

[0056] When implementing data collection, such as Figure 1 As shown, in this embodiment, a mapping relationship (e.g., source database table A - target database table A) between the source database table in the business database 10 and the target database table in the data warehouse 20 is pre-established. The data warehouse 20 collects data from the source database table in the business database 10 using open-source tools and other methods, and fills the collected data from the source database table into the corresponding target database table based on the mapping relationship between the source database table and the target database table.

[0057] When changes occur in the source database tables of the business database, including the addition and deletion of source database tables in the business database, structural changes to existing source database tables (including the addition and deletion of fields, and changes in field enumeration values), the comprehensive and efficient automated processing flow constructed by the data processing method in this embodiment can promptly detect changes in the upstream business system. After detecting a change, an automated response mechanism is quickly activated to determine the consistency and accuracy of data between the data warehouse and the business database, and data adjustments are made based on data consistency. This achieves automated detection and response to changes in the business system database. This not only improves the level of automation in the data acquisition process, data timeliness, operation and maintenance costs, and information integrity, but also enables enterprises to build a more intelligent, efficient, and reliable data processing system, providing strong support for the enterprise's digital transformation and business innovation.

[0058] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the data processing method in this embodiment includes the following steps.

[0059] In step S210, change information of the business database is obtained. The change information is used to indicate that the source database table in the business database has been changed.

[0060] In this embodiment, change detection in the business database can be triggered in any way, thereby obtaining change information in the business database in a timely manner when changes occur. For example, the acquisition of change information in the business database can be triggered by real-time detection of the business database, or it can be triggered by detecting the data collection results in the data warehouse.

[0061] Optionally, since the business database is the source of change information, in this embodiment, when triggering by real-time detection of the business database, changes to the business database can be monitored in real time and change information of the business database can be obtained through technical means such as metadata management, log analysis, and database triggers.

[0062] Figure 3 This is a flowchart illustrating the process of obtaining change information according to an embodiment of the present invention. For example... Figure 3 As shown, in this embodiment, the change information of the business database is obtained through the following method.

[0063] In step S310, change events in the business database are detected using the database console, data log-based change tracking, or by setting database triggers.

[0064] In this embodiment, the change events of the business database include at least one of the following: adding or deleting a source database table in the business database, adding or deleting a field in the source database table, or modifying the enumeration value of a field in the source database table.

[0065] When using a database console (also known as a DB console) to detect change events, in an environment with a database console, all change operations on source database tables in the business database are submitted through this console. In this case, by integrating monitoring tools, the SQL statements executed within the console can be captured and analyzed in real time, thereby accurately capturing every change to the source database and achieving real-time and effective detection of change events in the business database.

[0066] When detecting change events based on data logs, the business database's own logging system, such as MySQL's binary log (binlog), can be utilized. By monitoring and analyzing these log files, all change events occurring within the business database can be captured.

[0067] When detecting change events by setting database triggers, triggers can be set at the database level of the business database. The triggers will be automatically triggered when the table structure or data content of the source database changes, and the specific changes will be synchronized to the specified recording system or analysis platform in real time.

[0068] Furthermore, in this embodiment, a suitable method can be selected to detect change events in the business database based on the specific use case. For example, in scenarios with high real-time requirements, the database console is preferred for detecting change events; in scenarios without a database console, change events can be detected by tracking changes based on data logs or by setting database triggers. Therefore, by providing multiple methods for detecting change events in the business database, this embodiment enables real-time capture and comprehensive insight into the dynamic changes of the business database in different scenarios.

[0069] In step S320, in response to the detection of a change event, change information in the business database is obtained.

[0070] In this embodiment, the change information includes change-related information such as the change source object, changed data, original data, and change event identifier. The change source object is the source database table in the business database that has undergone the change. The changed data represents the metadata of the source database table after the change. The original data represents the metadata of the source database table before the change. The change event identifier represents the type of change event, which includes changes to the business database structure (including adding or deleting the source database table), changes to the source database table structure (including adding or deleting fields in the source database table), and changes to database table data (including modifying field enumeration values ​​in the source database table).

[0071] Optionally, in this embodiment, after detecting a change event using the above method, the change information is determined by comparing the metadata of the business database before and after the change event, thereby obtaining the change information of the business database.

[0072] Therefore, in this embodiment, by using the database console, data log-based change tracking, or setting database triggers to detect change events in the business database, changes in the business database can be obtained, enabling real-time capture and comprehensive insight into the dynamic changes of the business database from the business database side.

[0073] Alternatively, considering that changes in business database data directly affect the data collection results of the data warehouse, the detection of whether the business database has changed and its change information can be triggered by the data collection results after the data warehouse data collection. Specifically, when triggering by detecting the data collection results in the data warehouse, this embodiment first detects the target database tables in the data warehouse to determine the data collection results. If the data collection results do not meet expectations, the change information of the business database is obtained. For example, suppose the purpose of the data collection task is to collect data from five source database tables in the business database, but after the data collection task is completed, only two target database tables corresponding to the source database tables exist in the data warehouse. At this time, the data collection results do not meet expectations, and the detection of the business database during the data collection process is triggered to obtain the change information of the business database. Thus, this embodiment can achieve dynamic changes and capture of the business database from the data warehouse side, realizing timely perception of business database changes.

[0074] Furthermore, after triggering the acquisition of change information in the business database, this embodiment can use the aforementioned methods of using the database console, change tracking based on data logs, or setting database triggers to detect change events in the business database to acquire change information in the business database.

[0075] In step S220, the adjustment status of the data warehouse is determined based on the change information and the mapping relationship between the source database table in the business database and the target database table in the data warehouse.

[0076] In this embodiment, after detecting a change in the business database and obtaining the change information, it is considered that not all changes to the business database will affect the data warehouse. For example, a new source database table may be added to the business database, but this added source database table is not a change to the source database table that the data warehouse needs to collect, and therefore will not affect the data collection of the data warehouse. Therefore, this embodiment determines the adjustment status of the data warehouse based on the change information of the business database and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse. This allows for responding to and processing only necessary changes, reducing unnecessary change processing operations, and improving the overall data collection efficiency of the data warehouse while ensuring the accuracy of data collection.

[0077] Optionally, when the change information obtained by change perception and the change information used when responding to changes are in the same format, in this embodiment, the source database table that has changed is first located according to the change information, and then the target database table corresponding to the source database table that has changed is determined according to the mapping relationship between the source database table and the target database table. Based on the data integrity of the target database table, the adjustment status of the target database table is determined.

[0078] Alternatively, considering that the change information during change perception and change response usually has different formats, this embodiment uses, for example... Figure 4 The method shown determines the adjustment status of a data warehouse, and specifically includes the following processing steps.

[0079] In step S410, the change information is identified to determine the change source object and change data. The change source object is the source database table in the business database that has undergone changes, and the change data is used to characterize the metadata of the changed source database table.

[0080] In step S420, the target object for the change is determined from the data warehouse based on the mapping relationship.

[0081] Optionally, in this embodiment, when determining the target object for change from the data warehouse based on the mapping relationship, the corresponding method is selected according to the type of the source object for change to determine the target object for change.

[0082] In one optional implementation, for existing source database tables in the business database, this embodiment directly determines the target object to be changed based on the mapping relationship between the source database table and the target database table. Specifically, the data processing method in this embodiment further includes: in response to the change of the source object being an existing source database table, determining the target object to be changed based on the mapping relationship, wherein the target object to be changed is a target database table in the data warehouse that has a mapping relationship with the source object to be changed. Therefore, this embodiment can quickly determine the target database table that needs to be adjusted when the source database table changes, using the above mapping relationship, thereby improving the data warehouse's response efficiency to changes in the business database and the data consistency between the target database table and the corresponding source database table, ensuring the accuracy of the data in the data warehouse.

[0083] In another alternative implementation, when the source database table in the business database does not exist in the original database, that is, when the source object is changed to a newly added source database table, there is generally no target database table in the data warehouse corresponding to the newly added source database table. In this case, the response to the "new source database table" change can be determined based on the relevance between the newly added source database table and the data collection task of the data warehouse.

[0084] Optionally, in this embodiment, the decision on whether to respond to changes in the newly added data source database table can be determined by data acquisition task information, the correlation between the newly added source database table and the existing source database table, or other methods. This can prevent data from being missed due to changes in the location of important data storage during data acquisition in the data warehouse, ensuring the integrity and accuracy of data acquisition and overcoming the risk of information omission or loss.

[0085] Furthermore, in this embodiment, the corresponding target database table is determined based on the correlation between the newly added source database table and the existing source database table, thereby responding to changes in the business database when a new source database table is added.

[0086] Figure 5 This is a flowchart illustrating the determination of the target database according to an embodiment of the present invention. For example... Figure 5 As shown, the target database is determined in this embodiment in the following way.

[0087] In step S510, in response to the change source object being a newly added source database table, a reference source object corresponding to the change source object is determined. The reference source object is a source database table in the business database that is related to the change source object.

[0088] Optionally, in this embodiment, when the change source object corresponding to the change information is a newly added source database table, the correlation between the existing source database tables in the business database and the change source object is first determined, and then the source database table with the strongest correlation is determined as the reference source object of the change source object.

[0089] Furthermore, in this embodiment, the relevance between the existing source database table and the changed source object can be determined based on the similarity of fields in the corresponding database tables. When determining similarity, existing similarity analysis methods are first used to determine the similarity between different fields in the two database tables. Then, a weighted sum of the similarities of each field is calculated to determine the overall similarity between the two database tables. The larger the weighted sum, the higher the similarity between the database tables. Therefore, in this embodiment, by determining the similarity between the reference source object and the existing original database table in the business database, a reference source object similar to the changed source object is identified.

[0090] In step S520, the target object for change is determined based on the mapping relationship. The target object for change is a target database table in the data warehouse that has a mapping relationship with the reference source object.

[0091] In this embodiment, after determining the target database table corresponding to the reference source object based on the mapping relationship, the target database table is determined as the target database table of the source object to be changed.

[0092] In step S430, the changed data is compared with the metadata of the target object to determine the comparison result, which is used to characterize the integrity of the target object.

[0093] In this embodiment, the metadata of the target object describes the attributes of the corresponding target database table itself, including the field names, fields, and sources of the target database table. By comparing the changed data with the metadata of the target object, it can be determined whether the changed data of the source object is present in the metadata of the target object. When the metadata of the target object includes the changed data, it indicates that the changes in the source object have been synchronized to the target object, meaning the data warehouse has responded to the changes in the business database, and the data in the target object and the corresponding target database table are complete. When the metadata of the target object does not include the changed data, it indicates that the changes in the source object have not yet been synchronized to the target object, meaning the data warehouse has not yet responded to the changes in the business database, and the data in the target object and the corresponding target database table are incomplete.

[0094] Optionally, in this embodiment, the change information is represented by DDL (Data Definition Language) instructions. After obtaining the change information from the business database, the corresponding DDL instructions are transmitted to the Kafka message queue in real time. During comparison, the stream processing system built on Flink (an open-source stream processing framework) consumes the changed data from the Kafka message queue. During the comparison process, the metadata corresponding to the changed target object in the data warehouse is used as a dimension for correlation and comparison analysis, thereby quickly identifying the data integrity of the changed target object.

[0095] It should be noted that in this embodiment, other real-time processing components (such as Hive, SQL, etc.) besides Kafka message queues and Flink system components can also be used to compare the metadata of the changed data with the metadata of the target object. This is not a limitation.

[0096] Furthermore, if the comparison result indicates that the target object of the change is complete, continue to execute step S440; if the comparison result indicates that the target object of the change is incomplete, continue to execute step S450.

[0097] In step S440, in response to the comparison result indicating that the target object of the change is complete, the adjustment status of the data warehouse is determined to be no adjustment.

[0098] In this embodiment, when the target object is completely changed, it indicates that the data warehouse has completed the response to the changes in the business database. At this time, there is no need to adjust the target database table and the data in the target database table in the data warehouse, and the adjustment status of the data warehouse is determined to be no adjustment.

[0099] Furthermore, after determining that the data warehouse adjustment status is no longer needed, the response operation for this change ends, and the detection of changes to the business database during subsequent data collection processes continues, so as to promptly detect changes to the business database and respond quickly to them, thereby achieving automated perception and response to changes in the business system database.

[0100] In step S450, in response to the comparison result indicating that the target object of the change is incomplete, the adjustment status of the data warehouse is determined to be pending adjustment.

[0101] In this embodiment, when the target object is incomplete, it indicates that the data warehouse has not yet completed the response to the changes in the business database. At this time, it is necessary to adjust the target database table and the data in the target database table in the data warehouse, and determine that the adjustment status of the data warehouse is pending adjustment.

[0102] Optionally, to improve the flexibility of the process and the granularity of data management, this embodiment will further confirm the adjustment status of the data warehouse based on preset verification rules when determining the adjustment status of the data warehouse based on the comparison results, so as to ensure that the data in the data warehouse can reflect business needs while avoiding unnecessary data redundancy and potential risks.

[0103] Furthermore, in this embodiment, when the comparison result indicates that the target object to be changed is incomplete and the adjustment status of the data warehouse is determined to be pending adjustment, the specific steps include: in response to the comparison result indicating that the target object to be changed is incomplete, verifying the data type of the changed data; and in response to the data type of the changed data being a preset type, determining that the adjustment status of the data warehouse is pending adjustment.

[0104] Optionally, the preset type in this embodiment can be selected according to business analysis needs, and the preset type can be configured through a rule configuration method to determine the adjustment status of the data warehouse based on the preset type. Further, in this embodiment, when the data type of the changed data is necessary data for business analysis and is suitable for inclusion in the data collection scope (such as non-sensitive data and non-test data), the data type of the changed data is determined to be a preset type, and the adjustment status of the data warehouse is determined to be "to be adjusted." However, when the changed data is non-necessary data, sensitive data, test data, or other data types unsuitable for inclusion in the data collection scope, the data type of the changed data is determined to be not a preset type, and the adjustment status of the data warehouse is determined to be "not adjusted." Therefore, by using preset types to further evaluate the changed data and determine the adjustment status of the data warehouse, the data collection process can achieve on-demand collection and accurate analysis, thereby improving data processing efficiency and accuracy.

[0105] Optionally, in this embodiment, when the adjustment status of the data warehouse is determined to be pending adjustment, a data quality control network covering the entire data acquisition chain is also constructed to further improve the reliability and security of data management, so that the system and relevant personnel can keenly identify any abnormalities or potential data quality problems in the data acquisition process.

[0106] Furthermore, the data processing method in this embodiment also includes: in response to the data warehouse's adjustment status being "to be adjusted," sending a change warning message to relevant personnel via a preset communication interface (such as an IM (instant messaging) group chat or personal notification). Thus, by triggering a warning mechanism when the data warehouse's adjustment status is "to be adjusted" and sending a change warning message to relevant personnel, relevant personnel can be aware of the change and quickly intervene, effectively reducing the risk of data errors and ensuring data accuracy and security.

[0107] In step S230, in response to the adjustment status being pending adjustment, the metadata of the data warehouse and / or the data acquisition task of the data warehouse are adjusted according to the change information.

[0108] In this embodiment, when the data warehouse is in the "pending adjustment" state, the data warehouse metadata and / or the data acquisition task are adjusted based on the change information. Adjusting the data warehouse metadata is a result-oriented adjustment, which synchronizes the relevant data from the changed source database table to the corresponding target database table by updating the target database table. Adjusting the data warehouse data acquisition task is a process-oriented adjustment, which enables the data warehouse to re-acquire the data from the source database table corresponding to the target database table by updating the data acquisition task. Therefore, in this embodiment, during the data acquisition process, whether by adjusting the data warehouse metadata, adjusting the data warehouse data acquisition task, or combining both methods, data synchronization between the target database table and the corresponding source database table can be achieved.

[0109] Optionally, since different adjustment methods have varying response efficiencies under different change events, to further improve the data warehouse's response efficiency to changes in the business database, this embodiment combines two adjustment methods—adjusting the data warehouse's metadata and adjusting the data warehouse's data acquisition tasks—to synchronize data in the target database table with the corresponding source database table. Furthermore, this embodiment determines the corresponding adjustment method for each change based on the change event type in the change information.

[0110] For example, when the change event type is an event that alters the business database structure, such as adding or deleting a source database table, the data warehouse data acquisition task is adjusted according to the change information. When the change event type is an event that alters the source database table structure, such as adding or deleting a field in a source database table, the data warehouse metadata and data acquisition task are adjusted according to the change information. When the change event type is an event that alters database table data, such as modifying field enumeration values ​​in a source database table, the data warehouse metadata is adjusted according to the change information. However, it should be understood that the adjustment methods given in this embodiment are only examples, and specific adjustments can be made according to the actual use case.

[0111] Therefore, in this embodiment, by adjusting the metadata of the data warehouse and the data collection tasks of the data warehouse according to the change information, the relevant data collection tasks can be automatically adjusted and optimized when the business database changes. This ensures that the changes in the business database can be correctly mapped and participate in the data collection process, thereby enabling the data warehouse to collect data from the business database comprehensively and accurately, thus ensuring the integrity and timeliness of the entire data ecosystem.

[0112] The technical solution of this embodiment obtains change information representing changes in the source database tables in the business database, determines the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse, and adjusts the metadata of the data warehouse and / or the data acquisition tasks of the data warehouse according to the change information when the adjustment status is pending. This enables timely detection of changes in the source database tables in the business database and timely adjustment of the metadata and / or data acquisition tasks of the data warehouse according to the source database tables, thereby achieving automated detection and response to changes in the business system database.

[0113] Figure 6 This is a schematic diagram of the processing procedure according to an embodiment of the present invention. For example... Figure 6As shown, in this embodiment, the data acquisition process is based on the mapping relationship between source database table A in business database 10 and target database table A in data warehouse 20. Data from source database table A is collected and populated into target database table B. Simultaneously, during the data acquisition process, change events in the business database are detected using techniques such as database console, change tracking based on data logs, or setting database triggers. Upon detecting a change in the business database, the data warehouse response chain is automatically triggered, transmitting the statement information (i.e., DDL instructions) corresponding to the change event to a Kafka message queue in real time. A stream processing system built on Flink (an open-source stream processing framework) consumes the change information from the Kafka message queue and compares and analyzes the change information with the metadata of the target database table. Subsequently, if the comparison results indicate that there is a data inconsistency between the target database table and the corresponding source database table, i.e., the target database table is incomplete, a change warning message is sent to relevant personnel via a preset communication interface to ensure that change issues are detected and addressed promptly. Furthermore, if the change to the business database involves a change to the source database table structure, the target database table structure needs to be adjusted. In this case, the system will automatically execute field modification operations on the target database table to ensure the data warehouse accurately reflects the latest state of the business data. After adjusting the target database table structure, the system will also automatically adjust and optimize related data acquisition tasks, ensuring that the changed target database table can be correctly mapped and participate in the data acquisition process. Afterward, the system will resubmit and restart the data acquisition task, thereby achieving comprehensive and accurate data acquisition from the changed business database. Therefore, this embodiment, through the above method, can detect changes to the business database in real time and seamlessly connect to the data warehouse management process after detecting the change, automatically executing data update operations in the data warehouse. This ensures the continuity of data flow and the accuracy of data without interrupting service, greatly improving data processing efficiency and flexibility. It not only enables automated perception and response to changes in the business system database but also facilitates the construction of a more intelligent, efficient, and reliable data processing system.

[0114] Figure 7 This is a schematic diagram of a data processing system according to an embodiment of the present invention. Figure 7As shown, the data processing system in this embodiment includes a metadata management unit 1, a sensing unit 2, and a response unit 3. The metadata management unit 1 establishes a comprehensive metadata management system, recording and maintaining information such as data mapping relationships, table structures, and field definitions between all business databases and the data warehouse, providing a foundation for change identification and response. The sensing unit 2 acquires change information from the business databases, which indicates changes to source database tables in the business databases, enabling timely detection of changes. The response unit 3 determines the adjustment status of the data warehouse based on the change information and the mapping relationship between source database tables in the business databases and target database tables in the data warehouse; and, in response to the adjustment status being "to be adjusted," adjusts the metadata of the data warehouse and / or the data collection tasks of the data warehouse based on the change information, thereby automatically triggering the corresponding adjustment process of the data warehouse when a change in the business database is detected, ensuring the continuity and accuracy of data collection.

[0115] Optionally, the data processing system in this embodiment further includes an early warning unit 4. The early warning unit 4 is used to send change warning information to relevant personnel based on a preset communication interface in response to the data warehouse's adjustment status being pending adjustment.

[0116] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 8 As shown, Figure 8 The illustrated electronic device is a general-purpose data processing device, comprising a general-purpose computer hardware architecture, including at least a processor 81 and a memory 82. The processor 81 and memory 82 are connected via a bus 83. The memory 82 is adapted to store instructions or programs executable by the processor 81. The processor 81 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 81 executes the instructions stored in the memory 82, thereby performing the method flow of the embodiments of the present invention as described above to process data and control other devices. The bus 83 connects the aforementioned components together, and also connects these components to a display controller 84, a display device, and an input / output (I / O) device 85. The input / output (I / O) device 85 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 85 is connected to the system via an input / output (I / O) controller 86.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0119] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0120] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0121] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0122] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain change information from the business database, wherein the change information is used to indicate that the source database tables in the business database have been changed; Based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse, the adjustment status of the data warehouse is determined. In response to the adjustment status being pending, the metadata of the data warehouse and / or the data acquisition task of the data warehouse are adjusted according to the change information.

2. The method according to claim 1, characterized in that, Determining the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database tables in the business database and the target database tables in the data warehouse includes: The change information is identified to determine the change source object and change data. The change source object is the source database table in the business database that has undergone the change, and the change data is used to characterize the metadata of the changed source database table. The target object for change is determined from the data warehouse based on the mapping relationship; The changed data is compared with the metadata of the target object to determine the comparison result, which is used to characterize the integrity of the target object. In response to the comparison result indicating that the target object for the change is incomplete, the adjustment status of the data warehouse is determined to be pending adjustment.

3. The method according to claim 2, characterized in that, The step of determining the data warehouse's adjustment status as pending adjustment in response to the comparison result indicating that the target object for change is incomplete includes: In response to the comparison result indicating that the target object to be changed is incomplete, the data type of the changed data is validated. In response to the fact that the data type of the changed data is a preset type, the adjustment status of the data warehouse is determined to be pending adjustment.

4. The method according to claim 2, characterized in that, The step of determining the target object for change from the data warehouse based on the mapping relationship includes: In response to the fact that the source object of the change is an existing source database table, the target object of the change is determined according to the mapping relationship. The target object of the change is a target database table in the data warehouse that has a mapping relationship with the source object of the change.

5. The method according to claim 2, characterized in that, The step of determining the target object for change from the data warehouse based on the mapping relationship includes: In response to the change source object being a newly added source database table, a reference source object corresponding to the change source object is determined, wherein the reference source object is a source database table in the business database that is related to the change source object; The target object for change is determined based on the mapping relationship. The target object for change is a target database table in the data warehouse that has a mapping relationship with the reference source object.

6. The method according to claim 1, characterized in that, The acquisition of change information from the business database includes: Use the database console, data log-based change tracking, or set database triggers to detect change events in the business database; In response to the detection of a change event, the change information of the business database is obtained.

7. The method according to claim 6, characterized in that, The change events include at least one of the following: adding or deleting a source database table in the business database, adding or deleting a field in the source database table, or modifying the enumeration value of a field in the source database table.

8. The method according to claim 1, characterized in that, The acquisition of change information from the business database includes: The target database tables in the data warehouse are examined to determine the data collection results; In response to the data collection results not meeting expectations, change information in the business database is obtained.

9. The method according to claim 1, characterized in that, The method further includes: In response to the adjustment status being pending, a change warning message is sent to relevant personnel via a preset communication interface.

10. A data processing system, characterized in that, The system includes: A sensing unit is used to acquire change information of the business database, the change information being used to indicate that the source database table in the business database has been changed; A response unit is configured to determine the adjustment status of the data warehouse based on the change information and the mapping relationship between the source database table in the business database and the target database table in the data warehouse; and in response to the adjustment status being pending adjustment, to adjust the metadata of the data warehouse and / or the data acquisition task of the data warehouse based on the change information.

11. The system according to claim 10, characterized in that, The system also includes: The early warning unit is used to send change warning information to relevant personnel based on a preset communication interface in response to the adjustment status being pending adjustment.

12. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1-9.

13. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images