Data processing method and device, server and medium

Abstract

The invention provides a data processing method and device, a server and a medium. In response to the initialization instruction for the data warehouse, the offline data is acquired from the business system associated with the data warehouse, and the data structure information of the data warehouse is acquired based on the offline data, so that when the real-time data from the business system is received, the data structure information of the data warehouse is acquired. And obtaining structure updating information based on the real-time data and the data structure information, and then updating the obtained data structure information based on the structure updating information, so as to write the real-time data into the data warehouse based on the updated data structure information. Through the scheme, the structure change in the data warehouse can be monitored in time, so that the structure of the data warehouse is updated according to the monitored structure change information, data writing is performed in the data warehouse after the structure is updated, smooth writing of the data can be ensured, and the data storage efficiency is improved. The situation that data writing fails due to inconsistent data structures is avoided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, server and medium. Background Art

[0002] With the advent of the big data era, data warehouses, as an important part of the enterprise informatization process, are increasingly being used. As a data structure specifically used to store, analyze and process data, data warehouses can integrate data from different sources, clean, transform and integrate them, and ultimately present the data to users in a way that is easy to understand and use.

[0003] In the process of building a data warehouse, the Extract-Transform-Load (ETL) process is one of the most critical links. The ETL process involves the data transmission process. When transmitting data, it usually passes through multiple landing points (that is, data nodes). The structure of each data node is not static. Once the structure of a data node changes, it may cause data transmission failure. Summary of the invention

[0004] The present application provides a data processing method, device, server and medium.

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, the method comprising:

[0006] In response to an initialization instruction for the data warehouse, acquiring offline data from a business system associated with the data warehouse;

[0007] Obtain data structure information of the data warehouse based on offline data;

[0008] In response to receiving the real-time data from the business system, acquiring structure update information based on the real-time data and the data structure information;

[0009] Updating the data structure information based on the structure update information;

[0010] Write real-time data into the data warehouse based on the updated data structure information.

[0011] According to a second aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0012] An acquisition module, for acquiring offline data from a business system associated with the data warehouse in response to an initialization instruction for the data warehouse;

[0013] The acquisition module is also used to obtain data structure information of the data warehouse based on offline data;

[0014] The acquisition module is further used to acquire structure update information based on the real-time data and the data structure information in response to receiving the real-time data from the business system;

[0015] An updating module, used for updating the data structure information based on the structure update information;

[0016] The data writing module is used to write real-time data into the data warehouse based on the updated data structure information.

[0017] According to a third aspect of an embodiment of the present application, a server is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the operations performed by the above-mentioned data processing method when executing the computer program.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a program is stored, and the program is executed by a processor to perform the operations performed by the above-mentioned data processing method.

[0019] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the operations performed by the above-mentioned data processing method when the computer program is executed by a processor.

[0020] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0021] It can monitor the structural changes in the data warehouse in a timely manner, and update the structure of the data warehouse according to the monitored structural change information, so as to write data into the data warehouse after the structural update, and ensure that the data can be written smoothly, so as to avoid data writing failure due to inconsistent data structure.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0024] Figure 1 It is a schematic diagram of an application scenario of a data processing method shown in the present application according to an exemplary embodiment.

[0025] Figure 2 It is a schematic diagram of an implementation environment of a data processing method according to an exemplary embodiment of the present application.

[0026] Figure 3 It is a flow chart of a data processing method shown in the present application according to an exemplary embodiment.

[0027] Figure 4 It is a flowchart of a data processing method according to an exemplary embodiment of the present application.

[0028] Figure 5 It is a schematic diagram of an interface of a library table query interface shown in the present application according to an exemplary embodiment.

[0029] Figure 6 It is a system architecture diagram of a data management system shown in the present application according to an exemplary embodiment.

[0030] Figure 7 It is a block diagram of a data processing device according to an exemplary embodiment of the present application.

[0031] Figure 8 It is a structural diagram of a server shown in the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as described in detail in the present application.

[0033] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] When building the real-time data transmission link and offline data transmission link of the data warehouse, some open source components are used. The use of these open source components will generate some metadata (such as technical metadata). However, the relevant technology cannot effectively manage these metadata, which makes it impossible to effectively monitor and manage the data structure. There will be multiple data nodes in the data transmission link. In the relevant technology, once the data structure of a data node changes, the data structure to be written will be inconsistent with the actual data structure, resulting in data writing failure.

[0036] In view of this, the present application hopes to provide a data processing method to monitor the structural changes of each data system node in the data warehouse in real time, so as to promptly discover the structural changes in the data warehouse, so that the structure of the data warehouse can be updated in time to ensure the real-time and validity of the data structure in the data warehouse, and provide strong guarantees for the smooth writing of data.

[0037] See also Figure 1 , Figure 1 is a schematic diagram of an application scenario of a data processing method according to an exemplary embodiment of the present application. Figure 1 As shown, the application scenario may include a server and multiple business systems (denoted as business system 1, business system 2, ..., business system n).

[0038] The server may be one server, multiple servers, a server cluster, a cloud computing platform, etc. This application does not limit the device type of the server. Figure 1 The description is made by taking only one server as an example, which does not constitute a limitation on the device type of the server.

[0039] Optionally, the business system may be a marketing planning system, a sales system, a customer management system, a risk prevention system, etc. This application does not limit the specific type of the business system.

[0040] The server and each business system can communicate through a wired or wireless connection. Each business system can serve as a data source (or source application), and each business system will generate a large amount of data during operation. The data generated by each business system can be transmitted to the server through a pre-established connection, so that the server can manage the received data through the data processing method provided in this application.

[0041] Above Figure 1 This is only an exemplary description of the application scenario of the present application. In more possible implementations, the data processing method provided by the present application can also be used in other scenarios that require data management. The present application does not limit the specific application scenario.

[0042] After introducing the application scenario of the present application, the implementation environment of the data processing method provided by the present application is introduced with an exemplary description as follows.

[0043] See also Figure 2 , Figure 2 is a schematic diagram of an implementation environment of a data processing method according to an exemplary embodiment of the present application, such as Figure 2 As shown, the implementation environment may include a server and multiple business systems.

[0044] The server may be one server, multiple servers, a server cluster, a cloud computing platform, etc. Figure 2 The example of multiple servers is used for illustration only, and does not limit the device type of the server. The server can be associated with n business systems, and the data generated by the n business systems during operation can all be transmitted to the server, so that the data generated by the n business systems can be managed by the server.

[0045] Optionally, a data warehouse may be deployed in the server, and the data warehouse may include multiple data tables, so as to manage the data generated in the business system associated with the data warehouse through the multiple data tables.

[0046] It should be noted that data warehouses can adopt distributed storage architecture or single-node storage architecture. Since the amount of data stored in data warehouses is large, data warehouses often adopt distributed storage architecture.

[0047] When the data warehouse adopts a distributed storage architecture, the data warehouse can manage the data generated in the business system associated with the data warehouse through data tables specially stored in the distributed storage system. For example, the data warehouse may include multiple Hive tables, for example, the data warehouse may include Hive table 1, Hive table 2, ..., Hive table m, so that the data generated by n business systems can be managed through these m Hive tables.

[0048] After introducing the implementation environment of the present application, the data processing method provided by the present application is described in detail below.

[0049] See also Figure 3 , Figure 3 is a flow chart of a data processing method according to an exemplary embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0050] Step 301: In response to an initialization instruction for a data warehouse, offline data is obtained from a business system associated with the data warehouse.

[0051] Optionally, the initialization instruction for the data warehouse may be automatically triggered when the data warehouse is started for the first time, or may be automatically triggered when the data warehouse is restarted after being reset, etc. The present application does not limit the triggering timing of the initialization instruction.

[0052] After obtaining the initialization instruction for the data warehouse, offline data can be pulled from the business system associated with the data warehouse at one time.

[0053] It should be noted that a data warehouse can be associated with multiple business systems. After obtaining the initialization instruction for the data warehouse, the generated business data can be pulled from each business system associated with the data warehouse to obtain offline data from multiple business systems at one time.

[0054] Step 302: Acquire data structure information of the data warehouse based on offline data.

[0055] The data structure information may be used to indicate the table structure of each data table in the data warehouse and the data lineage relationship in the data warehouse. For example, the data structure information may include data file description information and data table structure information, which may indicate the table structure of each data table in the data warehouse, wherein the data file description information may indicate how to describe the elements in the data file, and the data table structure information may be used to describe the table structure of each data table in the data warehouse. The data lineage relationship in the data warehouse may be the lineage relationship between each data table in the data warehouse.

[0056] Optionally, the data file may be an Extensible Markup Language (XML) file, and accordingly, the data file description information may be Schema information, and the Schema information may indicate how to describe elements in the XML file.

[0057] It should be noted that data lineage relationships can be used to indicate upstream and downstream dependencies between data sets and between computing tasks. Correspondingly, the lineage relationships between data tables can be used to indicate the dependencies between data tables. That is, the lineage relationships between data tables can be used to indicate the upstream and downstream data tables corresponding to each data table.

[0058] Step 303: In response to receiving the real-time data from the business system, obtain structure update information based on the real-time data and the data structure information.

[0059] The real-time data may be streaming data. Streaming data is an unbounded data sequence that is constantly flowing and changing. Streaming data flows with the passage of time.

[0060] Step 304: Update the data structure information based on the structure update information.

[0061] Step 305: Write the real-time data into the data warehouse based on the updated data structure information.

[0062] Through the solution provided in the present application, structural changes in the data warehouse can be monitored in a timely manner, so that the structure of the data warehouse can be updated according to the monitored structural change information, so as to write data in the data warehouse after the structural update, thereby ensuring that the data can be written smoothly, so as to avoid data writing failure due to inconsistent data structure.

[0063] After introducing the basic implementation process of the present application, various non-limiting implementation methods of the present application are specifically introduced below.

[0064] It should be noted that the data generated in the business system will not be of only one type. In order to distinguish different types of data, different business fields can be set to classify and store the generated data according to the business fields. Among them, the business field can be used to indicate the data type of the corresponding data. For example, the business field can include output value, transaction time, customer name, etc.

[0065] Based on this, when the business system transmits offline data to the server, it can construct the data structure description information of the offline data according to the business fields corresponding to the offline data, and use the offline data as the data part to be written, so as to send the offline data and the data structure description information of the offline data to the server together.

[0066] In some embodiments, for step 302, when acquiring the data structure information of the data warehouse based on the offline data, the data structure description information of the offline data may be acquired, thereby acquiring the data structure information of the data warehouse based on the acquired data structure description information.

[0067] The data structure description information of offline data may include data file description information and data table structure information, which may indicate the table structure of each data table in the data warehouse. Optionally, the data file may be an Extensible Markup Language (XML) file, and accordingly, the data file description information may be Schema information, which may indicate how to describe elements in the XML file; the data table may be a Hive table, and accordingly, the data table structure information may be Hive table structure information.

[0068] Optionally, data structure description information (including data file description information and data table structure information) can be expressed in the form of metadata, wherein metadata can be data used to describe data, and metadata can describe data from three aspects: entity, entity attribute, and relationship between entities. Therefore, when obtaining data structure information of the data warehouse based on the data structure description information of offline data, the data structure description information can be converted into metadata, and the converted metadata can be used as the data structure information of the data warehouse.

[0069] Optionally, after the data structure information is acquired through the above embodiment, the stored data structure information may be stored so that it can be updated later based on the stored data structure information.

[0070] The data structure information may include data file description information and data table structure information. When storing the data structure information, the data file description information and the data table structure information may be stored separately. For example, the data file description information may be stored in the metadata service provider layer (Schema Registry), and the data table structure information included in the data structure description information may be stored in the metastore component to implement the storage of the data structure information.

[0071] Taking the data table as a Hive table as an example, after obtaining the offline data, the obtained offline data can be submitted to the Yarn task, so that the resource management scheduling and task allocation of the offline data writing task can be performed through the Yarn task, so as to write the offline data to the Hive component of the data warehouse, thereby achieving the purpose of writing the offline data into the data warehouse that realizes data management through the Hive table.

[0072] After obtaining the data structure information through the above process, the structure changes in the data link can be monitored in real time to update the acquired data structure information in a timely manner. Optionally, the monitoring of the structure changes in the data link can be achieved by monitoring the operation of changing the structured query language (SQL) in the archive log (Binlog).

[0073] It should be noted that the update of the data structure mainly includes the change of data file description information (that is, the Schema Registry change) and the change of data table structure information. The following introduces these two change functions respectively.

[0074] (1) Changes in data file description information: including information update and deletion.

[0075] The real-time data transmission party in the synchronous link can change the data file description information. The data file description information change includes structure addition and structure change (transmission field reduction, transmission field renaming, etc.). The data transmission party can organize the change requirements according to the actual technical requirements, and then submit the organized change points to the data warehouse operation and maintenance personnel.

[0076] Because deletion and change operations are high-risk operations, data warehouse personnel are required to evaluate the data before performing change operations to assess the safety of the update operation. If it is determined that the change operation is not a high-risk operation, a change request can be sent to the operation and maintenance personnel. The operation and maintenance personnel can perform operations on the server based on the change request to trigger real-time data for changing the data file description information.

[0077] (2) Changes to data table structure information: including information updates and deletions.

[0078] The data transmission party or data warehouse personnel in the synchronization link can change the data table structure information. The data table structure information change includes structure addition and structure change (data table field reduction, data table field renaming). The data transmission party or data warehouse personnel can organize the change requirements according to the actual technical requirements, and then submit the organized change points to the data warehouse operation and maintenance personnel.

[0079] Because data table structure deletion and modification operations are high-risk operations, data warehouse personnel are required to evaluate the data before performing the modification operation to assess the safety of the update operation. If it is determined that the modification operation is not a high-risk operation, a change request can be sent to the operation and maintenance personnel. The operation and maintenance personnel can perform operations on the server based on the change request to trigger real-time data for changing the data table structure information.

[0080] After the real-time data is triggered, the triggered real-time data can be sent to the server in the form of streaming data, and the server can obtain the real-time data.

[0081] Optionally, when the business system sends real-time data to the server, it can construct data structure description information of the real-time data according to the business fields corresponding to the real-time data, and use the real-time data as the data to be written. The data structure description information and the data to be written can constitute the real-time data that needs to be transmitted, so that the business system can send real-time data to the server.

[0082] After receiving the real-time data, the server may, in step 303 , obtain structure update information based on the real-time data and the data structure information in response to receiving the real-time data from the business system.

[0083] It should be noted that the data warehouse may be associated with more than one business system. When real-time data is received from any business system associated with the data warehouse, structural update information can be obtained based on the received real-time data and the acquired data structure information.

[0084] In some embodiments, when acquiring structure update information based on real-time data and data structure information, data structure description information may be acquired based on the real-time data to acquire structure update information based on the data structure description information and the data structure information.

[0085] Optionally, when acquiring data structure description information based on real-time data, the data structure description information included in the received real-time data may be acquired to achieve the purpose of acquiring data structure description information based on real-time data.

[0086] The data structure description information may include data file description information and data table structure information. Optionally, the data file may be an XML file, and the data file description information may be Schema information.

[0087] In one possible implementation, after obtaining the data structure description information, the server may store the data file description information included in the data structure description information in the Schema Registry, and store the data table structure information included in the data structure description information in the Metastore, so that the stored data file description information and data table structure information may be directly obtained when needed in the future.

[0088] It should be noted that when storing the data file description information in the Schema Registry, an incremental storage method can be used to store both the data file description information of the unmodified part and the modified data file description information corresponding to the modified part in the Schema Registry, so that the complete data file description information can be obtained from the Schema Registry.

[0089] In addition, when storing the data table structure information in the Metastore, a direct storage method can be used, that is, there is no need to modify or overwrite the stored content, but directly store the newly acquired data table structure information in the Metastore. Therefore, multiple data table structure information can be stored in the Metastore, and different version numbers can be set for different data table structure information. The later the data table structure information is stored, the larger the version number value is. Therefore, based on the version number value of the stored data table structure information, the latest version of the database structure information can be determined as the data table structure information currently used by the data warehouse.

[0090] Optionally, the data structure description information may be obtained periodically, so as to update the data structure of the data warehouse periodically based on the obtained data structure description information.

[0091] In a possible implementation, the data structure description information may be obtained once every first preset time period, so that the data structure may be updated once every first preset time period. That is, the stored data file description information may be obtained from the metadata service provider layer and the stored data table structure information may be obtained from the metadata storage component as the data structure description information every first preset time period, so that the structure update information may be obtained based on the data structure description information and the data structure information, so that the data structure of the data warehouse may be updated according to the structure update information.

[0092] The first preset time length may be any time length, for example, the first preset time length may be half an hour, or the first preset time length may be other time lengths. The present application does not limit the specific value of the first preset time length.

[0093] The above embodiment is explained by changing the data structure by triggering the real-time data used to change the data structure through the change operation. In more possible implementations, the data structure can also be updated once by manually entering metadata (such as real-time Kafka message body structure data, Hive table structure data, source data table structure data) by relevant technicians. It should be noted that no matter which method is used, it is necessary to undergo a security assessment by the data warehouse personnel, and the change process can be started after the assessment is passed.

[0094] Optionally, after obtaining the data structure description information, the obtained data structure description information can be stored in a distributed data storage (such as Redis). Then, when obtaining structure update information based on the data structure description information and the data structure information, the structure update information can be obtained based on the data structure information and the data structure description information stored in the distributed data storage.

[0095] In a possible implementation, the data structure description information and the data structure information may be compared. If the data structure description information is consistent with the data structure information, there is no need to obtain the structure update information. Otherwise, the structure update information is obtained based on the data structure description information and the data structure information.

[0096] For example, the difference information between the data structure information and the data structure description information may be determined, and the determined difference information may be used as the structure update information.

[0097] To facilitate understanding, the process of obtaining structure update information is introduced below with an example. If the data structure information indicates that data table A includes business field 1, business field 2 and business field 3, and the data structure description information indicates that data table A includes business field 1, business field 2, business field 3 and business field 4, then the difference information between the data structure information and the data structure description information can be determined. The determined difference information is used to indicate that the data table A corresponding to the data structure information does not include business field 4, while the data table A corresponding to the data structure description information includes business field 4, so that the structure update information can be determined to be to add business field 4 to data table A.

[0098] The above embodiment mainly introduces the process of obtaining structure update information. In more possible implementations, the blood relationship between multiple data tables in the data warehouse can also be obtained based on offline data.

[0099] Optionally, the blood relationship between multiple data tables in the data warehouse can be determined based on the business fields corresponding to the offline data and the conversion operations (such as filtering, calculation, connection, etc.) in the offline data generation process. For example, the correlation between business fields can be inferred by viewing the business field name, content, type and format, and combined with the conversion operations in the offline data generation process, the generation dependency between data can be determined, and the correlation between business fields can be further predicted, so as to determine the blood relationship between data tables based on the correlation between business fields.

[0100] Among them, for the generation dependency relationship between data, taking data A generated based on data B and data C as an example, it can be determined that there is a generation dependency relationship between data A and data B, and between data A and data C, specifically, data A is generated based on data B, and data A is generated based on data C.

[0101] Optionally, after the blood relationship between the data tables is obtained, the obtained blood relationship can also be stored. For example, the blood relationship can be stored in a distributed data storage.

[0102] After the blood relationship is stored, the stored blood relationship can be updated regularly. In a possible implementation, the data currently stored in the data warehouse can be obtained every second preset time period; the current blood relationship of the data warehouse is obtained based on the data currently stored in the data warehouse, and the stored blood relationship is updated based on the current blood relationship of the data warehouse.

[0103] For example, the source data source, database, data table, transmission field (that is, business field) in the real-time transmission link, and data table structure information used in the data link corresponding to the data warehouse can be obtained every second preset time period to establish a new blood relationship.

[0104] It should be noted that, since blood relationships do not change frequently, the value of the second preset time period can be larger. For example, the second preset time period can be one day, that is, the blood relationship can be calculated once a day and the blood relationship can be summarized and stored in the distributed data storage.

[0105] By updating the blood relationship regularly, the accuracy of the stored blood relationship is guaranteed, thereby ensuring the accuracy of the data processing process.

[0106] In some embodiments, when the stored data structure information is updated based on the structure update information, the stored data structure information may be updated based on the structure update information and the blood relationship.

[0107] In a possible implementation, when the stored data structure information is updated based on the structure update information and the blood relationship, it can be implemented through the following steps:

[0108] Step 1: detecting the structure update information based on the preset structure change indication information to detect whether the structure update information satisfies the preset structure change indication information.

[0109] It should be noted that the structure change indication information may be a pre-set structure change rule, and the structure change rule may be set according to actual technical requirements. For example, the structure change rule may include not allowing changes to library table names, not allowing changes to field names, not allowing changes to field types, etc., to lock changes to some frequently used tables, implement change restrictions on frequently used tables, and control the scope of business impact.

[0110] Optionally, the structural change rules may be updated according to actual technical requirements, such as adding, modifying or deleting the structural change rules.

[0111] After the change process is triggered, the pre-set structure change indication information can be queried to determine whether the structure update information meets the pre-set structure change indication information. If so, it is determined that the structure update information has passed the detection and can continue to be processed. Otherwise, the change process is terminated.

[0112] Step 2: When the structure update information passes the detection, based on the blood relationship, a review request is sent to the upstream and downstream data table managers corresponding to the data table indicated by the structure update information.

[0113] Optionally, when the structure update information passes the detection, the upstream and downstream data tables corresponding to the data table indicated by the structure update information may be determined based on the blood relationship, so that a review request may be sent to the managers of the upstream and downstream data tables.

[0114] Among them, the upstream and downstream data table managers can review the structure change information after receiving the review request to determine the legitimacy of the structure change information. If they agree to change the data structure based on the structure change information, they can return positive feedback; if they disagree to change the data structure based on the structure change information, they can return negative feedback.

[0115] Step 3: After receiving positive feedback for the review request, the stored data structure information is updated based on the structure update information.

[0116] Optionally, when the change operation is a data file description information change operation, SchemaRegistry can be called to delete the corresponding Schema information from SchemaRegistry. In the case where the data table structure information is Hive table structure information, if the change operation is for changing the data table structure information, the Hive component can be called to update the structure of the Hive table that needs to be changed.

[0117] By conducting upstream and downstream association audits in the operation chain, the scope of business impact can be reduced, achieving the effect of one-time operation and evaluation for all parties, reducing communication and confirmation costs. In addition, the safety judgment of change operations can be implemented based on structural change indication information, which can reduce data risks caused by human operation omissions.

[0118] The above embodiment mainly introduces the process of updating the data structure of the data warehouse. In addition, data can be written after the data structure change is completed to achieve data synchronization between the business system and the data warehouse.

[0119] In some embodiments, the data to be written can be obtained based on real-time data, and the data to be written can be stored in the target message queue; thus, the data to be written can be obtained from the target message queue, and the data to be written can be written into the data warehouse based on the updated data structure information. The target message queue can be a Kafka queue, and by storing the data to be written in the Kafka queue, the data to be written can be directly obtained from the Kafka queue in batches when the Spark Streaming task processes the streaming data. Optionally, the data to be written can be Avro message body data.

[0120] It should be noted that when writing the data to be written into the data warehouse based on the updated data structure information, the business field corresponding to the data to be written can be determined based on the data structure description information included in the real-time data. Since the data structure information of the data warehouse has been updated in advance based on the data structure description information, it can be ensured that the current data structure information of the data warehouse is consistent with the data structure to be written. Therefore, according to the business field corresponding to the data to be written, the data to be written can be written to the storage location corresponding to the corresponding business field.

[0121] It should be noted that, since the data structure update process of the data warehouse can be performed periodically, during the data writing process, in order to ensure data consistency, a data consistency check can be performed on the data to be written before the data is written.

[0122] In a possible implementation, after obtaining the Avro message body data, the Spark Streaming task can first obtain the data description structure information from the Schema Registry to perform data consistency verification on the Avro message body data based on the obtained message data. After the verification passes, Hive data is assembled based on the Avro message body data to write the Hive data into the Hive table.

[0123] Through the data processing method provided by the present application, the structural change process and the data writing process can be decoupled, and the maintenance of the data structure can be achieved outside the data synchronization link, so that the data structure can be changed at the metadata level based on real-time data to ensure that the data structure in the data warehouse is already the latest when the data is synchronized later, and the structure is consistent during data synchronization to avoid the interruption of the transmission link due to structural inconsistency, which leads to data writing failure, and ensure the smooth progress of the data writing process and the smooth storage of data. There is no need to change the data structure during the data writing process, but directly perform data consistency verification on the data to be written to further prevent data writing failure and improve the success rate of the data writing process.

[0124] See also Figure 4 , Figure 4 is a flowchart of a data processing method according to an exemplary embodiment of the present application, such as Figure 4As shown, the metadata of the data warehouse can be managed by the data processing method provided by the present application. That is, data structure information can be obtained based on the offline data provided by the source data source, and the offline data can be written to the data table through the Yarn task. In addition, real-time data can be obtained from the source application, and the data structure description information corresponding to the real-time data can be stored in the Schema Registry, so that the metadata of the data warehouse can be managed based on the data structure description information stored in the Schema Registry; optionally, the acquired real-time data can also be stored in the Kafka queue, so that the real-time data in the Kafka queue can be processed by Spark Streaming to process streaming data, so as to write the data to the data table.

[0125] In more possible implementations, the data processing method provided in the present application can also provide users with a library table query function.

[0126] In some embodiments, after obtaining the data relationship between each data table in the data warehouse, in response to receiving a target query instruction, the upstream and downstream data tables corresponding to the target data table indicated by the target query instruction can be obtained based on the blood relationship; and the table information of the upstream and downstream data tables can be displayed.

[0127] Optionally, a library table query interface may be provided, and the library table query interface may include an input control for inputting the identifier of the target data table to be queried through the input control, thereby triggering a target query instruction based on the input identifier, and the server may respond to the target query instruction and obtain the upstream and downstream data tables corresponding to the target data table indicated by the target query instruction based on the blood relationship, so as to display the table information of the upstream and downstream data tables. Among them, the table information of the upstream and downstream data tables may include the upstream and downstream data table managers.

[0128] See also Figure 5 , Figure 5 is a schematic diagram of a library table query interface according to an exemplary embodiment of the present application. Figure 5 As shown in the figure, if the user enters "tableA" in the input control, the target query request can be triggered by triggering the button labeled "Library Table Query". After obtaining the query results, the query can be performed as follows Figure 5 The library table query interface shown in the figure displays the table information of the upstream and downstream Hive tables corresponding to tableA.

[0129] By collecting the blood relationship of upstream and downstream data tables and providing library table query function, the upstream and downstream table structures of the data table to be queried can be quickly matched. In addition, by providing a visual interface, based on the blood relationship of the collected upstream and downstream table metadata of the link, the relationship between the upstream and downstream tables can be quickly displayed in the visual interface, which is convenient for user query, realizes interface management, and reduces manual operation.

[0130] It should be noted that the above data processing method can be implemented by a metadata management system deployed in a server, see Figure 6 , Figure 6 is a system architecture diagram of a data management system according to an exemplary embodiment of the present application. Figure 6 As shown, the metadata management system can mainly provide three functions:

[0131] (1) Metadata collection service: Regularly monitors structural changes during data transmission (including Hive table structure information monitoring, real-time message monitoring in Schema Registry, and source data structure information monitoring), and provides the function of building bloodline relationships.

[0132] (2) Library and table query function: The source data source, database, data table, transmission field in the real-time link, and data table structure information (such as Hive table structure information) used in the data link are used as the query basis, and the query results are obtained through the established blood relationship. The query results can be the latest source table structure, real-time transmission message structure, warehouse data table structure (such as warehouse Hive table structure), etc.

[0133] (3) Structural change function: including Schema Registry changes, data table structure changes (such as Hive structure changes), change rule management, and manual entry of metadata.

[0134] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of the device and the server to which it is applied.

[0135] like Figure 7 As shown, Figure 7 is a block diagram of a data processing device according to an exemplary embodiment of the present application, the device comprising:

[0136] The acquisition module 701 is used to acquire offline data from the business system associated with the data warehouse in response to the initialization instruction of the data warehouse;

[0137] The acquisition module 701 is also used to acquire data structure information of the data warehouse based on offline data;

[0138] The acquisition module 701 is further used to acquire structure update information based on the real-time data and the data structure information in response to receiving the real-time data from the business system;

[0139] An updating module 702, configured to update the data structure information based on the structure update information;

[0140] The data writing module 703 is used to write the real-time data into the data warehouse based on the updated data structure information.

[0141] In some embodiments, the real-time data includes data structure description information;

[0142] The acquisition module 701, when used to acquire structure update information based on real-time data and data structure information, is used to:

[0143] If the data structure description information is inconsistent with the data structure information, the structure update information is obtained based on the data structure description information and the data structure information.

[0144] In some embodiments, the data structure description information includes data file description information and data table structure information;

[0145] The device also includes:

[0146] The storage module is used to store data file description information in the metadata service provider layer and store database Hive structure information in the meta-storage component.

[0147] In some embodiments, the acquisition module 701 is further used to acquire the blood relationship between multiple component tables in the data warehouse based on offline data;

[0148] The updating module 702, when used to update the data structure information based on the structure update information, is used to:

[0149] Based on the structure update information and blood relationship, the data structure information is updated.

[0150] In some embodiments, the update module 702, when used to update the data structure information based on the structure update information and the blood relationship, is used to:

[0151] Performing information detection on the structure update information based on the preset structure change indication information, wherein the information detection is used to detect whether the structure update information satisfies the preset structure change indication information;

[0152] When the structure update information passes the detection, based on the blood relationship, a review request is sent to the upstream and downstream data table managers corresponding to the data table indicated by the structure update information;

[0153] If positive feedback is received for the review request, the data structure information is updated based on the structure update information.

[0154] In some embodiments, the acquisition module 701 is further used to acquire the current stored data in the data warehouse;

[0155] The acquisition module 701 is also used to acquire the current blood relationship of the data warehouse based on the previously stored data;

[0156] The updating module 702 is also used to update the blood relationship based on the current blood relationship of the data warehouse.

[0157] In some embodiments, the real-time data also includes data to be written;

[0158] The data writing module 703 is further used to store the data to be written into the target message queue in response to receiving the real-time data from the business system;

[0159] The data writing module, when used to write real-time data into the data warehouse based on the updated data structure information, is used to: obtain the data to be written from the target message queue, and write the data to be written into the data warehouse based on the updated data structure information.

[0160] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0161] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0162] This application also provides a server, see Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a server structure according to an exemplary embodiment of the present application. Figure 8As shown, the server includes a processor 810, a memory 820 and a network interface 830. The memory 820 is used to store computer instructions that can be run on the processor 810. The processor 810 is used to implement the data processing method provided in any embodiment of the present application when executing the computer instructions. The network interface 830 is used to implement input and output functions. In more possible implementations, the server may also include other hardware, which is not limited in the present application.

[0163] The present application also provides a computer-readable storage medium, which can be in various forms. For example, in different examples, the computer-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. In particular, the computer-readable medium can also be paper or other suitable medium capable of printing programs. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the data processing method provided in any embodiment of the present application is implemented.

[0164] The present application also provides a computer program product, including a computer program, which implements the data processing method provided in any embodiment of the present application when the computer program is executed by a processor.

[0165] It should be understood by those skilled in the art that one or more embodiments of the present application may be provided as a method, device, server, computer-readable storage medium or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiment corresponding to the server, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0169] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0170] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0171] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0172] Although the application includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described above and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0173] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0174] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the present application. In some cases, the actions described in the present application can be performed in a different order and still achieve the desired results. In addition, the processing depicted in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0175] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. That is, the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

[0176] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to an initialization instruction for a data warehouse, acquiring offline data from a business system associated with the data warehouse; Acquire data structure information of the data warehouse based on the offline data; In response to receiving the real-time data from the business system, acquiring structure update information based on the real-time data and the data structure information; Updating the data structure information based on the structure update information; The real-time data is written into the data warehouse based on the updated data structure information.

2. The method according to claim 1, characterized in that The real-time data includes data structure description information; and the acquiring structure update information based on the real-time data and the data structure information includes: If the data structure description information is inconsistent with the data structure information, the structure update information is acquired based on the data structure description information and the data structure information.

3. The method according to claim 2, characterized in that The data structure description information includes data file description information and data table structure information; the method further includes: The data file description information is stored in the metadata service providing layer, and the data table structure information is stored in the metadata storage component.

4. The method according to claim 2, characterized in that: The method further comprises: Acquire the blood relationship between multiple component tables in the data warehouse based on the offline data; The updating of the data structure information based on the structure update information includes: updating the data structure information based on the structure update information and the blood relationship.

5. The method according to claim 4, characterized in that The updating of the data structure information based on the structure update information and the blood relationship includes: Performing information detection on the structure update information based on preset structure change indication information, wherein the information detection is used to detect whether the structure update information satisfies the preset structure change indication information; If the structure update information satisfies the structure change indication information, sending a review request to the upstream and downstream data table managers corresponding to the data table indicated by the structure update information; If a positive feedback is received for the audit request, the data structure information is updated based on the structure update information.

6. The method according to claim 4, characterized in that The method further comprises: Acquire the currently stored data in the data warehouse; Acquiring a current blood relationship of the data warehouse based on the current stored data; The blood relationship is updated based on the current blood relationship.

7. The method according to claim 2, characterized in that: The real-time data includes data to be written; and the method further includes: In response to receiving the real-time data from the business system, storing the data to be written into a target message queue; The step of writing the real-time data into the data warehouse based on the updated data structure information includes: The data to be written is obtained from the target message queue, and based on the updated data structure information, the data to be written is written into the data warehouse.

8. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire offline data from a business system associated with the data warehouse in response to an initialization instruction for the data warehouse; The acquisition module is further used to acquire data structure information of the data warehouse based on the offline data; The acquisition module is further configured to, in response to receiving the real-time data from the business system, acquire structure update information based on the real-time data and the data structure information; An updating module, used for updating the data structure information based on the structure update information; A data writing module is used to write the real-time data into the data warehouse based on the updated data structure information.

9. A server, characterized in that: The server includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the operations performed by the data processing method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program is used by a processor to execute the operations performed by the data processing method according to any one of claims 1 to 7.

Images