Data management method, device, server and storage medium

Abstract

The present invention provides a data management method, device, server and storage medium, the method comprising: determining the importance level of a data table of a data application according to the level of the data application; managing the data processing process of the data table according to the importance level of the data table. The present invention can reduce the workload of data operation and maintenance and improve data accuracy and timeliness.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data management method, device, server and storage medium. Background Art

[0002] With the development of big data technology, data assets are increasing, occupying a large amount of storage and computing resources.

[0003] In the vast data assets, the data either has no level or is randomly selected by users, and the accuracy of the data level cannot be guaranteed. At the same time, the management of the data processing process is also based on human configuration.

[0004] Data management based on manual configuration requires a large workload for operation and maintenance and is prone to errors. The accuracy and timeliness of the data cannot be guaranteed. Summary of the invention

[0005] The present invention provides a data management method, device, server and storage medium to reduce the workload of data operation and maintenance and improve the accuracy and timeliness of data.

[0006] In a first aspect, the present invention provides a data management method, which is applicable to a big data platform, and comprises:

[0007] Determining the importance level of the data table of the data application according to the level of the data application;

[0008] The data processing process of the data table is managed according to the importance level of the data table.

[0009] In a second aspect, the present invention provides a data management device, which is applicable to a big data platform, and includes:

[0010] A determination module, used to determine the importance level of the data table of the data application according to the level of the data application;

[0011] The management module is used to manage the data processing process of the data table according to the importance level of the data table.

[0012] In a third aspect, the present invention further provides a server, comprising: a memory and a processor; the memory is connected to the processor;

[0013] The memory is used to store program instructions;

[0014] The processor is used to implement the data management method shown in the first aspect above when the program instructions are executed.

[0015] In a fourth aspect, the present invention may also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data management method of the first aspect.

[0016] The present invention provides a data management method, device, server and storage medium, which can determine the importance level of a data table of the data application according to the level of the data application, and manage the data processing process of the data table according to the importance level of the data table. In this method, the importance level of the data table of the data application is determined based on the level of the data application, which can ensure the accuracy of the data level, and manage the data processing process of the data table based on the importance level, without the need for manual configuration, reducing the workload of data operation and maintenance, and ensuring the accuracy and timeliness of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a data system provided by an embodiment of the present invention Figure 1 ;

[0019] Figure 2 A schematic diagram of a data system provided by an embodiment of the present invention Figure 2 ;

[0020] Figure 3 The process of the data management method provided by the embodiment of the present invention Figure 1 ;

[0021] Figure 4 The process of the data management method provided by the embodiment of the present invention Figure 2 ;

[0022] Figure 5 A schematic diagram of determining the importance level of a data table in a data management method provided by an embodiment of the present invention;

[0023] Figure 6 The process of the data management method provided by the embodiment of the present invention Figure 3 ;

[0024] Figure 7 The process of the data management method provided by the embodiment of the present invention Figure 4 ;

[0025] Figure 8A schematic diagram of the structure of a data management device provided by an embodiment of the present invention;

[0026] Fig. 9 A schematic diagram of the structure of a server provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", and "third" in the various parts of the embodiments of the present invention and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0029] The method flow charts involved in the following embodiments of the present invention are only exemplary and do not necessarily include all the contents and steps, nor do they have to be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined. Therefore, the actual execution order can be changed according to the actual situation.

[0030] The functional modules in the block diagrams involved in the following embodiments of the present invention are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processors and / or microcontrollers.

[0031] The data system to which the embodiment of the present invention is applicable is first described as follows. Figure 1 A schematic diagram of a data system provided by an embodiment of the present invention Figure 1 .like Figure 1 As shown, the data system provided by the embodiment of the present invention may include: a database system 11, a big data platform 12 and a data application management system 13.

[0032] The database system 11 can be a column storage database such as a database management system, a relational database system, a distributed database system (Hadoop Database, referred to as HBase), etc. The database system 11 can also be called a database service platform, which provides database support for the business system. It can have a database of at least one business system, and each business system database can store the data of the business system. The database system 11 can be used as an access data source and a push data source of a big data platform. The database system 11 can be implemented on one or more servers.

[0033] After processing the data from the database system, the big data platform 12 can transmit the processed data to the data application system 13 so that the data application in the data application system 13 can use it. The big data platform 12 can process the data from the business system in the database system 11 using an Extract-Transform-Load (ETL) tool, also known as a data through train, to load the data into the data warehouse, and push the data from the data warehouse to the data mart; or, after the ETL tool processes the processed data, it directly pushes the data from the data warehouse to the data mart. In the big data platform 12, the data can be pushed to the data application in the data application system 13 after being calculated and processed in the data warehouse and the data mart, or pushed to the database of the data application in the data application system to be used by the data application and serve the data application. The big data platform 13 can be implemented on a server cluster composed of multiple servers.

[0034] The data application management system 13 can manage multiple data applications, for example, a reporting platform, an operation monitoring platform, a business analysis platform, a data compass platform, an external data reporting platform, etc. The data application management system 13 can also be implemented on one or more servers. The multiple data applications managed by the data application management system 13 can have their own levels, and the level of the data application can be the importance level of the data application, which is used to characterize the importance of the data application. The importance of the data application can be determined, for example, based on the business scope served by the data application and / or the importance of the business served, and can also be determined based on the importance of the data generated by the data application. Data applications applied to different industries can have different level division methods.

[0035] For example, in the e-commerce industry, the importance of data applications can be determined based on the importance of the data generated by the data applications. The importance of the data can be, for example, the scope of influence generated by the data. The greater the scope of influence generated by the data, the greater the importance of the data. The smaller the scope of influence generated by the data, the smaller the importance of the data. The importance of data applications determined based on the importance of the data can be any of the following levels:

[0036] L1: Used to indicate that the data generated by the data application involves asset loss and / or public relations risks, etc.

[0037] L2: Used to indicate that the data generated by the data application involves business impact within the enterprise, users, suppliers, etc., business efficiency loss, etc.

[0038] L3: Used to indicate that the data generated by data applications involves the impact of internal enterprise inputs such as data products and operational reports, and its impact is relatively small.

[0039] L4: used to indicate that the impact of the data generated by the data application is in an uncertain state, such as the usage scenario cannot be determined yet.

[0040] Figure 2 A schematic diagram of a data system provided by an embodiment of the present invention Figure 2 .like Figure 2 As shown, in the data system provided by the embodiment of the present invention, the database system 11 is an external system that the big data platform 12 relies on, and it is the access data source and push data source of the big data platform 12.

[0041] Among them, the demand management platform is also an external system that the big data platform 12 relies on. It can be a management system for enterprise needs and projects, and the business needs in the business system are managed and controlled by the demand management platform.

[0042] The data application management system 13 is an external system that the big data platform 12 relies on. It is a management system for data applications and a software platform that integrates the compilation, testing, online release and deployment of data applications.

[0043] The data management platform in the big data platform 12 has a software data management tool, which is a platform for life cycle management and information maintenance of data tables in the big data platform within the enterprise, and provides data table operation functions such as data table creation, modification, deletion, recovery, information maintenance, and viewing of various indicators of data tables.

[0044] The data development platform in the big data platform 12 can obtain the information of the data tables in the big data platform from the data management platform, and based on the obtained data table information, develop, test and publish scripts for scheduling tasks.

[0045] The data source management system in the big data platform 12 is used to manage the data sources used by the big data platform 12 when extracting and / or pushing data, and serves as a bridge between the big data platform 12 and the data sources.

[0046] The cluster management system in the big data platform 12 is a software platform for managing distributed clusters, marts and execution queues, and provides services for querying, creating, modifying and deleting cluster, mart and linear queue information.

[0047] The scheduling center in the big data platform 12 is the execution control center for scheduling tasks such as ETL tasks, computing tasks, and data pushing tasks.

[0048] The data quality monitoring platform of the big data platform 12 is a software platform for monitoring the data quality after executing the scheduling task. It can provide a variety of data quality monitoring indicators and corresponding monitoring strategies, and also has a monitoring and alarm function for data quality.

[0049] The distributed cluster may be a Hadoop cluster, which is a cluster of underlying technologies of the big data platform 12 and can provide storage and computing services for massive data.

[0050] The above-mentioned demand management platform, data management platform, data development platform, data source management system, cluster management system, scheduling center, data quality monitoring platform and distributed cluster can be implemented on one or more servers respectively.

[0051] Combine as follows Figure 2 The data system shown is used to illustrate the data management method provided by the embodiment of the present invention. Figure 3 The process of the data management method provided by the embodiment of the present invention Figure 1 The data management method can be obtained by Figure 1 or Figure 2 The big data platform 12 shown in FIG. Figure 3 As shown, the data management method may include:

[0052] S301. Determine the importance level of a data table of the data application according to the level of the data application.

[0053] In the method, the level of the data application can be determined as the importance level of the data table of the data application. When the importance level of the data table of the data application is determined, the data table of the data application can also be identified in the big data platform to indicate the importance level of the data table. Among them, identifying the data table can include: using the identification corresponding to the importance level to identify the data table in the big data platform, and using the identification corresponding to the importance level to identify the table in the business system that generates the data table, etc.

[0054] The data application may be any data application managed by the data application management system 13. Figure 2 The data management platform in the data management system 12 is implemented. The data application management system 13, as a management system for data applications, may have the level of the data application. After the level of the data application is determined, the data application management system 13 may transmit the level of the data application to the data management platform, so that the data management platform can determine the importance level of the data table of the data application in the big data platform 12 according to the level of the data application. When the data management platform determines the importance level of the data table, it can also use the identifier corresponding to the importance level to identify the data table in the big data platform.

[0055] S302: Manage the data processing process of the data table according to the importance level of the data table.

[0056] The data processing process of the data table may be processing the data of the data table. The data of the data table may be the data of the data table from the data source of the big data platform 12. For the big data platform 12, the above Figure 2 The database system 11 shown in is the data source. The data obtained from the data source refers to the data obtained from the business system and stored in the database system 11 in the database of the business system.

[0057] In the method, the data processing process of the data table can be managed according to the importance level of the data table. Specifically, in the data processing process of the data table, corresponding processing resources can be allocated to the data table according to the importance level, and the data of the data table can be processed according to the processing resources, that is, data processing is performed on the data table. Among them, managing the data processing process of the data table can include at least one of the following operations: allocating processing resources required for the data processing process of the data table, monitoring the structure of the data processing of the data table, changing the structure of the data table, etc.

[0058] Taking the allocation of resources in the data processing process as an example, the higher the importance level, the more processing resources corresponding to the importance level are allocated, and the lower the importance level, the fewer processing resources corresponding to the importance level are allocated. The processing resources may include, for example: storage and / or computing resources.

[0059] S302 can be performed by, for example, Figure 2 The computing and / or storage resources of the distributed cluster are managed by the cluster management system in Figure 2The cluster management system in the cluster manages and coordinates the data table. The cluster management system allocates processing resources to the data table according to the importance level of the data table. The cluster management system can allocate processing resources to the data table when obtaining cluster and market information from the distributed cluster.

[0060] According to the processing resources allocated based on the importance level, the data in the data table is processed, and the processed data is pushed to the data application so that the data application uses the processed data, thereby making the processed data play a role corresponding to its importance level.

[0061] The data management method provided in the embodiment of the present application can determine the importance level of the data table of the data application according to the level of the data application, and manage the data processing process of the data table according to the importance level of the data table. In this method, the importance level of the data table of the data application is determined based on the level of the data application, which can ensure the accuracy of the data level, and manage the data processing process of the data table based on the importance level, without the need for manual configuration, reducing the workload of data operation and maintenance, and ensuring the accuracy and timeliness of the data. In addition, in this method, the data processing process of the data table with a higher importance level can use more processing resources to achieve priority processing, ensuring the execution of the business and data quality of the data application with a higher importance level, ensuring that the important data plays an important role as much as possible, and improving resource utilization.

[0062] In the above Figure 3 Based on the data management method shown, an embodiment of the present invention can also provide a data management method. This embodiment can provide a possible implementation method for determining the importance level of a data table. The data table of the data application includes: a usage data table of the data application, and an upstream data table of the usage data table. The upstream data table is a data table used to generate the usage data table. Figure 4 The process of the data management method provided by the embodiment of the present invention Figure 2 .like Figure 4 As shown, in the above S301, determining the importance level of the data table of the data application according to the level of the data application may include:

[0063] S401. Determine the importance level of the usage data table according to the level of the data application.

[0064] S402: According to the importance level of the usage data table, reverse tracing is performed to determine the importance level of the upstream data table.

[0065] In this method, for example, the level of the data application can be directly determined as the importance level of the usage data table, and then based on the "bloodline" relationship between data in the big data platform, reverse tracing is performed to determine the importance level of the upstream data table of the usage data table of the data application.

[0066] The method performs reverse tracing according to the importance level of the data table used, determines the importance level of the upstream data table of the data table used, and thus obtains the importance level of each data table of the data application, which can make the data importance level more accurate, effectively ensure the data quality after processing of data with a higher importance level, ensure that the important data plays an important role as much as possible, and improve resource utilization.

[0067] Optionally, determining the importance level of the data table of the data application according to the level of the data application in S301 as shown above may also include:

[0068] S403: If a data table is an upstream data table of a data table used by multiple data applications, and the importance levels of the data table obtained according to the multiple data applications are different, the highest importance level is determined as the importance level of the data table.

[0069] For example, Figure 5 A schematic diagram of determining the importance level of a data table in a data management method provided by an embodiment of the present invention. Figure 5 As shown, since the operation monitoring platform can provide operation data for enterprises and improve monitoring, enterprise operation analysis, online supervision and other functions, the level of the operation monitoring platform in the data application system is L2. The data directly used by the operation monitoring platform, that is, the usage data table of the operation monitoring platform, is the order detail wide table. According to the level L2 of the operation monitoring platform, it can be obtained that the importance level of the order detail wide table is also L2. According to the importance level L2 of the order detail wide table, reverse tracing is performed to determine that the upstream data tables of the order detail wide table include: the order wide table, the order type dimension table, the order details table, and the order zipper table are all at an importance level of L2.

[0070] The external data report is related to the important business information of the enterprise and bears public relations responsibilities. The level of the external data report can be L1. The data directly used by the external data report, that is, the data table used by the external data report, is the order amount summary table. According to the level L1 of the external data report, it can be obtained that the importance level of the order amount summary table is also L1. According to the importance level L1 of the order amount summary table, reverse tracing is performed to determine that the upstream data tables of the order amount summary table include: the order details table and the order zipper table are both at the importance level L1.

[0071] Since the order details table and the order zipper table are upstream data tables of the data tables used by the operation monitoring platform and the external data report, the importance level of the order details table and the order zipper table obtained based on the operation monitoring platform is L2, while the importance level of the order details table and the order zipper table obtained based on the external data report is L1. By selecting the highest importance level, it can be determined that the importance level of the order details table and the order zipper table is L1.

[0072] In this method, for a data table that is an upstream data table of a data table used by multiple data applications, and the importance levels of the data table obtained according to the multiple data applications are different, the highest importance level can be determined as the importance level of the data table, which can effectively ensure the importance of the data table, so that the importance level can be used to manage the data processing process of the data table, such as executing the quality monitoring rules corresponding to the importance level, etc., to ensure the data quality of the data table after data processing.

[0073] In the above Figure 3 or Figure 4 Based on the data management method shown, an embodiment of the present invention may also provide a data management method. This embodiment may provide a possible implementation mode of determining data management. Figure 6 The process of the data management method provided by the embodiment of the present invention Figure 3 .like Figure 6 As shown, in the above S302, managing the data processing process of the data table according to the importance level of the data table may include:

[0074] S601. According to the importance level of the data table, determine the quality monitoring rule corresponding to the importance level.

[0075] S602: Check the data quality according to the quality monitoring rule to obtain the data quality information of the data table during or after the scheduling execution of the data task of the data table.

[0076] The scheduling execution of the data task is used to process the data in the data table.

[0077] Before the data task is scheduled for execution, corresponding processing resources may be allocated to the data table according to the importance level, and the data task corresponding to the data table may be scheduled according to the processing resources to perform data processing on the data table.

[0078] The processing resources may include: resources such as accounts and queues required for scheduling the data task. In the method, the data task corresponding to the data table is scheduled according to the processing resources, which can ensure that the data task is performed safely and effectively, thereby ensuring the data quality of the processed data.

[0079] The data task can also be called a scheduling task. The data task can be a task that encapsulates data collection, processing, transmission and output. Each execution of the data task can form a task instance. The allocation of the processing resources can be determined by Figure 2 The scheduling center shown in the figure can schedule the data task corresponding to the data table according to the processing resources allocated by the cluster management system, such as the account number, queue and other resources required for scheduling the data task, so that the data task is implemented in the distributed cluster to process the data table.

[0080] In this method, processing resources may be allocated preferentially to data tables with higher importance levels according to the importance levels of the data tables, thereby ensuring the service level agreement (SLA) time of the data tasks.

[0081] Optionally, the data tasks shown above may include at least one of the following: an ETL task, a data computing task, and a data pushing task.

[0082] In this method, the quality monitoring rules corresponding to the importance level of the data table can be determined according to the importance level of the data table, so that the higher the importance level, the more items of data quality information included in the quality monitoring rules corresponding to the importance level, and the lower the importance level, the fewer items of data quality information included in the quality monitoring rules corresponding to the importance level. Figure 2 Distributed cluster implementation in .

[0083] Optionally, the data quality information includes at least one of the following information: data integrity, data accuracy, data consistency and timeliness, etc.

[0084] For example, if the importance level of the data table is L1 or L2, the quality monitoring rules corresponding to the importance level of the data table must at least include: inspection rules for four data quality information, namely, data integrity, data accuracy, data consistency and timeliness, among which timeliness has the highest priority. If the importance level of the data table is L3, the quality monitoring rules corresponding to the importance level of the data table must at least include: inspection rules for one of the data quality information, namely, data integrity, data accuracy, data consistency and timeliness. If the importance level of the data table is L4, the quality monitoring rules corresponding to the importance level of the data table may include: inspection rules for at least one of the data quality information, namely, data integrity, data accuracy, data consistency and timeliness, and may not include any inspection rules for data quality information. At this importance level, no mandatory requirement is made. Among them, the importance level of L1 is higher than that of L2, the importance level of L2 is higher than that of L3, and the importance level of L3 is higher than that of L4.

[0085] The quality monitoring rules corresponding to the above-mentioned importance levels are only possible examples, and they may also be in other forms, including other information, which is not limited in the embodiments of the present invention.

[0086] Above Figure 2 When the distributed cluster in the data processing unit determines the quality monitoring rule, it can control the data quality monitoring platform to check the data quality according to the quality monitoring rule, so as to control the data quality monitoring platform to obtain the data quality information of the data table from the scheduling center during or after the scheduling execution of the data task.

[0087] In this method, the quality monitoring rules corresponding to the importance level of the data table can be determined according to the importance level of the data table, and the data quality can be checked according to the quality monitoring rules to obtain the data quality information of the data table during the scheduling execution of the data task of the data table or after the scheduling execution, so that the data tables with higher importance levels can implement strict data quality assurance measures to ensure the application accuracy of key data.

[0088] Optionally, if a scheduling failure occurs during the scheduling and execution of the above-mentioned data task, the corresponding fault response time may be different for different importance levels. The higher the importance level, the shorter the fault response time corresponding to the importance level, thereby ensuring the timeliness and correctness of the data task; the lower the importance level, the longer the fault response time corresponding to the importance level, thereby ensuring that the timeliness, correctness, and fault response time requirements of data tasks such as data collection and processing of data tables of different importance levels are different, thereby ensuring the scheduling and execution of data tasks corresponding to the data table based on the Service-Level Agreement (SLA).

[0089] Optionally, in the method shown above, if there is a quality problem with the data in the data table or a delay in the scheduling execution time of the data task during or after the scheduling execution of the data task, the method may further include:

[0090] S603: Determine an alarm mode corresponding to the importance level according to the importance level of the data table.

[0091] S604: issuing a corresponding alarm using the alarm method to indicate that during or after the scheduling execution of the data task, there is a quality problem with the data in the data table or that the scheduling execution time of the data task is delayed.

[0092] In this embodiment, if there is at least one piece of data quality information that does not meet the preset conditions in the scheduling execution results obtained based on the quality monitoring rule, it can be determined that there is a data quality problem in the data table after the data task is scheduled and executed. On the contrary, if each piece of data quality information in the scheduling execution results obtained based on the quality monitoring rule meets the preset conditions, it can be determined that there is no data quality problem in the data table after the data task is scheduled and executed. If the scheduling execution time of the data task is greater than or equal to the preset time, it can be determined that the scheduling execution time of the data task is delayed.

[0093] In this embodiment, the alarm modes corresponding to different importance levels may have corresponding response times.

[0094] For example, if the importance level of the data table is L1 or L2, the alarm method corresponding to the importance level of the data table may at least include: telephone alarm method; if the importance level of the data table is L3, the alarm method corresponding to the importance level of the data table may include: short message alarm method; if the importance level of the data table is L4, the alarm method corresponding to the importance level of the data table may include: email alarm method. The alarm methods corresponding to different importance levels may have corresponding response times.

[0095] The above-mentioned warning modes corresponding to the various importance levels are only possible examples, and may also be in other forms, including other information, which is not limited in the embodiments of the present invention.

[0096] The S603 and S604 can be Figure 2 Distributed cluster implementation in .

[0097] In this embodiment, the alarm method corresponding to the importance level of the data table is determined according to the importance level of the data table, so that each importance level has its own corresponding alarm method, and the corresponding alarm is issued using the alarm method to indicate the scheduling execution process of the data task or the existence of data quality problems in the data table after the scheduling execution of the data task or the delay in the scheduling execution time of the data task, thereby ensuring the timeliness, correctness and fault response time of key data, thereby ensuring the normal order of key data applications.

[0098] In the above Figure 3-Figure 6 Based on any of the data management methods shown in , an embodiment of the present invention may also provide a data management method. This embodiment may provide a possible implementation method for changing the structure of a data table in a big data platform when the table structure of a business system is changed. Figure 7 The process of the data management method provided by the embodiment of the present invention Figure 4 .like Figure 7 As shown, based on the above data management method, the method may also include:

[0099] S701: If the table structure of the business system is changed, determine whether to change the structure of the data table according to the importance level of the data table and the change type of the table structure of the business system.

[0100] S702: If yes, perform a change operation corresponding to the table structure of the business system on the data table.

[0101] In this embodiment, according to the importance level of the data table and the change type of the table structure, a preset change requirement table corresponding to the importance level and change type may be used to determine whether to change the table structure of the data table. The change requirement table corresponding to the importance level and change type may be, for example, as shown in Table 1 below.

[0102] Table 1

[0103]

[0104]

[0105] If the importance level of the data table is L1 or L2, and the change type of the table structure is adding a new field, then it can be determined that the change requirement is no, that is, there is no need to change the table structure of the data table; if the importance level of the data table is L1 or L2, and the change type of the table structure is modifying a field, deleting a field, modifying a field type, adding or modifying a comment, or deleting a table, then it can be determined that the change requirement is yes, that is, the table structure of the data table needs to be changed.

[0106] If the importance level of the data table is L3, and the change type of the table structure is adding a new field, or adding or modifying a comment, then it can be determined that the change requirement is no, that is, there is no need to change the table structure of the data table; if the importance level of the data table is L3, and the change type of the table structure is modifying a field, deleting a field, modifying a field type, or deleting a table, then it can be determined that the change requirement is yes, that is, the table structure of the data table needs to be changed.

[0107] If the importance level of the data table is L4, and the change type of the table structure is adding a new field, deleting a field, modifying the field type, or adding or modifying a comment, then it can be determined that the change requirement is no, that is, there is no need to change the table structure of the data table; if the importance level of the data table is L4, and the change type of the table structure is modifying a field, or deleting a table, then it can be determined that the change requirement is yes, that is, the table structure of the data table needs to be changed.

[0108] In this method, the above Figure 2The database system 11 determines whether to push the change request based on the importance level of the data table and the change type of the table structure. If so, the change request is transmitted to the demand management platform to request the demand management platform to transmit the information of the change table structure to the data management platform, so that the data management platform makes corresponding changes to the table corresponding to the change table structure of the data table based on the information of the change table structure.

[0109] In this embodiment, whether to change the table structure of the data table can be determined based on the importance level of the data table and the type of change in the table structure, and then the table structure of the data table can be changed, thereby achieving precise control of changes in the table structure on the big data platform after the table structure of the business system is changed.

[0110] The business system may be an online business system. If the table structure of the business system changes, the table structure of the data in the database of the business system in the database system 11 will also change. In order to ensure that the big data platform accurately processes the data table from the database of the business system, it is also necessary to perform a change operation corresponding to the table structure of the business system on the data table. S702 can be performed by the above Figure 2 The data management platform in the system processes the change of the table structure, calls the change resources corresponding to the importance level on the distributed cluster to change the table, and feeds back the change results to the data management platform in real time.

[0111] S703: According to the changed table, adjust the script of the data task corresponding to the data table.

[0112] In this method, after the change is made, the script of the data task can be determined according to the information of the data task corresponding to the data table, and the script of the data task can be adjusted. The script of the data task can include: a calculation script and / or an analysis script of the data task.

[0113] exist Figure 2 After the data management platform processes the table changes, it can also obtain the metadata information, storage information and data calculation execution log corresponding to the changed table from the distributed cluster, and send a change instruction to the data development platform based on the information obtained from the distributed cluster, so that the data development platform can adjust the script of the data task corresponding to the data table, such as the calculation script and / or analysis script of the data task.

[0114] When the script of the data task is changed, a notification can be sent to the responsible node of the downstream task of the data task to notify the downstream responsible node to make corresponding modifications to ensure the stability and smoothness of the data processing process by scheduling the data tasks corresponding to the data table.

[0115] S704: Determine a test strategy corresponding to the importance level according to the importance level of the data table.

[0116] S705: Test the adjusted script under the test strategy corresponding to the importance level.

[0117] Since the script of the data task corresponding to the data table has been adjusted, in order to ensure the normal operation of the adjusted script, the adjusted script needs to be tested. Since script testing also requires certain test resources, if the same test strategy is used for all adjusted scripts, it may cause a waste of test resources, or the test accuracy is not high, and it is difficult to ensure the test effect. Therefore, in this embodiment, according to the importance level of the data table, the test strategy corresponding to the importance level can be determined, and then the adjusted script can be tested according to the test strategy corresponding to the importance level. The test strategies corresponding to different importance levels include: information about different test environments.

[0118] Optionally, the test strategy is also called a test plan, which may include information of at least one of the following test environments: a development environment, a test environment, a grayscale environment, and an online environment.

[0119] Among them, the development environment refers to the software environment in the process of developing, debugging and unit testing the adjusted script. The test environment refers to the software environment for testing the adjusted script based on a preset test data set after the adjusted script is developed. The test scope of the grayscale environment is larger than the test environment, but smaller than the online environment. The grayscale environment can also be called a small-scale online environment, and the test in the grayscale environment can be called an internal test. The online environment is also called the formal environment, which is a software environment for testing in the online running environment after the adjusted script is developed.

[0120] For example, if the importance level of the data table is L1, the test strategy corresponding to the importance level may include: testing in test environments such as development environment, test environment, grayscale environment and online environment with formal data, and testing the adjusted script according to the test strategy corresponding to the importance level.

[0121] If the importance level of the data table is L2, the test strategy corresponding to the importance level may include: testing in test environments such as development environment, test environment, grayscale environment and online environment, and testing the adjusted script according to the test strategy corresponding to the importance level.

[0122] If the importance level of the data table is L3, the test strategy corresponding to the importance level may include: testing in development environment, test environment, grayscale environment and other test environments, and the adjusted script is tested according to the test strategy corresponding to the importance level.

[0123] If the importance level of the data table is L4, the test strategy corresponding to the importance level may include: testing in the development environment, test environment and other test environments, and the adjusted script is tested according to the test strategy corresponding to the importance level. Among them, the importance level of L1 is higher than the importance level of L2, the importance level of L2 is higher than the importance level of L3, and the importance level of L3 is higher than the importance level of L4.

[0124] The test strategies corresponding to the above-mentioned importance levels are only possible examples, and they may also be in other forms, including other information, which is not limited in the embodiments of the present invention.

[0125] In the method of this embodiment, the test strategy corresponding to the importance level can be determined according to the importance level of the data table, and then the adjusted script can be tested according to the test strategy corresponding to the importance level, so as to avoid waste of test resources, ensure test accuracy, and ensure test effect.

[0126] Optionally, the method may further include:

[0127] S706. According to the importance level of the data table, determine the online approval process corresponding to the importance level.

[0128] The online approval process includes: at least one online approval node in the big data platform. The at least one online approval node includes: the superior approval node of the responsible node of the script of the data task, the responsible approval node of the subordinate script of the script of the data task, the change requirement proposal approval node, the business responsible approval node corresponding to the data task, etc.

[0129] For example, if the importance level of the data table is L1 or L2, the online approval process corresponding to the importance level may include the following four online approval nodes: the superior approval node of the responsible node of the script of the data task, the responsible approval node of the subordinate script of the script of the data task, the change request proposal approval node, and the business responsible approval node corresponding to the data task.

[0130] If the importance level of the data table is L3, the online approval process corresponding to this importance level may include the following three online approval nodes: the superior approval node of the responsible node of the script of the data task, the responsible approval node of the subordinate script of the script of the data task, and the change request submission approval node.

[0131] If the importance level of the data table is L4, the online approval process corresponding to the importance level may include: the superior approval node of the responsible node of the script of the data task. Among them, the importance level of L1 is higher than the importance level of L2, the importance level of L2 is higher than the importance level of L3, and the importance level of L3 is higher than the importance level of L4.

[0132] The above-mentioned online approval processes corresponding to the various importance levels are only possible examples, and may also be in other forms, including other information, which is not limited in the embodiments of the present invention.

[0133] S707: According to the online approval process, the adjusted script is approved.

[0134] The embodiment of the present invention can also approve the adjusted script according to the upper limit approval process corresponding to the importance level of the data table to ensure the timeliness of the online release of the adjusted script and the operational stability of the script after it is online.

[0135] Based on any of the above methods, optionally, the method may further include:

[0136] If the database of the business system is expanded or migrated, the data source of the big data platform will be expanded or changed. Figure 2 The database system shown transmits the change information of the database that has been expanded or migrated to Figure 2 The data source management system shown in the figure expands or changes the data source of the big data platform according to the change information. The data source management system can also transmit the change information to the scheduling center, and the scheduling center adds or modifies the data source referenced by the data task according to the change information.

[0137] When the data source referenced by the data task changes, such as expansion or modification, the test strategy corresponding to the importance level can be determined according to the importance level of the data table involved in the data task, and then the data task after the referenced data source is changed can be tested according to the test strategy. In the method of this embodiment, the test strategy corresponding to the importance level can be determined according to the importance level of the data table, and then the data task after the referenced data source is changed can be tested according to the test strategy corresponding to the importance level, so as to avoid the waste of test resources, ensure the test accuracy, and ensure the test effect.

[0138] After testing the data task after the reference data source is changed, the online approval process corresponding to the importance level can be determined according to the importance level of the data table involved in the data task, and the data task after the reference data source is changed can be approved according to the online approval process corresponding to the importance level of the data table, so as to ensure the online timeliness of the data task after the reference data source is changed, and ensure the operational stability of the data task after the reference data source is changed after going online.

[0139] The following is an embodiment of the device of the present invention, which can be used to execute the above-mentioned method embodiment of the present invention, and its implementation principle and technical effect are similar.

[0140] Figure 8 Schematic diagram of the structure of the data management device provided by the embodiment of the present invention. Figure 8 As shown, the data management device 80 of this embodiment can be applied to a big data platform, and the data management device 80 may include:

[0141] The determination module 81 is used to determine the importance level of the data table of the data application according to the level of the data application.

[0142] The management module 82 is used to manage the data processing process of the data table according to the importance level of the data table.

[0143] Optionally, the data table of the data application includes: a usage data table of the data application, and an upstream data table of the usage data table; the upstream data table is a data table used to generate the usage data table.

[0144] The determination module 81 is specifically used to determine the importance level of the usage data table of the usage data application according to the level of the data application; and to perform reverse tracing according to the importance level of the usage data table to determine the importance level of the upstream data table.

[0145] Optionally, the determination module 81 is further specifically used to determine the highest importance level as the importance level of a data table if a data table is located at the upstream data table of a data table used by multiple data applications, and the importance levels of the data table obtained according to the multiple data applications are different.

[0146] Optionally, the management module 82 is specifically used to determine the quality monitoring rules corresponding to the importance level of the data table according to the importance level of the data table; and to check the data quality according to the quality monitoring rules to obtain the data quality information of the data table during or after the scheduling execution of the data task. The scheduling execution of the data task is used to perform data processing on the data table.

[0147] Optionally, the data task includes at least one of the following: an ETL task, a data computing task, and a data pushing task.

[0148] Optionally, the data quality information includes at least one of the following information: data integrity, data accuracy, and data consistency.

[0149] Optionally, the management module 82 is also used to determine the alarm method corresponding to the importance level of the data table according to the importance level of the data table, and issue a corresponding alarm using the alarm method to indicate that there is a data quality problem in the data table or there is a delay in the scheduling execution time of the data task during or after the scheduling execution of the data task.

[0150] Optionally, the data management device 80 further includes:

[0151] The change module is used to determine whether to change the structure of the data table according to the importance level of the data table and the change type of the table structure of the business system if the table structure of the business system changes; if so, perform the change operation corresponding to the table structure of the business system on the data table; and adjust the script of the data task corresponding to the data table according to the changed table;

[0152] The test module is used to determine the test strategy corresponding to the importance level according to the importance level of the data table; and test the adjusted script under the test strategy corresponding to the importance level. The test strategies corresponding to different importance levels include: tests under different test environments.

[0153] Optionally, the testing strategy includes testing in at least one of the following testing environments: a development environment, a test environment, a grayscale environment, or an online environment.

[0154] Optionally, the management module 82 is further used to determine, according to the importance level of the data table, an online approval process corresponding to the importance level; and to approve the adjusted script according to the online approval process.

[0155] Fig. 9 A schematic diagram of the structure of a server provided in one embodiment of the present invention. Fig. 9 As shown, the server 90 of this embodiment includes: a memory 91 and a processor 92. The memory 91 is connected to the processor 92 via a bus. The server 90 may be a server of the above-mentioned big data platform, and may be one server or may be implemented by multiple servers.

[0156] The memory 91 is used to store program instructions.

[0157] Processor 92, for executing the above Figure 3-Figure 7 Any of the methods in the data management method.

[0158] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which can be used by the above-mentioned Fig. 9The processor 92 executes the above Figure 3-Figure 7 Any of the methods in the data management method.

[0159] The data management device, server router and computer-readable storage medium provided by the embodiment of the present invention can be executed Figure 3-Figure 7 The specific implementation and effective effects of the data management method in any of the methods can be found above and will not be repeated here.

[0160] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: Read-Only Memory (ROM), Random Access Memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data management method, characterized in that: The data management method is applicable to a big data platform, and the method comprises: Determining the importance level of the data table of the data application according to the level of the data application; According to the importance level of the data table, determining a quality monitoring rule corresponding to the importance level; the higher the importance level, the more items of data quality information included in the quality monitoring rule corresponding to the importance level; Performing a data quality check according to the quality monitoring rule to obtain data quality information of the data table during or after the scheduling execution of the data task of the data table, wherein the scheduling execution of the data task is used to perform data processing on the data table; The method further comprises: if the table structure of the business system is changed, determining whether to change the structure of the data table according to the importance level of the data table and the change type of the table structure of the business system through a preset change requirement table; the change requirement table is used to indicate whether to change the structure of the data table under different combinations of each importance level of the data table and each change type of the table structure of the business system; If yes, perform a change operation corresponding to the table structure of the business system on the data table; According to the changed data table, adjusting the script of the data task corresponding to the data table; According to the changed importance level of the data table, a test strategy corresponding to the importance level is determined; under the test strategy corresponding to the importance level, the adjusted script is tested.

2. The method according to claim 1, characterized in that: The data table of the data application includes: a usage data table of the data application, and an upstream data table of the usage data table; the upstream data table is a data table used to generate the usage data table; the determining the importance level of the data table of the data application according to the level of the data application includes: Determining the importance level of the usage data table according to the level of the data application; According to the importance level of the usage data table, reverse tracing is performed to determine the importance level of the upstream data table.

3. The method according to claim 1, characterized in that The step of determining the importance level of the data table of the data application according to the level of the data application further includes: If a data table is an upstream data table of a data table used by multiple data applications, and the importance levels of the data table obtained according to the multiple data applications are different, the highest importance level is determined as the importance level of the data table.

4. The method according to claim 1, characterized in that The data task includes at least one of the following: an extract, transform and load (ETL) task, a data computing task, and a data pushing task.

5. The method according to claim 1, characterized in that The data quality information includes at least one of the following information: data integrity, data accuracy, and data consistency.

6. The method according to claim 1, characterized in that If there is a quality problem with the data in the data table during or after the scheduled execution of the data task, or there is a delay in the scheduled execution time of the data task, the method further includes: According to the importance level of the data table, determining an alarm mode corresponding to the importance level; The corresponding alarm is issued by adopting the alarm method to indicate that there is a quality problem with the data in the data table during or after the scheduling execution of the data task or that the scheduling execution time of the data task is delayed.

7. The method according to claim 1, characterized in that Different importance levels correspond to different test strategies, and different test strategies include: testing under different test environments.

8. The method according to claim 7, characterized in that The testing strategy includes testing of at least one of the following testing environments: a development environment, a test environment, a grayscale environment, or an online environment.

9. The method according to claim 7, characterized in that: The method further comprises: According to the importance level of the data table, determine the online approval process corresponding to the importance level; According to the online approval process, the adjusted script is approved.

10. A data management device, characterized in that: The data management device is applicable to a big data platform, and the data management device comprises: A determination module, used to determine the importance level of the data table of the data application according to the level of the data application; A management module, configured to determine, according to the importance level of the data table, a quality monitoring rule corresponding to the importance level; the higher the importance level, the more items of data quality information included in the quality monitoring rule corresponding to the importance level; Performing a data quality check according to the quality monitoring rule to obtain data quality information of the data table during or after the scheduling execution of the data task of the data table, wherein the scheduling execution of the data task is used to perform data processing on the data table; A change module is used to determine whether to change the structure of the data table according to the importance level of the data table and the change type of the table structure of the business system through a preset change requirement table if the table structure of the business system changes; the change requirement table is used to indicate whether to change the structure of the data table under different combinations of each importance level of the data table and each change type of the table structure of the business system; if so, perform the change operation corresponding to the table structure of the business system on the data table; and adjust the script of the data task corresponding to the data table according to the changed data table; The test module is used to determine the test strategy corresponding to the importance level according to the changed importance level of the data table; and test the adjusted script under the test strategy corresponding to the importance level.

11. A server, characterized in that: include: Memory and processor; The memory is connected to the processor; The memory is used to store program instructions; The processor is used to implement the data management method according to any one of claims 1 to 9 when the program instructions are executed.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data management method according to any one of claims 1 to 9 is implemented.

Images