Data management method and device, equipment and storage medium

Abstract

The invention provides a data management method and device, equipment and a storage medium. The data management method comprises the steps of obtaining a plurality of data tables from a plurality of data processing platforms; metadata information corresponding to the data tables is determined according to the data structure information of the data tables; constructing a data module according to metadata information of the data tables under the same data structure information in the plurality of data tables to obtain a plurality of data modules; and managing the plurality of data tables according to the plurality of data modules. The flexibility and efficiency of data table management are improved, and the data processing difficulty is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data management method, apparatus, device and storage medium. Background Art

[0002] At present, the process of data collection and processing involves the calculation, circulation and storage of data in the data processing platform, which can be Hive, HBase or Kafka.

[0003] However, due to the differences in data collection and access, processing logic and timeliness requirements during the data processing process, a data processing link often requires the use of multiple data processing platforms, and each data processing platform includes multiple data tables. The management and maintenance of the data tables in each data processing platform in the relevant technology are scattered and independent. Therefore, there are problems such as high difficulty in data dimension and low processing efficiency. Summary of the invention

[0004] The present application provides a data management method, apparatus, device and storage medium to improve the flexibility and efficiency of data table management and reduce the difficulty of data processing.

[0005] In a first aspect, the present application provides a data management method, the method comprising: obtaining multiple data tables from multiple data processing platforms; determining metadata information corresponding to each data table based on data structure information of each of the multiple data tables; constructing a data module based on the metadata information of the data tables under the same data structure information in the multiple data tables to obtain multiple data modules; and managing the multiple data tables based on the multiple data modules.

[0006] In a second aspect, the present application provides a data management device, including a first acquisition module, used to: acquire multiple data tables from multiple data processing platforms; a first determination module, used to: determine metadata information corresponding to each data table based on the data structure information of each of the multiple data tables; a construction module, used to: construct data modules based on the metadata information of the data tables under the same data structure information in the multiple data tables to obtain multiple data modules; and a management module, used to: manage multiple data tables based on the multiple data modules.

[0007] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method in the first aspect or its various implementations.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the method in the first aspect or its various implementations.

[0009] In a fifth aspect, the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method in the first aspect or its various implementations.

[0010] In a sixth aspect, the present application provides a computer program, which enables a computer to execute the method in the first aspect or its various implementations.

[0011] Through the technical solution of the present application, an electronic device can construct a data module based on the metadata information of a data table under the same data structure information in multiple data tables from multiple data processing platforms, and manage the data tables of multiple data processing platforms based on the data module to improve the flexibility and efficiency of data table management and reduce the difficulty of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 An application scenario diagram provided for an embodiment of the present application;

[0014] Figure 2 A flowchart of a data management method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of a data management method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of another data management method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of a data management device 600 provided in an embodiment of the present application;

[0018] Figure 6 A schematic block diagram of an electronic device 600 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] As mentioned above, the management and maintenance of data tables in each data processing platform in the prior art are decentralized and independent. Therefore, there are problems such as high difficulty in data dimensioning and low processing efficiency.

[0022] In order to solve the above technical problems, the inventive concept of the present application is: to construct a data module based on the metadata information of the data table under the same data structure information in multiple data tables from multiple data processing platforms, and to manage the data tables of multiple data processing platforms based on the data module, so as to improve the flexibility and efficiency of data table management and reduce the difficulty of data processing.

[0023] It should be understood that the technical solution of the present application can be applied to the following scenarios, but is not limited to:

[0024] In one embodiment, the technical solution of the present application can be used in data management scenarios, specifically in scenarios of management and maintenance of data tables in multiple data processing platforms. For example, it can be applied to scenarios of management and maintenance of multiple data tables in Hive, HBase, and Kafka, and the present application does not impose any restrictions on this.

[0025] In one embodiment, Figure 1 An application scenario diagram provided for an embodiment of the present application, such as Figure 1 As shown, the application scenario may include a terminal device 110 and an electronic device 120. The terminal device 110 may establish a connection with the electronic device 120 via a wired network or a wireless network.

[0026] Exemplarily, the terminal device 110 may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) or virtual reality (VR) device, etc., but is not limited thereto. The electronic device 120 may be a server, a server cluster consisting of multiple servers, or a cloud platform control center, which is not limited in the embodiments of the present application.

[0027] Exemplarily, the electronic device 120 or the terminal device 110 may include multiple data processing platforms, such as Hive, HBase or Kafka, etc., or multiple data processing platforms are managed by other servers. The electronic device 120 can obtain data tables from multiple data processing platforms and construct data modules corresponding to multiple data tables, so as to manage multiple data tables according to the data modules. In addition, the electronic device 120 can also obtain creation operations or deletion operations for data tables from the terminal device 110, and perform corresponding data table creation or deletion. The electronic device 120 can also construct bloodline links for multiple data tables and send the bloodline links to the terminal device 110 for display.

[0028] also, Figure 1 An electronic device and a terminal device are given as an example, and in fact other numbers of electronic devices and terminal devices may be included, and this application does not impose any limitation on this.

[0029] In another embodiment, the technical solution of the present application may also be executed by the above-mentioned terminal device 110, or the technical solution of the present application may also be executed by the above-mentioned electronic device 120, and the present application does not impose any limitation on this.

[0030] It should be noted that all technical solutions in the present application can be arbitrarily combined to form optional embodiments of the present application. To avoid repetition, they will not be described in detail.

[0031] It should be noted that in the specific implementation of this application, related data such as data tables, data structure information, metadata information, data modules, data lineage relationships, etc. are involved. When the embodiments of this application are applied to specific products or technologies, user permission, consent or authorization is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0032] The technical solution of this application will be described in detail below:

[0033] Figure 2 A flowchart of a data management method provided in an embodiment of the present application, the method can be performed as follows Figure 1 The electronic device 120 shown in the figure performs, but is not limited to this. Figure 2 As shown, the method may include the following steps:

[0034] S210: Acquire multiple data tables from multiple data processing platforms;

[0035] S220: Determine metadata information corresponding to each data table according to data structure information of each of the multiple data tables;

[0036] S230: constructing data modules according to metadata information of data tables under the same data structure information in multiple data tables to obtain multiple data modules;

[0037] S240: managing multiple data tables according to multiple data modules.

[0038] It is understandable that due to the requirements of data collection, access, processing logic and timeliness, electronic devices need to calculate, circulate, store, etc. data based on different data processing platforms. Therefore, the data processing platform can realize multiple data tables involved in data collection and data processing. Specifically, different data processing platforms can store and process data (such as business data) through different data storage media. The data storage medium can be any of the following, but not limited to: Hive, HBase or Kafka.

[0039] In addition, the data processing platform can be understood as the media involved in the link of data collection and processing. In addition to the above-mentioned Hive, HBase or Kafka, the media can also be (that is, the data processing platform can also be): storage devices, memory, computing devices, network devices and software platforms. Among them, storage devices include but are not limited to: hard disks, flash memory, tapes, and optical disks; memory includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM); computing devices include but are not limited to servers, personal computers, and cloud computing devices; network devices include but are not limited to switches and routers; software platforms include but are not limited to operating systems and data processing software.

[0040] The following is an introduction to the data tables, data structure information, and metadata information of the data processing platform:

[0041] A data table, also known as a database table or data record table, is a storage structure in a data processing platform that can store data in the form of rows and columns. Each column can represent a specific data attribute or field, and each row can represent a data record or entity.

[0042] The data structure information of a data table describes the organization and data attributes of the data in the data table. The data structure information may include, but is not limited to, the fields in the data table, the data types of the data entities corresponding to the fields, and constraint information. The constraint information is the rules that restrict the data in the data table, for example, the rules that restrict the length of the data entity in the data table, and the rules for changing the data entity.

[0043] The metadata information of a data table may include descriptive information of the data table (e.g., the name of the data table, creation time, modification time, and data structure information of the data table, etc.) and additional information (e.g., the source, purpose, version, creator, etc. of the data table). In other words, the metadata information of a data table may include the data structure information of the data table.

[0044] In one embodiment, the electronic device may use Structured Query Language (SQL) to determine the data structure information of the data table, but is not limited thereto.

[0045] Exemplarily, for a MySQL database, an electronic device can use the DESCRIBE (DESC) command to obtain data structure information of a data table, wherein the DESCRIBE command can be used to obtain information such as the name, type, and key of each field included in the data table. Specifically, the electronic device can use DESCRIBE and the table name of the data table to be queried to obtain data structure information of the data to be queried. For example, the command "DESCRIBE table name of the data table to be queried" can be input into MySQL, and MySQL can execute the above command to output the data structure information of the data to be queried.

[0046] In one embodiment, the electronic device may directly determine the data structure information of the data table as metadata information of the data table. Alternatively, the electronic device may also obtain additional information of the data table and other descriptive information other than the data structure information, and determine the data structure information, additional information and other descriptive information as metadata information.

[0047] For example, the electronic device may obtain a change log of the data table and obtain the version of the data table from the change log. Alternatively, when creating a data table, the creator, creation time, purpose, source, etc. of the data table may be recorded through other data tables or documents, and the electronic device may obtain corresponding metadata information through the other data tables or documents.

[0048] It is understandable that the above determination of metadata information based on data structure information can ensure that each constructed data module includes the same part of the metadata information of the data table under the same data structure information, so that efficient and consistent management of metadata information for multiple data tables and multiple data tables can be achieved to avoid repeated entry of data. This application will be described in detail in subsequent embodiments.

[0049] In one embodiment, it is assumed that the data processing platform is a MySQL database, and there is a data table in the MySQL database: a customer table (Customer), which can be as follows:

[0050] {Customer ID: integer type (INT), automatic increase (AUTO_INCREMENT), primary key (PRIMARY KEY);

[0051] Customer name (FirstName): variable character type (VARCHAR), maximum length is 50 (50);

[0052] Customer last name (LastName): variable character type (VARCHAR), maximum length is 50 (50);

[0053] Email: variable character type (VARCHAR), maximum length is 100 (100);

[0054] JoinDate: Date type (DATE)}

[0055] Among them, the above-mentioned customer table includes 5 fields: customer ID, customer first name, customer last name, email address, and registration date; the data types of the data entities corresponding to the 5 fields are: integer type, variable character type, variable character type, variable character type, and date type; the constraint information corresponding to the first 4 fields is: the corresponding data entity can automatically grow and the field is the primary key of the customer table, the maximum length of the corresponding data entity is 50, the maximum length of the corresponding data entity is 50, and the maximum length of the corresponding data entity is 100.

[0056] Then the data structure information of the customer table can be: [customer ID: integer type, automatic growth, primary key; customer name: variable character type, maximum length of 50; customer surname: variable character type, maximum length of 50; email address: variable character type, maximum length of 100; registration date: date type]. This application does not limit the form of the data structure information.

[0057] Correspondingly, the electronic device can directly determine the above data structure information as metadata information, that is, determine the metadata information of the above customer table as: [customer ID: integer type, automatic growth, primary key; customer name: variable character type, maximum length of 50; customer last name: variable character type, maximum length of 50; email address: variable character type, maximum length of 100; registration date: date type].

[0058] Alternatively, assuming that the source of the above customer table is: customer registration form, the creator of the customer table is: Administrator 1, and the creation time of the customer table is: 2024, then combined with the above data structure information, it can be determined that the metadata information of the customer table is [Customer ID: integer type, automatic growth, primary key; Customer name: variable character type, maximum length of 50; Customer last name: variable character type, maximum length of 50; Email: variable character type, maximum length of 100; Registration date: date type; Source: Customer registration form; Creator: Administrator 1; Creation time: 2024].

[0059] In one embodiment, the electronic device may group multiple data tables according to data structure information of the multiple data tables to obtain multiple data table sets, wherein each data table set includes at least one data table having the same data structure information; then, for each data table set, the electronic device may construct a data module according to metadata information of each data table in the data table set.

[0060] The data table set may refer to a set of data tables having the same data structure information. Specifically, for each data table set, the electronic device may construct a data module based on the same information in the metadata information of each data table in the data table set. For example, the data module may be constructed based on the data structure information of at least one data table, and the data module may include the data structure information of at least one data table. In this embodiment, the data module may include the same metadata information.

[0061] Alternatively, the electronic device may construct a data module according to metadata information of at least one data table in the data table set. For example, the data module may include metadata information of at least one data table. In this embodiment, the data module may include a set of metadata information.

[0062] Exemplarily, assuming that the institution user table is a data table with the same data structure information as the above-mentioned customer table, the source of the institution user table is: the institution user registration form, the creator of the institution user table is: Administrator 2, and the creation time of the institution user table is: 2023. In combination with the above embodiments, the electronic device can group the institution user table and the customer table into the same data table set, and determine that the data modules corresponding to the institution user table and the customer table include: [Customer ID: integer type, automatic growth, primary key; Customer name: variable character type, maximum length of 50; Customer last name: variable character type, maximum length of 50; Email: variable character type, maximum length of 100; Registration date: date type]. Alternatively, the data module for determining the correspondence between the institution user table and the customer table includes: [Customer table: (customer ID: integer type, automatic growth, primary key; customer name: variable character type, maximum length of 50; customer surname: variable character type, maximum length of 50; email: variable character type, maximum length of 100; registration date: date type; source: customer registration form; creator: administrator 1; creation time: 2024); institution user table: (customer ID: integer type, automatic growth, primary key; customer name: variable character type, maximum length of 50; customer surname: variable character type, maximum length of 50; email: variable character type, maximum length of 100; registration date: date type; source: institution user registration form; creator: administrator 2; creation time: 2023)].

[0063] It can be understood that a data module is a logical concept, which is a set abstracted on the basis of a data table. The set can include the same metadata information corresponding to a data table under the same data structure information, or the metadata information of each data table under the same data structure information. The data module does not directly correspond to the data table, but can be associated with the data table (this will be explained in the subsequent embodiments). Therefore, the electronic device can be based on the data module, that is, the data module can be used as the minimum granularity to maintain and manage multiple data tables associated with the data module and the metadata information of multiple data tables. In other words, the management of the data module can be implemented for multiple data tables with the same data structure information and originating from different data processing platforms, thereby not only ensuring the consistency of the metadata information of data tables from different data processing platforms under the same data module, but also avoiding repeated entry and repeated management of data, improving the flexibility and efficiency of data table management, making the maintenance of the data table more simple and easy to use, and reducing the difficulty of data processing.

[0064] This application will introduce the process of managing the data table according to the data module in the following embodiments:

[0065] In one embodiment, the electronic device can determine the changed target metadata information and the target data table under the target data module in response to the change operation on the metadata information corresponding to the target data module in multiple data modules; and adjust the data structure information of the target data table according to the changed target metadata information.

[0066] In combination with the above content, it can be seen that the data module includes metadata information corresponding to the data table under the same data structure information. When the data structure information of a data table changes, that is, when the metadata information of a data table changes, the electronic device only needs to update at the data module level, that is, only the metadata information included in the data module needs to be updated to achieve the update of the data structure information of all data tables under the data module. In other words, through the technical solution of the present application, it is possible to achieve the modification of the data structure of all relevant data tables corresponding to the data module by only modifying the metadata information of the data module, which can ensure the synchronous and consistent changes of the metadata information, avoid omissions, wrong changes, online problems, data problems, etc. caused by manual synchronous modifications of multiple data tables from different data processing platforms, and ensure the consistency and accuracy of data definitions.

[0067] Among them, the electronic device can map the data table to the corresponding data module through the automatic association mechanism. For example, the electronic device can build the automatic association mechanism into the electronic device, identify different types of data tables with the same data structure and from different data processing platforms through the automatic association mechanism, and combine them into a data table set. Then, the data tables in the same data table set can be mapped to the data module corresponding to the data table set, so as to efficiently maintain and manage the data tables with the same data structure.

[0068] Specifically, the automatic association mechanism can be an intelligent algorithm or program built into an electronic device. The automatic association mechanism may include a parser. The parser can scan multiple data tables from different data processing platforms according to the method in the above embodiment, and analyze the data structure information of the multiple data tables; then, the data structure information of the multiple data tables can be compared, and the data tables with the same data structure can be combined into a data table set. The data tables in the data table set share the same data structure but may contain different data contents, i.e., data entities; then, after creating a data module based on the metadata information of the data tables in each data table set, the automatic association mechanism can associate the data tables in the corresponding data table set with the data module, and establish a mapping relationship between the data table and the data module, so that when the metadata information corresponding to the target data module changes, the data table that needs to be changed synchronously is determined according to the mapping relationship, i.e., the target data table under the target data module, thereby realizing the synchronous change of the target data table.

[0069] Furthermore, the electronic device can also monitor and collect the structural change information of each data table in each data processing platform in real time, respond to the structural change information of the target data table, determine the corresponding target data module according to the data structure information of the target data table, obtain the changed target metadata information of the target data module according to the structural change information, and then adjust the data structure information of each target data table under the target data module according to the changed target metadata information.

[0070] Through the above embodiments, a unified metadata information management entry and management system can be implemented through the creation of data modules, solving the problems of decentralized data table management, the need to repeatedly enter metadata information, and inefficient operation of multiple data tables. It is possible to automatically pull data tables, and automatically distinguish and classify data tables with different data structure information.

[0071] In addition, if a new data table is created, the automatic association mechanism can also automatically check its data structure information and attribute it to the corresponding data module to achieve the association between the newly added data table and the data module.

[0072] In one embodiment, in response to a creation operation of a new data table, an electronic device can obtain the new data table; determine metadata information of the new data table based on data structure information of the new data table; determine a target data module corresponding to the data structure information of the new data table in multiple data modules based on the metadata information of the new data table; and manage the new data table through the target data module.

[0073] It should be noted that if there is no data module corresponding to the data structure information of the newly added data table among the multiple data modules that have been created, the electronic device can create a new data module based on the metadata information of the new data table, and establish an association relationship between the new data module and the new data table, and manage the new data table through the new data module.

[0074] Through the above embodiments, newly added data tables can be implemented based on data modules. For example, creation, description and management of data tables from newly added data processing platforms can be implemented.

[0075] Furthermore, in one embodiment, in response to a newly connected first data processing platform, a new data table can be created on the newly connected first data processing platform based on a data module; specifically, according to metadata information corresponding to the target data module, a first data table is constructed on the newly connected first data processing platform, and then the business data of the first data processing platform is stored in the first data table according to the data structure information indicated by the metadata information.

[0076] Furthermore, in one embodiment, in response to a newly created first data module, a target data processing platform specified by the first data module is determined among multiple data processing platforms, and the data storage medium format corresponding to the target data processing platform is obtained, and then based on the data storage medium format of the database corresponding to the target data processing platform and the metadata information corresponding to the first data module, a second data table associated with the first data module is constructed in the database corresponding to the target data processing platform, wherein the data structure information of the second data table corresponds to the metadata information corresponding to the first data module.

[0077] In one embodiment, after obtaining multiple data tables from multiple data processing platforms, the electronic device can also determine multiple data processing tasks involved in multiple data processing platforms processing multiple data tables; then, it can parse the data processing statements corresponding to the multiple data processing tasks, and identify the data lineage relationship between the multiple data tables based on the parsing results of the data processing statements; then, it can construct a lineage link of the multiple data tables based on the data lineage relationship; wherein the lineage link includes: a lineage link between multiple data tables, a lineage link between fields in multiple data tables, and a lineage link between multiple data tables and multiple data processing tasks.

[0078] Among them, the data processing task refers to a link in the process of generating and processing data by the data processing platform, which may include multiple data processing statements, which are used to query data tables in the data processing platform or data tables in other data processing platforms, and save the queried data as a new data table after processing. It is understandable that the data processing platform can process the data table according to the data processing task, and the obtained data table is a downstream data table with a data blood relationship with the data table. Among them, the data processing platform can issue, manage, and schedule data processing tasks based on Airflow, Flink, or TBDS (Tencent Big Data Suite).

[0079] Exemplarily, the electronic device may periodically scan the code repository and the operating platform of the data processing platform to obtain a plurality of data processing tasks and data processing statements corresponding to the plurality of data processing tasks.

[0080] Exemplarily, the data processing statement may be at least one of SQL, HiveQL, and PySpark, but is not limited thereto. The data processing task may be data cleaning, data conversion, data analysis, etc., but is not limited thereto.

[0081] Exemplarily, the electronic device may first use a regular expression or a lexer to decompose a data processing statement into keywords, identifiers, operators, string constants and other words; then, the grammatical rules of the data processing statement may be obtained, and the words decomposed above may be identified according to the grammatical rules to determine the data tables, fields, data processing functions and logical information involved in the data processing statement; then, based on the data tables, fields, data processing functions and logical information involved above, the reference relationships between data tables, the reference relationships between fields and the data processing functions and logical information involved in each reference relationship may be determined; finally, the data lineage relationships between multiple data tables may be determined based on the above reference relationships and the data processing functions and logical information involved in each reference relationship.

[0082] For example, the electronic device can parse data processing statements based on the Hypertext Transfer Protocol (HTTP) microservice and ANTLR (ANother Tool for Language Recognition). Among them, each microservice involved in the HTTP microservice can be developed, deployed and expanded separately, can run on an independent process, and interact through a lightweight HTTP communication mechanism. It is a service with high development efficiency and maintainability. ANTLR is a tool for reading, processing, executing or translating structured text or binary files.

[0083] Specifically, the electronic device can first use the HTTP microservice to obtain the data processing statement; then, use ANTLR inside the HTTP microservice to define the syntax of the data processing statement, such as the syntax of the SQL statement; and generate the code of the lexical analyzer and the parser according to the above-defined syntax; then, the data processing statement can be converted into the parse tree (Parse Tree) and abstract syntax tree (AST) of ANTLR according to the code, wherein the parse tree is a direct tree representation of the data processing statement, and each node of the tree representation corresponds to a structure in the data processing statement, such as variable declaration, function call, control structure, etc., and the abstract syntax tree is a tree representation of the abstract syntax structure of the data processing statement, including information related to program semantics in the data processing statement; then, traverse the parse tree and the abstract syntax tree, and identify the involved elements in the parse tree and the abstract syntax tree, such as data table name, field name, etc.; then, the data lineage relationship between multiple data tables can be determined according to the identified elements; finally, the above data lineage relationship can be returned through the HTTP microservice.

[0084] Exemplarily, the bloodline link may be in the form of a directed graph, the nodes of the directed graph represent data tables, fields or data processing tasks, and the edges of the directed graph may represent the bloodline flow direction.

[0085] In one embodiment, Figure 3 As shown, assuming that the data processing platform is Hive, Hive includes data table 1, the electronic device can create and manage multiple data processing tasks based on Airflow, Flink or TBDS, and process multiple fields in data table 1 respectively based on the multiple data processing tasks to generate 4 new data tables, namely, 4 downstream data tables of data table 1: data table 2, data table 3, and data table 4.

[0086] For example, combined with Figure 3 , data table 2 includes field 6, field 6 is obtained by data processing task 1 processing field 1 and field 2 in data table 1, then the lineage links that can be constructed based on this data lineage relationship include: the lineage link between data table 1 and data table 2, the lineage link between field 1 and field 6, the lineage link between field 2 and field 6, and the lineage link between data table 1 and data processing task 1 and data table 1.

[0087] In one embodiment, the electronic device can associate the bloodline link with the data module based on the association relationship between the data table or the data table corresponding to the field and the above-mentioned data module (specifically, the field or data table in the bloodline link can be associated with the data module), so that the data link can be viewed and managed through the data module, for example, the bloodline relationship and data processing tasks involved in the data link can be viewed and managed.

[0088] In one embodiment, the electronic device can perform data lineage tracking based on the lineage link. Specifically, based on the lineage link, through the specified data table or the specified data module, the data table that has a lineage relationship or association relationship with the specified data table or the specified data module can be retrieved, so that the lineage information with a single data table or the lineage information of the entire data module can be queried, realizing a two-way tracking mechanism with more transparent and easy-to-manage data flow and dependency.

[0089] Exemplarily, the electronic device may respond to a search operation for a data table to be retrieved from multiple data tables, and determine the data processing tasks and associated data tables associated with the data table to be retrieved based on lineage links, specifically, based on lineage links between multiple data tables and lineage links between multiple data tables and multiple data processing tasks. Alternatively, the electronic device may respond to a search operation for a data module to be retrieved from multiple data modules, and determine the data table to be queried corresponding to the data module to be retrieved, specifically, based on the mapping relationship between the above-mentioned data module and the data table, and then, based on lineage links, specifically, based on lineage links between multiple data tables, determine the upstream data table and / or downstream data table that has a data lineage relationship with the data table to be queried.

[0090] For example, combining the above and Figure 3 Assuming that the data table to be retrieved is Table 2, it can be determined based on the lineage link related to Table 2 that the data processing task associated with Table 2 is Data Processing Task 1, and the data table associated with Table 2 is Table 1.

[0091] For example, combining the above and Figure 3 Assuming that the data module to be retrieved is data module 1, and the data tables associated with data module 1 are data table 3 and data table 4, it can be determined that the data tables to be queried are data table 3 and data table 4, and based on the data link, it can be determined that the upstream data table with data lineage relationship with data table 3 is data table 1, and the upstream data table with data lineage relationship with data table 4 is data table 1.

[0092] In one embodiment, Figure 4 As shown, in combination with the above embodiment, the user can send the above retrieval operation to the blood relationship query service of the electronic device based on the data management platform. The blood relationship query service can respond to the retrieval operation, obtain the blood relationship link from the blood relationship storage of the electronic device, and determine the data table related to the data table or having an upstream and downstream blood relationship based on the blood relationship link and the name of the data table involved in the retrieval operation (i.e., the name of the data table to be retrieved). Among them, the blood relationship storage service can write the extracted data blood relationship into the blood relationship storage, and the blood relationship storage can perform data modeling on the data blood relationship, obtain the blood relationship link, and store it in the graph database.

[0093] Specifically, the data processing platform can issue, manage, and schedule data processing tasks based on the respective service machines of Airflow, Flink, or TBDS. When the Airflow service machine, Flink service machine, or TBDS service machine processes data processing tasks, data processing statements, such as SQL statements, are involved. The SQL parsing service of the electronic device can obtain data processing statements from the Airflow service machine, Flink service machine, or TBDS service machine, and parse the obtained data processing statements based on the internal ANTLR to implement lineage extraction and obtain the data lineage relationship between data tables. The lineage storage service can extract the data lineage relationship.

[0094] In one embodiment, when obtaining a deletion operation for a data table, the electronic device can automatically determine the downstream data table that has a blood relationship with the data table based on the above-mentioned blood relationship link, and automatically notify the downstream user of the downstream data table. At the same time, the electronic device can also set a data table recycle bin function to store the data table to be deleted in the data table recycle bin before deleting it, so that after deleting the data table to be deleted, the data table to be deleted in the data table recycle bin can be restored through the recovery function. The problem of low efficiency caused by the need for developers to manually notify relevant downstreams when deleting data tables can be avoided, and the risk of data table deletion can be reduced through the recovery mechanism.

[0095] Exemplarily, the electronic device can respond to a delete operation on a data table to be deleted by determining a target downstream data table that has a data lineage relationship with the data table to be deleted based on the lineage link; determine a target data processing task between the data table to be deleted and the target downstream data table; and send a delete notification corresponding to the deletion operation to the task processor of the target data processing task.

[0096] The target downstream data table is obtained by processing the data table to be deleted according to the target data processing task.

[0097] For example, the deletion notification may be in the form of an email, text message, or software platform notification, but is not limited thereto.

[0098] Furthermore, table structure change is a stage that must be experienced during the database operation and maintenance process. When the amount of data in the database is huge and involves many data processing platforms, table structure change is more likely to cause data table omissions. Therefore, in one embodiment, in response to the change operation on the metadata information corresponding to the target data module, after determining the changed target metadata information and the target data table under the target data module, the step of adjusting the data structure information of the target data table according to the changed target metadata information can specifically include: determining the target field to be adjusted according to the changed target metadata information; traversing the table nodes in the bloodline link corresponding to the target data table according to the target field to be adjusted, and obtaining the table node path set of the target data table; determining the associated data table associated with the target data table according to the table node path set; determining the change order of the target data table and the associated data table based on the path information in the table node path set; finally, according to the changed target metadata information, adjusting the data structure information of the target data table and its associated data tables in the change order to update the target data table and its associated data tables.

[0099] Among them, the change operation for the metadata information corresponding to the target data module can specifically be the change of the data type, constraint field and other information of the corresponding data entity of the target field in the data table structure information; after obtaining the changed target metadata information, the changed target field can be determined from the target metadata information.

[0100] Specifically, a data table graph can be generated according to the bloodline link corresponding to the target data table, wherein a table node in the data table graph represents a data table, and the connection information between the table nodes indicates that the data tables represented by the nodes have a bloodline relationship, wherein the connection information includes table information of the data tables with bloodline relationships, such as table identifiers. For example, the bloodline link between data table A and data table B, whose connection information may be L(A, B); further, the connection information may also include field information with bloodline relationships, such as field identifiers. For example, the bloodline link between field a of data table A and field b of data table B, whose connection information may be L(Aa, Bb).

[0101] After determining the target data table associated with the target data module, in the data table graph corresponding to the bloodline link, first determine the target table node corresponding to the target data table, and then based on the target table node as the starting node, perform node traversal in the data table graph. During the node traversal process, the associated table node corresponding to the target table node can be determined based on the connection information between the target table node and the currently traversed candidate table node.

[0102] Specifically, the connection information between the target table node and the currently traversed candidate table node can be obtained to determine whether the connection information includes field information. If the connection information does not include field information, the changed field information can be matched with the field information in the data table corresponding to the candidate table node. If the same field information is matched, the candidate table node is determined as the first associated table node. If the connection information includes field information, the association degree between the target table node and the currently traversed candidate table node under the metadata information change is obtained according to the changed target field and the field information in the connection information. For example, the association degree can be calculated by the following formula (1):

[0103] corr(node1,node2)=α*corr(field1,field sub )+β*corr(field2,field sub ) Formula (1)

[0104] Among them, field1 represents the field information corresponding to the target table node in the connection information; field2 represents the field information corresponding to the candidate table node in the connection information; field subIndicates the target field to be changed; α and β are pre-set weight values, α is greater than β. Taking the bloodline link between field a of data table A and field b of data table B as an example, the connection information can be L(Aa, Bb), where field1 represents the field information a corresponding to the target table node A in the connection information, and field2 represents the field information b corresponding to the candidate table node B in the connection information.

[0105] Among them, corr(field1,field sub ) indicates the correlation between the field information corresponding to the target table node in the connection information and the target field that has changed, corr(field2,field sub ) represents the correlation between the field information corresponding to the candidate table node in the connection information and the target field that has changed; specifically, the correlation between the field information corresponding to the target table node or the candidate table node and the target field that has changed can be calculated by a string matching algorithm, such as a string search algorithm (knuthmorrispratt, KMP algorithm), etc.; the field information and the target field can also be converted into vector information, and the correlation between the field information corresponding to the target table node and the target field that has changed is calculated based on the vector information corresponding to the field information. Specifically, taking the correlation between the field information corresponding to the target table node and the target field that has changed as an example, it can be calculated by the following formula (2):

[0106]

[0107] Among them, X represents the vector information of the field information field1 corresponding to the target table node in the connection information, and Y represents the target field field that has been changed sub Vector information of .

[0108] Among them, corr(node1,node2) represents the correlation between the target table node and the currently traversed candidate table node. If corr(node1,node2) is greater than the preset correlation threshold, the candidate table node is determined as the first associated table node. If corr(node1,node2) is less than the preset correlation threshold, the candidate table node is determined to be a non-associated table node.

[0109] After obtaining the first association table node, you can continue to determine the first association table node as a new starting node, continue to traverse the nodes in the data table graph to obtain the first association table node associated with the new starting node, and repeat the above steps until there is no association table node corresponding to the starting node among the candidate nodes connected to the starting node in the data table graph.

[0110] After obtaining the associated table node of the target table node, the table node path corresponding to the target table node (i.e., the target data table) can be generated according to the associated table node and the traversal order of the associated table node, and finally the table node path set of all target data tables is obtained. After determining the target data table associated with the target data module, the target data table is further searched for the associated data table through the blood relationship with other data tables in the blood relationship link, combined with the target field that has changed, so as to change the data structure information of the data table with the field related to the target field that has changed, so as to ensure the consistency of the same field in different data tables.

[0111] After determining the table node path corresponding to the target data table, the associated data table having a direct association relationship and / or an indirect association relationship with the target data table can be determined according to the table nodes recorded in the table node path, and the change order of the target data table and the associated data table can be determined according to the order of each table node in the table node path, so as to adjust the target field in the target data table and the associated table field corresponding to the target field in the associated data table according to the change order, so as to update the data structure information of the target data table and its associated data table. Through the order of each table node recorded in the table node path, the changed fields in each target data table or associated data table are adjusted one by one to avoid missing the adjustment of the data structure information of any target data table or associated data table.

[0112] Furthermore, in the process of adjusting the target field in the target data table and the associated table field corresponding to the target field in the associated data table, when completing the adjustment of the target field of any target data table and / or completing the adjustment of the associated table field corresponding to the target field in any associated data table, the corresponding table node can be marked in the table node path corresponding to the target data table, so that in the event of an interruption, the adjustment progress mark at the time of the interruption can be obtained to determine which data tables in the target data table and / or the associated data table have had their data structure information adjusted successfully and which data tables have not been adjusted successfully, and then the data structure information of the data tables corresponding to the table nodes can be continued to be adjusted starting from the unmarked table nodes in the table node path, without having to readjust the data structure information of the data tables from scratch, thereby improving the management efficiency of the data structure information corresponding to the data tables.

[0113] In one embodiment, the managing the multiple data tables according to the multiple data modules includes: managing the multiple data tables according to the multiple data modules based on the data management system.

[0114] Among them, the data management system includes: a data module layer, which is used to read the metadata information of multiple data tables from multiple data processing platforms, and manage the multiple metadata information according to multiple data modules; a business database layer, which is used to collect and store data tables corresponding to the metadata information in the data module layer; and a derived data layer, which is used to manage downstream data tables that have a data lineage relationship with the data tables in the business database layer.

[0115] Exemplarily, the data module layer can connect to the business database, read the metadata information of the data table, and store and manage multiple metadata information in the form of a dictionary according to the data module.

[0116] The business database layer can obtain and store data tables with configured metadata information from the data warehouse, that is, data tables corresponding to the data modules created in the data module layer. The data warehouse can include data tables stored in the business database.

[0117] Downstream data tables in the derived data layer that have a data lineage relationship with data tables in the business database layer can be intermediate tables in the calculation process, fact tables generated based on analysis, result tables, dimension tables, etc.

[0118] Figure 5 A schematic diagram of a data management device 500 provided in an embodiment of the present application.

[0119] like Figure 5 As shown, the data management device 500 includes: a first acquisition module 501, a first determination module 502, a construction module 503, a management module 504, a second acquisition module 505, a second determination module 506, a third determination module 507, a fourth determination module 508, a parsing and identification module 509, a blood relationship construction module 510, a retrieval association module 511, a fifth determination module 512, a sixth determination module 513, and a sending module 514, wherein:

[0120] In one embodiment, a first acquisition module 501 is used to acquire multiple data tables from multiple data processing platforms; a first determination module 502 is used to determine metadata information corresponding to each data table based on the data structure information of each of the multiple data tables; a construction module 503 is used to construct data modules based on the metadata information of the data tables under the same data structure information in the multiple data tables to obtain multiple data modules; a management module 504 is used to manage the multiple data tables based on the multiple data modules.

[0121] In one embodiment, the management module 504 is specifically used to: determine the changed target metadata information and the target data table under the target data module in response to the change operation of the metadata information corresponding to the target data module in multiple data modules; and adjust the data structure information of the target data table according to the changed target metadata information.

[0122] In one embodiment, the second acquisition module 505 is used to: acquire the newly added data table in response to the creation operation of the newly added data table; the second determination module 506 is used to: determine the metadata information of the newly added data table based on the data structure information of the newly added data table; the third determination module 507 is used to: determine the target data module corresponding to the data structure information of the newly added data table in multiple data modules based on the metadata information of the newly added data table; the management module 504 is also used to: manage the newly added data table through the target data module.

[0123] In one embodiment, the fourth determination module 508 is used to: determine multiple data processing tasks involved in multiple data processing platforms processing multiple data tables; the parsing and identification module 59 is used to: parse the data processing statements corresponding to the multiple data processing tasks, and identify the data lineage relationship between the multiple data tables based on the parsing results of the data processing statements; the lineage construction module 510 is used to: construct lineage links of multiple data tables based on the data lineage relationship; wherein the lineage links include: lineage links between multiple data tables, lineage links between fields in multiple data tables, and lineage links between multiple data tables and multiple data processing tasks.

[0124] In one embodiment, the retrieval association module 511 is used to: in response to a retrieval operation for a data table to be retrieved among multiple data tables, determine the data processing tasks and associated data tables associated with the data table to be retrieved according to the bloodline link; or, in response to a retrieval operation for a data module to be retrieved among multiple data modules, determine the data table to be queried corresponding to the data module to be retrieved, and determine the upstream data table and / or downstream data table that has a data bloodline relationship with the data table to be queried according to the bloodline link.

[0125] In one embodiment, the fifth determination module 512 is used to: in response to a delete operation on the data table to be deleted, determine the target downstream data table that has a data lineage relationship with the data table to be deleted according to the lineage link; the sixth determination module 513 is used to: determine the target data processing task between the data table to be deleted and the target downstream data table; the sending module 514 is used to: send the delete notification corresponding to the delete operation to the task processor of the target data processing task.

[0126] In one embodiment, the management module 504 is specifically used to: manage multiple data tables according to multiple data modules based on the data management system; wherein the data management system includes: a data module layer, used to read the metadata information of multiple data tables from multiple data processing platforms, and manage the multiple metadata information according to multiple data modules; a business database layer, used to collect and store data tables corresponding to the metadata information in the data module layer; a derived data layer, used to manage downstream data tables that have a data lineage relationship with the data tables in the business database layer.

[0127] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 5 The device 500 shown can execute the above method embodiments, and the above and other operations and / or functions of each module in the device 500 are respectively for implementing the corresponding processes in the above methods, which will not be repeated here for the sake of brevity.

[0128] The above describes the device 500 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0129] Figure 6 A schematic block diagram of an electronic device 600 provided in an embodiment of the present application.

[0130] like Figure 6 As shown, the electronic device 600 may include:

[0131] The memory 610 and the processor 620, the memory 610 is used to store the computer program and transmit the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the method in the embodiment of the present application.

[0132] For example, the processor 620 may be configured to execute the above method embodiments according to instructions in the computer program.

[0133] In some embodiments of the present application, the processor 620 may include but is not limited to:

[0134] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0135] In some embodiments of the present application, the memory 610 includes but is not limited to:

[0136] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be ROM, programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically erasable programmable read-only memory (Electric ally EPROM, EEPROM) or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDR AM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ES DRAM), synchronous connection dynamic random access memory (Synch Link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0137] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0138] like Figure 6 As shown, the electronic device may also include:

[0139] The transceiver 630 may be connected to the processor 620 or the memory 610 .

[0140] The processor 620 may control the transceiver 630 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include an antenna, and the number of antennas may be one or more.

[0141] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0142] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0143] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instruction is loaded and executed on a computer, the computer can be made to perform the corresponding flow in each method in the embodiment of the present application in whole or in part, and generate the functions that can be realized by each method in the embodiment of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0144] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0146] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0147] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data management method, characterized in that: include: Obtain multiple data tables from multiple data processing platforms; Determining metadata information corresponding to each of the data tables according to the data structure information of each of the multiple data tables; According to the metadata information of the data tables under the same data structure information in the multiple data tables, a data module is constructed to obtain multiple data modules; The plurality of data tables are managed according to the plurality of data modules.

2. The method according to claim 1, characterized in that The managing the multiple data tables according to the multiple data modules includes: In response to a change operation on metadata information corresponding to a target data module among the multiple data modules, determining the changed target metadata information and a target data table under the target data module; The data structure information of the target data table is adjusted according to the changed target metadata information.

3. The method according to claim 1, characterized in that Also includes: In response to a creation operation of a new data table, acquiring the new data table; Determining metadata information of the newly added data table according to the data structure information of the newly added data table; According to the metadata information of the newly added data table, determining a target data module corresponding to the data structure information of the newly added data table from among the multiple data modules; The newly added data table is managed through the target data module.

4. The method according to any one of claims 1 to 3, characterized in that: After obtaining multiple data tables from multiple data processing platforms, the method further includes: Determining a plurality of data processing tasks involved in processing the plurality of data tables by the plurality of data processing platforms; Parsing the data processing statements corresponding to the multiple data processing tasks, and identifying the data kinship relationship between the multiple data tables according to the parsing results of the data processing statements; Constructing a bloodline link of the plurality of data tables according to the data bloodline relationship; The lineage links include: lineage links between the multiple data tables, lineage links between fields in the multiple data tables, and lineage links between the multiple data tables and the multiple data processing tasks.

5. The method according to claim 4, characterized in that Also includes: In response to a search operation on a data table to be searched among the multiple data tables, determining a data processing task and an associated data table associated with the data table to be searched according to the lineage link; or, In response to a search operation for a data module to be retrieved among the multiple data modules, a data table to be queried corresponding to the data module to be retrieved is determined, and an upstream data table and / or a downstream data table having a data lineage relationship with the data table to be queried is determined according to the lineage link.

6. The method according to claim 4, characterized in that Also includes: In response to a deletion operation on the data table to be deleted, determining a target downstream data table having a data lineage relationship with the data table to be deleted according to the lineage link; Determine a target data processing task between the to-be-deleted data table and the target downstream data table; A deletion notification corresponding to the deletion operation is sent to a task processor of the target data processing task.

7. The method according to any one of claims 1 to 3, characterized in that: The managing the multiple data tables according to the multiple data modules includes: Based on the data management system, managing the multiple data tables according to the multiple data modules; Wherein, the data management system comprises: A data module layer, used for reading metadata information of each of the plurality of data tables originating from the plurality of data processing platforms, and managing the plurality of metadata information according to the plurality of data modules; The business database layer is used to collect and store data tables corresponding to the metadata information in the data module layer; The derived data layer is used to manage downstream data tables that have a data lineage relationship with the data tables in the business database layer.

8. A data management device, characterized in that: include: A first acquisition module is used to: acquire multiple data tables from multiple data processing platforms; A first determining module, configured to: determine metadata information corresponding to each of the data tables according to data structure information of each of the plurality of data tables; A construction module is used to: construct a data module according to metadata information of the data tables under the same data structure information in the multiple data tables to obtain multiple data modules; The management module is used to manage the multiple data tables according to the multiple data modules.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images