Data management method

Abstract

The invention discloses a data management method comprising the following steps: obtaining metadata from a first data node in a data flow link, the first data node comprising a data warehouse data node and at least one target data node except the data warehouse data node, the at least one target data node comprises a micro-service related data node and / or a data calculation engine related data node; and performing data consanguinity analysis processing according to the metadata to obtain data consanguinity information. In this way, metadata is obtained from a plurality of data nodes such as the data warehouse data nodes in the data flow link and the data nodes related to the micro-service and / or the data nodes related to the data calculation engine and the like, and data blood relationship analysis is carried out, so that the metadata used for data blood relationship analysis is more comprehensive; the integrity and accuracy of the obtained data blood relationship information can be improved, so that the management effects on the aspects of data life cycle, data quality, data security, data governance and the like can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data management method. Background Art

[0002] The production and life activities in today's society generate a large amount of data. The data flow process includes data production, processing, storage / analysis, consumption and other links. The entire data flow chain is long, and there are many flow links and stakeholders involved. Therefore, the data generated comes from a wide range of sources, large volumes, and multiple scenarios, and the relationships between data are complex. Managing this data helps related companies better manage data assets, ensure data quality and security, and also helps to troubleshoot and resolve data issues. For example, data management is achieved through data lineage analysis of the metadata of these data. Data lineage is also called data origin or data genealogy. It refers to a naturally formed relationship between data in the life cycle of data, from data generation, processing, processing, fusion, circulation to final extinction, similar to human blood relationship. Therefore, data lineage can be used to trace the source, operation and flow of data. Currently, data lineage analysis is only performed based on the metadata in the data node of the big data platform data warehouse related to the data storage / analysis link involved in the data flow link. There is a problem that the metadata for data lineage analysis is relatively single, which will affect the integrity and accuracy of the obtained data lineage information, and further affect the management effect of data life cycle, data quality, data security and data governance. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that the metadata for data lineage analysis in the prior art is relatively single, which affects the integrity and accuracy of the obtained data lineage information, and further affects the management effect on data life cycle, data quality, data security and data governance. It is possible to obtain more comprehensive metadata from multiple data nodes, such as data warehouse data nodes and at least one target data node other than the data warehouse data node, in the data flow link to obtain more complete and accurate data lineage information, which can better help solve problems in data life cycle, data quality, data security and data governance, and improve the management effect on data life cycle, data quality, data security and data governance.

[0004] To solve the above technical problems, an embodiment of the present invention discloses a data management method, which includes: obtaining metadata from a first data node in a data flow link, the first data node includes a data warehouse data node and at least one target data node other than the data warehouse data node, and the at least one target data node includes a microservice-related data node and / or a data computing engine-related data node; performing data lineage analysis and processing according to the metadata to obtain data lineage information.

[0005] By adopting the above technical solution, data lineage analysis is performed by obtaining metadata from multiple data nodes such as microservice-related data nodes and / or data computing engine-related data nodes other than data warehouse data nodes related to the data storage / analysis link in the data flow link. This can make the metadata used for data lineage analysis more comprehensive, thereby improving the integrity and accuracy of the obtained data lineage information, thereby improving the management effects on data life cycle, data quality, data security, and data governance.

[0006] Furthermore, since microservices and data computing engines will perform corresponding data processing in the production, processing, storage / analysis, and consumption of data, obtaining corresponding metadata from data nodes related to microservices and / or data nodes related to data computing engines can obtain more comprehensive metadata in the data flow link, which can be used to obtain more complete and accurate data lineage information, and can better improve the management effects of data life cycle, data quality, data security, and data governance.

[0007] In addition, since both microservices and data computing engines are used to perform corresponding data processing in the production, processing, storage / analysis, and consumption of data, which basically involves the full-link processing of the data flow link, the metadata obtained can be considered as full-link metadata, and the generated data lineage information is full-link data lineage. Therefore, based on this implementation method, more complete and accurate full-link data lineage information can be obtained, which can better improve the management effects of data life cycle, data quality, data security, and data governance.

[0008] According to another specific embodiment of the present invention, the metadata includes first metadata obtained from a data warehouse data node and second metadata obtained from at least one target data node, and data lineage analysis is performed based on the metadata to obtain data lineage information, including: performing data lineage analysis based on the first metadata and the second metadata obtained at the first time to obtain data lineage information including metadata and association relationships between metadata.

[0009] According to another specific embodiment of the present invention, the method further includes: updating the data lineage information according to the first metadata acquired at a second time after the first time, to obtain updated data lineage information.

[0010] According to another specific embodiment of the present invention, data lineage information is updated based on first metadata obtained at a second time after the first time to obtain updated data lineage information, including: verifying metadata included in the data lineage information based on the first metadata obtained at the second time after the first time to obtain a verification result; and correcting the data lineage information based on the verification result to obtain updated data lineage information.

[0011] In this way, more complete and accurate data lineage information can be obtained, which can better improve the management effects in terms of data life cycle, data quality, data security and data governance.

[0012] According to another specific embodiment of the present invention, the method further includes: displaying data lineage information, so that users can conveniently view metadata information and other information of data nodes in the data flow link based on the data lineage information, and perform corresponding processing or data management.

[0013] According to another specific embodiment of the present invention, the method further comprises: implementing the first data processing according to the data lineage information. In this way, corresponding management processing can be conveniently implemented based on the data lineage information.

[0014] According to another specific embodiment of the present invention, the first data processing includes at least one of the following processing: metadata query processing; data lineage analysis processing; data impact analysis processing; data anomaly monitoring and alarm processing.

[0015] According to another specific embodiment of the present invention, performing data lineage analysis on metadata to obtain data lineage information includes: performing data lineage analysis on metadata based on a data lineage analysis engine to obtain data lineage information. In this way, data lineage information can be obtained conveniently.

[0016] According to another specific embodiment of the present invention, the method further includes: acquiring service data from a second data node in the data flow link; and performing second data processing according to the service data and data lineage information. The second data node may be the same as or different from the first data node.

[0017] In this way, corresponding management processing can be easily implemented based on data lineage information and business data.

[0018] According to another specific embodiment of the present invention, the second data processing includes at least one of the following processing: data archiving and delisting processing; cold and hot data analysis and optimization processing; task analysis and task scheduling optimization processing; abnormal data alarm processing.

[0019] According to another specific embodiment of the present invention, the second data processing is implemented according to the business data and the data lineage information, including: inputting the business data and the data lineage information into a data management model, and performing the second data processing through the data management model.

[0020] According to another specific embodiment of the present invention, the data management model includes a data value model, a hot and cold data model, a task analysis model, and an abnormal data model. The business data and data lineage information are input into the data management model, and the second data processing is performed through the data management model, including: inputting the business data and data lineage information into the data value model, the hot and cold data model, the task analysis model, and the abnormal data model, respectively, and the data value model, the hot and cold data model, the task analysis model, and the abnormal data model respectively perform data archiving and delisting processing, hot and cold data analysis and optimization processing, task analysis and task scheduling optimization processing, and abnormal data alarm processing.

[0021] According to another specific embodiment of the present invention, the data warehouse data nodes are data nodes that implement data processing based on Hive technology, and the data nodes related to the data computing engine include data nodes that implement data processing based on Flink technology and data nodes that implement data processing based on Spark technology.

[0022] In a second aspect, an embodiment of the present invention further discloses an electronic device, comprising: a memory for storing a computer program, the computer program comprising program instructions; a processor for executing the program instructions so that the electronic device executes the data management method provided by any possible embodiment described above.

[0023] In a third aspect, an embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and the program instructions are executed by an electronic device to execute the data management method provided by any possible embodiment described above.

[0024] In a fourth aspect, an embodiment of the present invention further discloses a computer program product, including a computer program / instruction, which implements the data management method provided by any possible embodiment described above when the computer program / instruction is executed by an electronic device.

[0025] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can also be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a flow chart of a data management method provided by an embodiment of the present invention is shown;

[0027] Figure 2 A schematic diagram of a process for obtaining data lineage information provided by an embodiment of the present invention is shown;

[0028] Figure 3 A schematic diagram of a flow chart of another data management method provided by an embodiment of the present invention is shown;

[0029] Figure 4 A schematic diagram of a flow chart of another data management method provided by an embodiment of the present invention is shown;

[0030] Figure 5 A process schematic diagram of another data management method provided by an embodiment of the present invention is shown;

[0031] Figure 6 A schematic diagram showing a process of a method for generating and applying data lineage information provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram showing a process of a node data storage method provided by an embodiment of the present invention is shown;

[0033] Figure 8 A schematic diagram of a process of a node data query method provided by an embodiment of the present invention is shown;

[0034] Fig. 9 A process schematic diagram of a data governance method provided by an embodiment of the present invention is shown;

[0035] Fig.10 A schematic diagram of a process of an invalid data / task analysis and cleaning method provided by an embodiment of the present invention is shown;

[0036] Fig.11 A schematic diagram of a process of a cold and hot data analysis and optimization method provided by an embodiment of the present invention is shown;

[0037] Fig.12 A process schematic diagram of a data security and auditing method provided by an embodiment of the present invention is shown;

[0038] Fig.13 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0039] As mentioned above, the data flow link is complex, the data generated comes from a wide range of sources, the amount is large, and there are many scenarios. The relationship between the data is complex, and the entire data flow link is long, passing through many links and related parties. However, currently, only the big data platform data warehouse (such as Hive data warehouse) related to the data storage / analysis link involved in the data flow link has the metadata lineage analysis function. It performs data lineage analysis based on the metadata in the data node of the big data platform data warehouse, which makes the data lineage analysis incomplete and inaccurate, affecting the management effect of data life cycle, data quality, data security and data governance. Problems caused by incomplete and inaccurate data lineage analysis include the following aspects: First, it makes data traceability difficult. When abnormal data fluctuations occur, it is difficult to quickly locate the problem link, and the problem troubleshooting is difficult and costly. Second, it cannot capture data anomalies in time, and can only passively accept errors reported by the business party. Some data anomalies cannot be identified, which will affect business decisions. Third, it affects data quality and compliance issues, and lacks necessary guarantee means for data security, integrity, consistency, etc.

[0040] To solve these problems, the embodiments of the present application provide a data management method, such as Figure 1 As shown, the method comprises the following steps:

[0041] S100, obtaining metadata from a first data node in a data flow link, where the first data node includes a data warehouse data node and at least one target data node other than the data warehouse data node, and the at least one target data node includes a microservice-related data node and / or a data computing engine-related data node.

[0042] S200, performing data lineage analysis and processing according to metadata to obtain data lineage information.

[0043] Among them, the data warehouse data nodes are, for example, data nodes that implement data processing based on Hive technology, and the data computing engine-related data nodes include, for example, data nodes that implement data processing based on Flink technology and data nodes that implement data processing based on Spark technology.

[0044] By adopting this data management method, metadata is obtained from multiple data nodes such as microservice-related data nodes and / or data computing engine-related data nodes other than data warehouse data nodes related to data storage / analysis links in the data flow link for data lineage analysis. This can make the metadata used for data lineage analysis more comprehensive, thereby improving the integrity and accuracy of the obtained data lineage information, thereby improving the management effects on data life cycle, data quality, data security, and data governance.

[0045] Furthermore, since microservices and data computing engines are used to perform corresponding data processing in the production, processing, analysis, and consumption of data, obtaining corresponding metadata from data nodes related to microservices and / or data nodes related to data computing engines can obtain more comprehensive metadata, which can be used to obtain more complete and accurate data lineage, and can better improve the management effects of data life cycle, data quality, data security, and data governance.

[0046] In addition, since both microservices and data computing engines are used to perform corresponding data processing in the production, processing, analysis, and consumption of data, which basically involves the full-link processing of the data flow link, the metadata obtained can be considered as full-link metadata, and the generated data lineage information is full-link data lineage. Therefore, based on this implementation method, a more complete and accurate data lineage of the entire link can be obtained, which is a full-link data lineage implementation method that can better improve the management effect of data asset-related aspects such as data life cycle, data quality, data security, data value, and data governance.

[0047] Furthermore, metadata can be periodically acquired according to a preset metadata acquisition cycle. Figure 2 As shown, data lineage analysis is performed based on metadata to obtain data lineage information, which can be:

[0048] S210, performing data lineage analysis and processing according to the first metadata and the second metadata acquired at the first time, to obtain data lineage information including metadata and association relationships between metadata.

[0049] Furthermore, the data management provided by the implementation mode of the present application further includes: updating the data lineage information according to the first metadata acquired at a second time after the first time, to obtain updated data lineage information.

[0050] like Figure 2 As shown, the data lineage information is updated according to the first metadata obtained at the second time after the first time to obtain the updated data lineage information, which may be:

[0051] S220, performing verification processing on the metadata included in the data lineage information according to the first metadata obtained at a second time after the first time, to obtain a verification processing result.

[0052] S230, correcting the data lineage information according to the verification result to obtain updated data lineage information.

[0053] The first time and the second time are two metadata acquisition times corresponding to the metadata acquisition period, and the value of the metadata acquisition period can be specifically set as needed. In this way, by periodically updating and generating data lineage information, more complete and accurate data lineage information can be obtained.

[0054] Furthermore, if Figure 3 After obtaining the data lineage information, the data management provided by the implementation mode of the present application further includes the following steps:

[0055] S300, displaying data lineage information; and / or implementing first data processing according to the data lineage information.

[0056] The first data processing may include, for example, at least one of metadata query processing, data lineage analysis processing, data impact analysis processing, and data anomaly monitoring and alarm processing. Of course, the first data processing may also include other processing.

[0057] In this way, data lineage information can be displayed so that users can view relevant information or perform corresponding operations, and / or conveniently implement the first data processing, improve the management effect of data life cycle, data quality, data security and data governance, and improve user experience.

[0058] Furthermore, if Figure 4 As shown, the data management provided by the implementation mode of the present application also includes the following steps:

[0059] S400, obtaining service data from a second data node in the data transfer link.

[0060] S500, implementing second data processing based on business data and data lineage information.

[0061] The second data node may be the same as or different from the aforementioned first data node, and may be set as required.

[0062] Implementing the second data processing according to the business data and the data lineage information includes: inputting the business data and the data lineage information into the data management model, and performing the second data processing through the data management model.

[0063] The second data processing may include, for example, at least one of data archiving and delisting, cold and hot data analysis and optimization, task analysis and task scheduling optimization, and abnormal data alarm processing. The corresponding data management model includes a data value model, a cold and hot data model, a task analysis model, and an abnormal data model.

[0064] Exemplarily, business data and data lineage information are respectively input into the data value model, hot and cold data model, task analysis model, and abnormal data model, which respectively perform data archiving and delisting processing, hot and cold data analysis and optimization processing, task analysis and task scheduling optimization processing, and abnormal data alarm processing.

[0065] Of course, the second data processing may also include other processing, and the corresponding data management model may also include other models.

[0066] In this way, the data management method provided by the implementation of the present application can realize the second data processing through data lineage and combined with business data, for example, it can realize the full-link business data query and alarm function. And by customizing the aforementioned model and combining data lineage, the data governance capability can be realized, including data storage optimization, data task optimization and other data asset optimization. By analyzing data lineage, valuable data can be identified and data quality can be continuously optimized. It can improve the management effect of data life cycle, data quality, data security and data governance, and improve user experience.

[0067] The following will take the data flow link in the vehicle field as an example and combine with the accompanying drawings to further illustrate the data management method provided in the implementation mode of this application.

[0068] like Figure 5 As shown, the entire data flow chain in the vehicle field (i.e., the entire data chain) includes data production, data processing, data storage / analysis, data consumption, etc. The data management method provided in this implementation is applied to a data management system, which includes relevant modules in the entire data flow chain, as well as a data analysis engine (including a data lineage analysis engine), data middleware, and a data governance platform.

[0069] Among them, the data production link in the entire link of data flow includes the data production source (i.e., data source), and the data production source includes, for example, the car, car-related mobile phone applications (applications, apps), and other more data production sources.

[0070] The data processing stage includes data processing tools, such as ETL (Extract-Transform-Load). ETL can collect logs from data production sources such as vehicle and mobile phones through vehiclelog, applog, binglog, etc., obtain data from the logs corresponding to the data production sources, perform data conversion, analysis, data verification, cleaning and other data processing, and obtain metadata and business data. Data that has passed the data processing stage can enter the data storage / analysis stage.

[0071] The data storage / analysis link includes a data warehouse and data feature engineering / data model. The data warehouse is used to store data processed by ETL. The data warehouse includes, for example, an operational data store (ods) layer, a data warehouse (dw) layer, a data mart layer (dm), etc. In addition, the data warehouse is, for example, a Hive data warehouse implemented based on Hive technology. The data feature engineering / data model is used to perform corresponding analysis on relevant data such as data processed by ETL. The data feature engineering / data model can be implemented based on a data governance platform, for example, and the data governance platform can implement data governance processing (Actor) such as intelligent monitoring and alarm, data value assessment, data delisting analysis, data security analysis, and task optimization analysis. Correspondingly, the data governance platform can establish corresponding data feature engineering / data models (Models) to perform corresponding analysis on the data and implement corresponding data governance capabilities. For example, the data governance platform can include a monitoring and alarm model, a data value model, a data delisting model, a data security model, a task processing model, etc., to implement corresponding data governance processing. In this way, by establishing relevant data feature engineering and data models to conduct model analysis on data, relevant data governance capabilities such as data value assessment, data security, and data value can be achieved. Data that has passed the data storage / analysis stage can enter the data consumption stage.

[0072] The data consumption link includes data application scenarios and business application scenarios. Data application scenarios include application processing such as data link generation, data report generation, and data query. Business application scenarios include application processing such as intelligent alarm, task analysis, and data analysis.

[0073] Data middleware includes message middleware, data storage middleware, data processing middleware, etc., which can produce, store and process corresponding data. Among them, message middleware includes Kafka, rocketMq, etc., data storage middleware includes mqsql, redis, hadoop, hbase, elasticsearch, clickhouse, etc., data processing middleware includes Spark, Flink, Hive, data gateway, springCloud, xxljob, etc.

[0074] In the data management method provided in this embodiment, the data analysis engine can obtain the corresponding data from the aforementioned data production link, data processing link, data storage / analysis link, data consumption link and related data middleware, perform data extraction and conversion, analysis and other processing, obtain the corresponding metadata (for example, the data lineage analysis engine obtains metadata), and perform data lineage analysis processing based on the metadata to obtain data lineage information such as data lineage graph. Further, the data lineage information can also be sent to the data governance platform for corresponding processing. In addition, the data analysis engine can also obtain business data from the obtained data (for example, the business data analysis engine obtains business data), and send the business data to the data governance platform for corresponding processing. Alternatively, if the data analysis engine only includes the data lineage analysis engine, the data governance platform can obtain metadata or data lineage information from the data lineage analysis engine, obtain business data from the data middleware and other locations, and perform corresponding processing based on the metadata or data lineage information, as well as the business data. In addition, according to the processing results obtained by the data governance platform, the relevant information in the entire link of data flow can be reversely processed. And, the data consumption link can also implement corresponding data consumption processing based on the data lineage information obtained by the data analysis engine. In this way, data-related applications, data lineage, monitoring alarms and other related processing can be finally realized.

[0075] Furthermore, in the data management method provided by this embodiment, after the data lineage information is generated, the data lineage information can also be displayed according to the user operation to realize a visualized data lineage relationship link, so that the user can view the metadata information of the data nodes in the data flow link and other information. In this way, it is convenient for the user to view the corresponding information based on the data lineage information and perform corresponding management.

[0076] Exemplarily, the main capabilities that can be achieved based on the data management method provided in this embodiment include the following capabilities:

[0077] (a) Data tracing and traceability (i.e., data observability): Based on the display of data lineage information, a visualized data lineage relationship link is realized, so that users can analyze the source and flow of specified data, and can help users understand the meaning of data, locate data problems in the whole process, perform root cause analysis, and perform data association impact analysis, etc., to improve the efficiency and accuracy of troubleshooting. Therefore, the data analysis function can be better realized.

[0078] (b) Abnormal monitoring and alarm: Based on the above processing, the abnormal conditions in the data flow link can be monitored in real time, such as data delay, interruption, error, etc., as well as the impact and scope on downstream nodes. When an abnormality is detected in the scheduling of the link node, an alarm is issued to all downstream nodes of the current node. Therefore, the intelligent alarm function can be better realized.

[0079] (c) Data value assessment: Data lineage provides a value assessment method and basis based on the actual application of data. The more data business audiences, the larger the data update volume, and the more frequent the data update, the more valuable it is. Therefore, data governance functions can be better implemented based on the data governance platform.

[0080] (d) Data security compliance and auditing: Complete data lineage information can effectively track the source, processing methods and protection measures of sensitive data, ensuring that data has achieved security compliance throughout the entire life cycle. Specifically, the nodes through which high-security data flows are analyzed to see whether the security processing of each node meets the security compliance requirements. If not, an alarm will be triggered and rectification will be prompted. Therefore, data governance functions can be better implemented based on the data governance platform.

[0081] (e) Data quality management: Data lineage analysis can help users discover problems such as data inconsistency, redundancy or omission. By tracing the source and flow path of data, users can find the root cause of data quality problems, clean and optimize data, improve data quality, and support the accuracy of business decisions. Therefore, data governance functions can be better implemented based on the data governance platform.

[0082] (f) Providing a basis for archiving / removing data: Data lineage records the whereabouts of data, and can clearly understand how the data is used. If a piece of data has no business consumption, the data can be further evaluated and considered for archiving or removal, thereby reducing data storage costs and improving the quality of data assets. Therefore, data analysis functions can be better realized.

[0083] Of course, based on the data management method provided in this embodiment, other capabilities can also be achieved, such as the aforementioned data lineage information generation, business data query and search, data governance and other related functions, which can be set as needed and will not be explained one by one here.

[0084] The following is a further explanation of the process of obtaining data lineage information and related processing.

[0085] like Figure 6 As shown, exemplarily, the aforementioned data lineage analysis engine includes a data parsing and processing layer, which obtains metadata from a metadata data source, performs corresponding processing, and can obtain data lineage information.

[0086] Exemplarily, in one implementation of the present application, the metadata data source (i.e., the first data node) includes a data node that implements data processing based on Hive technology (as an example of a data warehouse data node), a data node that implements data processing based on Flink technology, a data node that implements data processing based on Spark technology, and a microservice-related data node (these three data nodes are an example of at least one target data node other than the data warehouse data node). The metadata obtained includes Hive metadata, Flink metadata, Spark metadata, and microservice metadata, with Hive metadata being an example of the first metadata, and Flink metadata, Spark metadata, and microservice metadata being an example of the second metadata.

[0087] The data analysis and processing layer, for example, collects Hive metadata to the metadata storage node of Atlas in real time through the Hive plug-in of Atlas, collects Flink metadata to the metadata storage node of Atlas in real time through the Flink plug-in of Atlas, and collects Spark metadata to the metadata storage node of Atlas in real time through the Spark plug-in of Atlas. The metadata storage node can be, for example, a Kafka data node related to the Kafka component, which can be called the first metadata storage node. In addition, obtaining metadata through the aforementioned plug-in can also be understood as the aforementioned plug-in pushing the corresponding metadata to the metadata storage node.

[0088] In addition, the microservice metadata of the data nodes related to the microservice is collected from the data nodes related to the microservice by the log collection component (such as the filebeat component) based on the Java Agent plug-in through the business point software development kit (software development kit, sdk), and stored in the corresponding metadata storage node. The metadata storage node can be, for example, another Kafka data node related to the Kafka component, which can be called the second metadata storage node.

[0089] The data analysis and processing layer obtains Hive metadata, Flink metadata, and Spark metadata from the first metadata storage node, and reads microservice metadata from the second metadata storage node through the log conversion service based on the Application Programming Interface (API). Then, the data analysis and processing layer analyzes the metadata in real time and builds metadata relationships (relation) based on the metadata information collected by the Atlas backend service based on the corresponding plug-in, obtains data lineage information, and stores the obtained data lineage information in the Atlas graph database. The data lineage information can be, for example, a data lineage graph. In this way, by collecting metadata for the entire link through the Agent and Atlas products, a full-link data lineage map can be obtained.

[0090] Furthermore, the data analysis processing layer can store the acquired metadata in another metadata storage node, which can be another Kafka data node related to the Kafka component, and can be called a third metadata storage node, so that the data lineage service can implement corresponding services based on the metadata stored in the metadata storage node. In addition, if the acquired metadata changes, the data analysis processing layer sends the changed metadata to the metadata storage node.

[0091] Furthermore, the data analysis and processing layer can use the obtained data lineage information as the initialization information of the data lineage service (i.e., the self-developed data lineage service), import it into the data lineage service, and implement the initialization processing of the data lineage service, so as to implement the data lineage service based on the data lineage information. The data lineage service, for example, includes processing such as metadata query, data lineage analysis, data impact analysis, and data anomaly monitoring and alarming through the user interface (UserInterface, UI) (as an example of the first data processing). And, the relevant data corresponding to the data lineage service is stored in the mysql database. In addition, the data lineage service can also include processing such as displaying a data lineage diagram through the UI.

[0092] Furthermore, the data analysis and processing layer also includes a data lineage timing task. The data lineage timing task periodically obtains Hive metadata from Hive-related data nodes through an API interface based on the timing task (for example, according to the metadata acquisition cycle of T+1). This data acquisition method can be understood as an offline acquisition method. Then, the relevant metadata information in the mysql database is checked and corrected according to the obtained Hive metadata, and the data lineage information stored in the Atlas graph database and / or the relevant metadata information in the mysql database is checked and processed to obtain the verification processing results, and the data lineage information is corrected according to the verification processing results to obtain updated data lineage information. In addition, data lineage services can also be implemented based on the updated data lineage information.

[0093] The following is a further explanation of the query and search technical solutions for the full-link business data involved in the data management method provided in this application.

[0094] like Figure 7 As shown, exemplarily, the storage of node data (i.e., business data) in the data flow link includes two links: data storage related to the digital twin link and dynamic data collection (i.e., dynamic collection) storage related to the existing data link.

[0095] Among them, the digital twin link involves the vehicle operation end, command center, command parsing service, shadow service and other links. Currently, the data storage related to the digital twin link only has the Kafka data node between the command parsing service and the shadow service. In one implementation of this solution, three new Kafka data nodes are added, including a Kafka data node between the vehicle operation end and the command center, a Kafka data node between the command center and the command parsing service, and a Kafka data node after the shadow service. The business data in these four Kafka data nodes is collected through Flink and stored synchronously in the Operation Support Systems (OSS).

[0096] The dynamic data collection and storage related to the existing data link involves the vehicle operation end, gateway (i.e. dynamic collection gateway), parsing service (i.e. dynamic collection parsing service), Flink, Hive processing (Process) service and other links, as well as the Kafka data nodes between these links. Based on the dynamic collection of business data of the corresponding links by Flink, the business data of the corresponding Kafka data nodes is obtained and stored in OSS. In this way, the data before and after parsing in the existing data link can be synchronized to OSS.

[0097] The Kafka data node related to the above digital twin link and the existing data link is used as an example of the second data node. In this way, multiple business data can be obtained for better data management.

[0098] In addition, if Figure 7 As shown, a query UI interface and interface corresponding to OSS can be provided for users to set business data collection-related information such as business data collection type, collection start time, and collection end time, implement query settings such as query conditions, and perform related query operations.

[0099] Regarding the query of node data (i.e., business data), the user enters the business data related query conditions through the corresponding UI interface to complete the query of business data. Figure 8 As shown, the data flow link may include Figure 8 For example, the vehicle operation end, gateway, parsing service, relational database service (RDS), Flink, Hive and other links shown in the data flow link, as well as Hive engine, HiveSql and other Hive-related links. The query process of the business data corresponding to the data flow link can be, for example, inputting the corresponding vehicle identification code (Vehicle Identification Number, VIN) (for example, 123456) and the corresponding start time (for example, 20241125) and end time (for example, 20241126) on the corresponding UI interface, and the business data of the relevant nodes can be queried. For example, Figure 8 As shown in the figure, business data related to Kafka (topic1) and Kafka (topic3) are queried. The queried business data includes VIN (for example, 20241126), generation time (for example, 202411, 202412, etc.), specific data (for example, {*} in the figure), and data lineage identification information (i.e., lineage id, such as aaaaaa, bbbbbb, cccccc in the figure), etc.

[0100] Furthermore, for the business data corresponding to the Kafka data node, the corresponding business data can be queried from OSS. For the node data in the Hive table, the corresponding business data can be directly queried from the Hive table. For the business data in the Hive table, the relevant business data can be directly queried according to the conditions entered by the user.

[0101] The following is a further explanation of the process of implementing data governance functions based on metadata (that is, data lineage information) and business data.

[0102] For example, Fig. 9As shown, in the data management method provided by the present application, the data analysis engine obtains metadata from the data source corresponding to the metadata (such as Atlas) based on the aforementioned Spark / scheduled task, etc., and obtains business data from the data source corresponding to the business data (such as Kafka data node) based on Flink, etc., and then performs corresponding analysis and processing on the metadata and business data based on the data analysis engine, and then inputs them into the corresponding business model (i.e., data feature engineering / data model), and the corresponding scenario output can be obtained. Among them, the metadata can be obtained through the aforementioned Atlas, for example, and the business data can be obtained through the aforementioned Kafka data node. The business model includes, for example, a hot and cold data model, a data value model, a task analysis model, and an abnormal data model. The corresponding model processing is hot and cold data analysis and optimization processing, task analysis and task scheduling optimization processing, data archiving and delisting processing, and abnormal data alarm processing (as some examples of the second data processing), and the corresponding model scenario output is hot and cold data differentiated storage, task automatic tuning, data archiving / delisting processing, and abnormal data alarm processing.

[0103] In this way, by establishing different business scenario models and analyzing metadata (that is, data lineage information) and business data through real-time stream analysis engines such as flink / spark, the existing data flow links can be optimized and data assets can be optimized.

[0104] For example, Fig.10 As shown, the data archiving and delisting processing corresponding to the above-mentioned data value model, including invalid data / task analysis and cleaning processing, can analyze and clean up invalid data and invalid tasks in the data, tasks and other related contents involved in the business in the data flow link, thereby reducing computing costs, and can analyze and clean up or archive data nodes that have no business consumption in the data flow link, thereby reducing data storage costs.

[0105] For example, Fig.11 As shown, the cold and hot data corresponding to the above-mentioned cold and hot data models are stored and processed differently (i.e., cold and hot data analysis and optimization processing), including analyzing the cold / hot data in the data flow link and formulating different storage strategies to reduce data storage costs. For example, a setting is used to identify the access popularity of a table, determine the access popularity based on the number of accesses to the table, and automatically classify the data corresponding to the task into Hot, Warn, Cold data, or cold and hot data based on the access popularity, and formulate different data governance strategies for cold and hot data. For example, cold data is optimized and stored based on the corresponding strategy, and hot data is optimized and stored based on the corresponding strategy.

[0106] Exemplarily, the automatic task tuning (i.e., task analysis and scheduling optimization) processing corresponding to the above-mentioned task analysis model can analyze the lineage information of the library and table, evaluate the cold tasks from the dimensions of task access frequency, usage mode, timeliness level, etc., solve the problems of incorrect task dependency, lack of task dependency, unreasonable task configuration, too long time consumption, excessive resource consumption, task tilt, etc. and optimize them to reduce computing costs.

[0107] Furthermore, based on the data management method provided by the present application, data security and audit processing can also be achieved (as another example of the second data processing). Fig.12 As shown, a data flow link includes the vehicle operation end, the user mobile phone end, the business system end, and the gateway, microservices, Flink, Hive processing (Process) and other links in between, as well as the corresponding Kafka data nodes, ClickHouse, etc.

[0108] Regarding the implementation of data security and auditing, such as Fig.12 As shown, it includes security monitoring processing, security control processing and security management processing. For the data nodes corresponding to the vehicle operation end, the user mobile phone end, the gateway, microservices, Kafka, Flink, HiveProcess, and ClickHouse between the business system, corresponding analysis and processing can be performed based on the online data obtained from the gateway (for example, including the aforementioned metadata, business data, etc.), so as to perform security monitoring processing such as asset identification, sensitive data identification, and risk identification. Corresponding analysis and processing can be performed based on the online data obtained from the gateway, so as to perform security control processing such as dynamic desensitization, data encryption, and access control. Corresponding analysis and processing can be performed based on the offline storage data obtained from Hive and ClickHouse, so as to perform security management processing such as data classification and grading, and security auditing.

[0109] Furthermore, if Fig.12 As shown, sensitive data identification and processing can be implemented based on regular matching rules, sensitive character libraries, model matching, etc., data risk identification and processing can be implemented based on rule engine matching, model matching, etc., and data desensitization / encryption processing can be implemented based on data desensitization algorithms and data encryption algorithms.

[0110] Of course, the data management method provided by the implementation of the present application can also implement other data processing based on the aforementioned data lineage information, as well as data lineage information and business data, to effectively manage data assets, which can be set as needed.

[0111] To summarize, the data management method provided by the implementation of this application, based on data lineage analysis technology, can implement the data full-link governance platform in stages, effectively govern data assets, and solve current pain points. Among them, the first stage is to dynamically generate the full-link node metadata lineage relationship, and the complete link node metadata information can be viewed in a visual way. The second stage is to access the full-link business data, collect and store lineage data, and support the search and query of the full-link business data. The third stage is to define relevant data models and analyze the full-link data lineage (including metadata and business data) to improve the overall data governance capabilities, including full-link data lineage diagrams, business data anomaly analysis and alarms, data asset optimization, data security audits and other capabilities.

[0112] Furthermore, the data management method provided by the implementation of the present application is connected to the lineage analysis of the big data platform to build the lineage relationship of the metadata of the full-link nodes, and form a digital full-link corresponding to the physical link. In addition, the lineage data of each link of the data link is collected and stored, and based on the data lineage analysis engine, the details and processing logic of the data flowing through each node can be viewed. The full-link data lineage analysis topology diagram is displayed in a visual way through the data map (the data lineage diagram is generated through Atlas), including the status, path, distribution of the data in the entire link, and consumption on the business side. In addition, based on the strategies and rules of data monitoring and analysis, a data lineage analysis model (model module) is constructed, and data anomalies are analyzed in real time through Flink, and displayed visually. In this way, it is possible to better manage the data life cycle, data quality, data security, and data governance, and improve management effectiveness.

[0113] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0114] See also Fig.13 , Fig.13 FIG. 1 is a block diagram of a structure of an electronic device provided by an implementation of the present invention. In an implementation of the present invention, the electronic device may be a computer, a server, etc. Fig.13 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0115] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes part of the technical solutions of the data management method in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital data processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0116] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0117] By way of example and not limitation, the memory 123 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 123 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 123 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 123 is a non-volatile solid-state memory. In a particular embodiment, the memory 123 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0118] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0119] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0120] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the data management method in the above embodiment.

[0121] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions / programs. When the computer instructions / programs are executed on a processor of an electronic device, the processor of the electronic device executes the technical solution of the data management method of the above embodiment.

[0122] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product runs on a processor of an electronic device, the program code is used to enable the processor of the electronic device to execute the steps of the method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may execute the data management method recorded in the embodiments of the present application.

[0123] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CDROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0124] The implementation method of the present application also provides a computer program product, which includes a computer program / instructions stored in a computer-readable storage medium. At least one processor can read the computer program / instructions from the computer-readable storage medium. When at least one processor executes the computer program / instructions, the technical solution of the data management method in the above embodiment can be implemented.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0129] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application may also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0130] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0131] Although the present application has been illustrated and described by referring to certain preferred implementations of the present application, it should be understood by those skilled in the art that the above contents are further detailed descriptions of the present application in combination with specific implementations, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A data management method, characterized in that: The method comprises: Acquire metadata from a first data node in a data flow link, where the first data node includes a data warehouse data node and at least one target data node other than the data warehouse data node, and the at least one target data node includes a data node related to a microservice and / or a data node related to a data computing engine in the data flow link; Data lineage analysis is performed based on the metadata to obtain data lineage information.

2. The data management method according to claim 1, characterized in that: The metadata includes first metadata obtained from the data warehouse data node and second metadata obtained from the at least one target data node, Performing data lineage analysis and processing according to the metadata to obtain data lineage information includes: Performing data lineage analysis and processing according to the first metadata and the second metadata acquired at the first time to obtain the data lineage information including metadata and association relationships between metadata; The method further comprises: The data lineage information is updated according to the first metadata acquired at a second time after the first time to obtain updated data lineage information.

3. The data management method according to claim 1, characterized in that: The method further comprises: Display the data lineage information; and / or A first data processing is performed according to the data lineage information.

4. The data management method according to claim 3, characterized in that: The first data processing includes at least one of the following processing: Metadata query processing; Data lineage analysis and processing; Data impact analysis and processing; Data anomaly monitoring and alarm processing.

5. The data management method according to any one of claims 1 to 4, characterized in that: Performing data lineage analysis and processing according to the metadata to obtain data lineage information includes: The metadata is subjected to data lineage analysis processing based on a data lineage analysis engine to obtain the data lineage information.

6. The data management method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire service data from a second data node in the data flow link; A second data processing is performed according to the business data and the data lineage information.

7. The data management method according to claim 6, characterized in that: The second data processing includes at least one of the following processing: Data archiving and delisting; Cold and hot data analysis and optimization processing; Task analysis and task scheduling optimization processing; Abnormal data alarm processing.

8. The data management method according to claim 6, characterized in that: Implementing second data processing according to the business data and the data lineage information includes: The business data and the data lineage information are input into a data management model, and the second data processing is performed through the data management model.

9. The data management method according to claim 8, characterized in that: The data management model includes a data value model, a hot and cold data model, a task analysis model, and an abnormal data model. The business data and the data lineage information are input into the data management model, and the second data processing is performed through the data management model, including: The business data and the data lineage information are respectively input into the data value model, the hot and cold data model, the task analysis model, and the abnormal data model, which respectively perform the data archiving and delisting processing, the hot and cold data analysis and optimization processing, the task analysis and task scheduling optimization processing, and the abnormal data alarm processing.

10. The data management method according to any one of claims 1 to 9, characterized in that: The data warehouse data nodes are data nodes that implement data processing based on Hive technology, and the data nodes related to the data computing engine include data nodes that implement data processing based on Flink technology and data nodes that implement data processing based on Spark technology.

Images