Data asset management method and platform
By accessing cloud services to obtain full-chain data of data assets, full-chain monitoring and customized information push can be achieved, which solves the problem of insufficient monitoring in data asset management, improves the stability of data assets and cross-team collaboration efficiency, and reduces operating costs.
Patent Information
- Application Number
- CN202410989657.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
AI Technical Summary
The existing data asset management process cannot achieve comprehensive, granular, and personalized monitoring, resulting in high operating costs, increased risks, and a lack of customized information push capabilities, which affects the efficiency of cross-team collaboration.
By accessing cloud services, the system can acquire end-to-end data of target data assets, enabling end-to-end monitoring, tracking multi-dimensional monitoring metrics, and pushing customized information to different roles using cloud service notification events.
It enables fine-grained monitoring of data assets, reduces the occurrence of anomalies, optimizes data quality, reduces operating costs, improves cross-team collaboration efficiency, and ensures the reliability of data asset management.
Smart Images

Figure CN118972444A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data asset management method and platform. Background Art
[0002] With the rapid development of computer technology and cloud service technology, the digital economy era has arrived. Internet services are becoming increasingly rich and diverse, and the number of users and online time are continuing to grow, bringing about a large amount of data accumulation. Data is the core asset of an enterprise. It has irreplaceable value that can be sustainably mined in product function verification, driving business growth, refined operations, personalized services, etc., and the demand for data asset management continues to expand.
[0003] At present, there are various big data frameworks for data asset management, but comprehensive, fine-grained, and personalized monitoring cannot be achieved in the data asset management process, which increases operating costs and risks. Insufficient data asset monitoring leads to low reliability of data asset management. Summary of the invention
[0004] In view of this, an embodiment of this specification provides a data asset management method. One or more embodiments of this specification also relate to a data asset management platform, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, a data asset management method is provided, which is applied to a data asset management platform, wherein the data asset management platform is connected to a cloud service, and the method includes:
[0006] Using the cloud service to obtain full-link data of the target data asset, wherein the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation;
[0007] For the full-link data, tracking monitoring indicators in at least one dimension, and obtaining indicator tracking information corresponding to each monitoring indicator;
[0008] The monitoring roles corresponding to the monitoring indicators are determined, and the indicator tracking information corresponding to the monitoring indicators is pushed to the corresponding monitoring roles respectively by using the notification events of the cloud service.
[0009] According to a second aspect of an embodiment of this specification, a data asset management platform is provided, the data asset management platform is connected to a cloud service, and the platform includes:
[0010] A data acquisition module is configured to acquire full-link data of a target data asset using the cloud service, wherein the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation;
[0011] An indicator monitoring module is configured to track monitoring indicators in at least one dimension for the full-link data and obtain indicator tracking information corresponding to each monitoring indicator;
[0012] The first push module is configured to determine the monitoring role corresponding to each monitoring indicator, and use the notification event of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role.
[0013] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0014] Memory and processor;
[0015] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned data asset management method are implemented.
[0016] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data asset management method are implemented.
[0017] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned data asset management method are implemented.
[0018] An embodiment of the present specification provides a data asset management method, which is applied to a data asset management platform, wherein the data asset management platform is connected to a cloud service and uses the cloud service to obtain full-link data of a target data asset, wherein the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation; for the full-link data, monitoring indicators under at least one dimension are tracked to obtain indicator tracking information corresponding to each monitoring indicator; the monitoring role corresponding to each monitoring indicator is determined, and the indicator tracking information corresponding to each monitoring indicator is pushed to the corresponding monitoring role respectively by using the notification event of the cloud service.
[0019] In the embodiments of this specification, the data asset management platform is connected to the cloud service and deeply integrated with the cloud service. The cloud service is used to obtain the full-link data of the target data assets, and the full-link monitoring of the target data assets in each stage of construction and operation is realized. By comprehensively monitoring each link of the data assets, possible anomalies can be discovered in time, thereby greatly reducing the occurrence of abnormal situations. It also provides tracking and monitoring of monitoring indicators in at least one dimension to achieve fine-grained monitoring of the target data assets, and uses the notification events of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role, providing different roles with the ability to customize information push, realizing personalized information push, and improving the efficiency of cross-team collaboration, so that relevant roles can obtain the required monitoring information in a timely manner, which not only improves the stability of data assets, but also ensures the continuous optimization of data asset quality, provides reliable data support for decision-making, reduces operating costs and risks, and ensures the reliability of data asset management. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 is a flow chart of a data asset management method provided by an embodiment of this specification;
[0021] Figure 2a It is a schematic diagram of a metadata management process provided by an embodiment of this specification;
[0022] Figure 2b This is a data monitoring requirement diagram for setting a role provided by an embodiment of this specification;
[0023] Figure 3 It is a processing diagram of a data asset management method provided by an embodiment of this specification;
[0024] Figure 4 It is a structural diagram of a data asset management platform provided by an embodiment of this specification;
[0025] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0026] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0027] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0028] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0029] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0030] First, the terms involved in one or more embodiments of this specification are explained.
[0031] Data assets: refers to data resources that are owned or controlled by individuals or enterprises and can bring future economic benefits to the enterprise and are recorded in physical or electronic form. For example, data assets can be data tables of data storage types such as relational databases (MySQL), non-relational databases (NoSQL), data warehouses (Hive), and data lakes. Data assets can also be resources such as data text files.
[0032] Atlas: Aims to manage metadata in Hadoop projects and develop into a comprehensive data governance framework. It provides core capabilities for Hadoop clusters, including data classification, centralized policy engine, data lineage, security, and lifecycle management. Due to its horizontal scalability, strong integration capabilities, and open source characteristics, Atlas has become the first choice for many vendors, and many vendors choose to use or further develop it. Atlas' deep integration with the Hadoop ecosystem, especially with Hive, enables it to support table-level and field-level data lineage, native integration with HDP through Ranger, and support row-column-level data permission control. Easy installation and powerful metadata model allow customization and extension, and the directness of the source code makes it widely adopted and customized into commercial products.
[0033] DataHub: A metadata platform designed for modern data stacks to address metadata management challenges in diverse and complex data ecosystems. DataHub provides metadata search, data discovery, data monitoring, and governance capabilities, simplifying the complexity of data management. Running under Apache License 2, it uses a push-based architecture for metadata collection (pull-based methods are also supported) to ensure continuous capture of metadata changes. DataHub has integrated access capabilities to a wide range of data ecosystems including Kafka, Airflow, MySQL, SQL Server, Postgres, LDAP, Snowflake, Hive, and BigQuery.
[0034] Open Metadata: As an open standard, it lays the foundation for end-to-end metadata management solutions. Open Metadata provides all the necessary components for data discovery, data governance, data collaboration, data quality, and data observability, representing an improvement over the existing data governance model. Its interface design is beautiful, and its operation logic fits the usage habits of business personnel. By providing an online Demo environment, it not only facilitates user experience, but also helps promote and attract new users. Although the project itself is young, it has been able to draw on the experience of many existing data asset management projects, especially the integration of a data quality module, which demonstrates a deep understanding of data governance while also supporting open data standards.
[0035] AWS (Amazon Web Services): is a cloud platform that provides more than 200 full-featured services from data centers around the world. AWS aims to provide a flexible and secure cloud computing environment.
[0036] It should be noted that there are currently a variety of big data frameworks for data asset management. The metadata management function in these frameworks is an important component of the data management system. However, comprehensive, fine-grained, and personalized monitoring cannot be achieved in the data asset management process, which increases operating costs and risks. Insufficient data asset monitoring leads to low reliability of data asset management.
[0037] The current big data frameworks for data asset management are often used for data governance and some classification or management of data domains. They cannot provide deep integration or flexible customization with public cloud infrastructures such as AWS. In the process of data asset development and deployment, it is usually impossible to fully monitor the entire data asset development and deployment process, including code management, cloud resource allocation, and service configuration. The monitoring capabilities are limited. This lack of monitoring may cause problems to be ignored at an early stage, resulting in inefficiency and potential risk management challenges, increasing operational risks and costs. In addition, there are usually limitations in monitoring the operating status of data assets, including but not limited to real-time performance monitoring, fault warnings, and detailed tracking of data processing processes. This incomplete monitoring may lead to insufficient management of data asset operating efficiency, security, and data quality. Especially when dealing with large-scale data sets and complex data processes, existing solutions often cannot provide sufficient support and flexibility to cope with rapidly changing business needs and technical environments. In addition, fine-grained monitoring is often not available, especially in terms of tracking and managing costs throughout the data asset construction and operation process. This lack of monitoring means that enterprises may find it difficult to optimize their resource use and reduce operating costs. Cost management and optimization are particularly important when using infrastructure services. Lack of detailed insight and control over costs may lead to waste of resources and reduced economic benefits. Furthermore, the lack of the ability to provide customized information push for different roles (such as data engineers, project managers, business analysts, etc.) limits the efficiency of cross-team collaboration, making it possible for relevant participants to fail to obtain the required monitoring information and alerts in a timely manner.
[0038] Therefore, a data asset management method is provided in an embodiment of the present specification, which is applied to a data asset management platform. The data asset management platform is connected to a cloud service, deeply integrated with the cloud service, and uses the cloud service to obtain full-link data of the target data asset, thereby realizing full-link monitoring of the target data asset at various stages of construction and operation. By comprehensively monitoring each link of the data asset, possible anomalies can be discovered in a timely manner, thereby greatly reducing the occurrence of abnormal situations. It also provides tracking and monitoring of monitoring indicators in at least one dimension to realize fine-grained monitoring of the target data asset, and uses notification events of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role, providing different roles with the ability to customize information push, realizing personalized information push, and improving the efficiency of cross-team collaboration, so that relevant roles can obtain the required monitoring information in a timely manner, which not only improves the stability of data assets, but also ensures the continuous optimization of data asset quality, provides reliable data support for decision-making, reduces operating costs and risks, and ensures the reliability of data asset management.
[0039] In this specification, a data asset management method is provided. This specification also involves a data asset management platform, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.
[0040] See also Figure 1 , Figure 1 A flow chart of a data asset management method provided according to an embodiment of the present specification is shown, which is applied to a data asset management platform, and the data asset management platform is connected to a cloud service. The data asset management method specifically includes the following steps 102-106.
[0041] Step 102: Use cloud services to obtain full-link data of the target data asset, where the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation.
[0042] Among them, each stage of job operation can generate a series of indicator data. The data asset management platform can obtain the relevant indicator data generated by the target data assets at each stage of construction and job operation, thereby realizing full-link monitoring of the target data assets at each stage of construction and job operation.
[0043] Specifically, cloud service is a cloud computing service connected to the data asset platform, which can provide basic management capabilities such as data collection, governance, and monitoring, as well as information push services. The data asset management platform relies on the connected cloud service to provide data asset management related services. The cloud service includes but is not limited to a private cloud deployment service provided by a public cloud service provider or a private hybrid cloud service jointly provided by multiple public cloud service providers. That is, the data asset management platform based on the cloud service has a firewall with the external Internet network, and the services provided by the data asset management platform cannot directly access the external Internet.
[0044] It should be noted that a data asset management platform based on cloud services refers to a platform that uses cloud infrastructure and services to build storage, processing, and analysis platforms for managing data assets. This data asset management platform is not limited to a single public cloud environment, but can also cover a variety of deployment models, especially private cloud or hybrid cloud solutions customized for enterprise-level needs. Among them, private cloud deployment services can be private cloud environments designed and maintained by public cloud service providers for enterprises. Although this environment is based on the technology of cloud service providers, the physical resources, network environment, and data management are independent and isolated, ensuring a high degree of security and customization capabilities. Private clouds are usually located in the internal network of an enterprise or accessed through a dedicated connection, with clear boundaries and strictly controlled access policies from the external Internet, adding an additional layer of security. Private hybrid cloud service is a more flexible architecture that combines the security and control of private cloud with the elasticity and cost-effectiveness of public cloud. Enterprise critical data and applications can reside in private cloud, while some non-sensitive operations or tasks requiring a lot of computing resources can utilize public cloud resources. To ensure security, these environments are interconnected through secure connection technologies (such as VPN, dedicated links, etc.). At the same time, firewalls and access control policies are set to limit direct interaction with the external Internet and protect data from unauthorized access and potential threats.
[0045] For example, the cloud service can be the public cloud AWS, and the data lake and data warehouse used by the data management platform are based on AWS. Data assets are related data resources managed by the data asset management platform during the operation of the enterprise. A data asset can be regarded as a unit database, and the target data asset is any data asset managed by the data asset management platform.
[0046] The full-link data includes the relevant indicator data generated by the target data assets at various stages of construction and operation. In other words, the full-link data includes the relevant indicator data generated by the target data assets from development and deployment to operation, such as code management, cloud resource allocation, service configuration and other related data, and can also include production, logistics, order production plan, order plan, etc. It should be noted that the data of certain links may not be generated on the data management platform, but on the upstream and downstream platforms. By using the cloud services connected to the data management platform, the data generated by the target data assets from development and deployment to operation can be obtained from the upstream and downstream platforms.
[0047] In actual implementation, the full-link data includes but is not limited to the metadata generated by the target data asset from development, deployment to job operation. Metadata is the data used to describe the target data asset, such as table name, table description, table creation statement, table field, field type and field description, source system, source system library table, asset job task and scheduling information, etc.
[0048] In addition, the data asset management platform can provide customized basic data collection functions based on the connected cloud services, collect metadata generated by the target data assets from development, deployment to operation in real time, and realize metadata monitoring and management. That is, the data collection and management in the data asset management platform are based on the self-developed metadata management framework, and do not adopt the existing open source metadata management framework. A set of exclusive metadata management frameworks are designed according to the company's unique business needs and data architecture, which can accurately match the company's business logic, data structure and job operation process, and realize highly customized data collection, management and analysis capabilities. It can more effectively support specific data processing rules and data quality checks, allowing enterprises to It allows enterprises to quickly adjust and expand functional modules when needs change, without being restricted by the update cycle and community support of open source software, ensuring the long-term sustainable development and competitiveness of the system. In addition, since the metadata management framework is independently developed and maintained, it can implement stricter security policies and access control mechanisms to ensure that sensitive information is not leaked and meet the security and compliance requirements of the enterprise. The independently developed metadata management framework facilitates the integration of more sophisticated data governance strategies, such as more complex metadata quality assessment models, data lineage tracking capabilities, and cross-system metadata consistency maintenance, thereby improving data accuracy and availability.
[0049] Furthermore, the data asset management platform can also be configured with a customized information collection module plug-in, which can be pre-tracked during the task operation of the target data asset. When the task runs to the tracking location, the corresponding data can be automatically generated for storage, that is, at each stage of the target data asset operation, the corresponding indicator data can be automatically generated in the background without perception, that is, the user will not be aware that the metadata supervision and collection tracking operation is running, so as to facilitate subsequent tracking, analysis and monitoring. For example, the start time of the task operation, the output assets and input assets of different data processing layers of the data warehouse or data lake can be automatically recorded, and the obtained library tables are "input asset library tables and output asset library tables".
[0050] In the embodiments of this specification, the data asset management platform can provide data injection and data consumption capabilities. The data asset management platform can use cloud services to obtain full-link data of target data assets, thereby achieving comprehensive monitoring of all aspects of the target data assets from development, deployment to operation.
[0051] In an optional implementation of this embodiment, after obtaining the full-link data of the target data asset by using the cloud service, the following further includes:
[0052] Generate and display a full-link monitoring data table of the target data asset based on the full-link data and set data analysis rules;
[0053] Push the full-link monitoring data table of the target data asset to the configured operation and maintenance role.
[0054] Specifically, the data analysis rules can be pre-configured rules on the data management platform, which are used to count and analyze the obtained full-link data, and generate a full-link monitoring data table for the target data asset according to the rules. The full-link monitoring data table monitors the metadata of the target data asset, not the target data asset itself, so that the full-link monitoring data table can be pushed to the corresponding operation and maintenance role to analyze and make decisions on the metadata of the target data asset. Among them, the operation and maintenance role is a pre-configured role that needs to obtain the full-link monitoring data table of the target data asset for data analysis, such as a project manager, business analyst, etc.
[0055] In one possible implementation, multiple roles and users included in various roles can be pre-configured in the data asset management platform. When pushing the full-link monitoring data table of the target data asset to the operation and maintenance role, the user corresponding to the operation and maintenance role can be obtained. The user can be one or more. The cloud service is used to trigger a notification event to push the full-link monitoring data table to the user corresponding to the operation and maintenance role.
[0056] It should be noted that the full-link data can be the metadata of the data generated by the target data asset from development, deployment to operation, and the metadata is written into the database, which can be RDS (Relational Database Service). In addition, by customizing the data analysis rules and combining the database tables on the cloud service, the metadata analysis function is provided, and the full-link monitoring data table of the target data asset generated by the analysis can also be written into the database.
[0057] In this embodiment, data collection and management in the data asset management platform are implemented based on a self-developed metadata management framework. When developing the metadata management framework independently, it can be combined with open source metadata management architectures such as Atlas, DataHub, and Open Metadata to realize customized metadata collection, analysis, and management capabilities of the data asset management platform.
[0058] In actual implementation, AWS GLUE can be customized and packaged so that the data asset management platform can automatically capture and record the target data asset's tables, tasks, triggers and other metadata, and based on this, these metadata can be effectively connected to avoid the formation of data islands. Of course, in actual implementation, it is not limited to the basic services provided by the AWS public cloud, and other cloud services can also be used, such as Azure function, which is a server computing service provided by the Azure cloud platform to help developers quickly build scalable applications and services without worrying about the underlying server management. In addition, through deep integration of DevOps tool chains, such as Bitbucket or GitHub for code management, the data asset management platform can automatically capture and analyze the key metadata of the target data assets during the construction and deployment process, so that the team can more effectively manage the life cycle of the target data assets, ensure data quality and consistency, and improve the efficiency of development and deployment.
[0059] For example, Figure 2a is a schematic diagram of a metadata management process provided by an embodiment of this specification, such as Figure 2a As shown in the figure, the data asset management platform uses the metadata center of the cloud service to obtain the metadata of the target data asset. Based on the obtained metadata, the data asset management platform generates the corresponding full-link monitoring data table, such as the job operation information table, input data set / data / data table, output data set / data / data table, metadata connection information table, trigger information table, etc., and writes the generated full-link monitoring data table into the RDS database. Figure 2aAs shown, the data asset management platform can generate information tables such as job operation information tables, input data sets / data / data tables, output data sets / data / data tables, metadata connection information tables, trigger information tables, etc., and write them into the database to achieve the connectivity of various types of data such as job operation information tables, input data sets / data / data tables, output data sets / data / data tables, metadata connection information tables, trigger information tables, etc., connect various metadata, and break down information barriers.
[0060] In the embodiments of this specification, the monitoring and governance of metadata in the DevOps environment are supported, including version control, deployment history and change tracking of data assets, etc., connecting multiple metadata, breaking down information barriers, and managing target data assets more effectively.
[0061] Step 104: For the full-link data, track the monitoring indicators in at least one dimension and obtain indicator tracking information corresponding to each monitoring indicator.
[0062] Specifically, at least one monitoring indicator under at least one dimension is a fine-grained monitoring indicator pre-configured in the data asset management platform, which is used to indicate which data to monitor and how to monitor it. For example, under the cost dimension, the monitoring indicators can be resource usage, storage cost, operation consumption, etc.; under the data governance dimension, the monitoring indicators can be data changes, data generation, data volume, computational complexity, etc.; under the data operation and maintenance dimension, the monitoring indicators can be data dependencies, upstream and downstream monitoring, etc.
[0063] It should be noted that for full-link data, it is possible to track monitoring indicators in at least one dimension, obtain indicator tracking information corresponding to each monitoring indicator, and achieve fine-grained monitoring of target data assets.
[0064] In actual implementation, at least one dimension of monitoring indicators and the tracking rules of each monitoring indicator can be pre-configured in the data asset management platform. Specifically, based on the basic services provided by AWS, you can develop personalized tool modules, customize personalized configuration libraries, run configured rules, and track and monitor data.
[0065] In an optional implementation of this embodiment, the dimension is a cost dimension, and the monitoring indicator is a used resource; and for the full-link data, tracking the monitoring indicator under at least one dimension and obtaining the indicator tracking information corresponding to each monitoring indicator include:
[0066] Track the usage resources of each task in the full-link data to obtain the resource usage of each task under the target data asset;
[0067] According to the resource usage of each task under the target data asset, a resource usage list of the target data asset is generated, and the resource usage list is used as the indicator tracking information of the resource usage.
[0068] In actual implementation, the resources used by each task in the full-link data can be tracked, the resource usage of each task under the target data asset can be obtained, and a resource usage list of the target data asset can be generated. The resource usage list can indicate the resource usage of each task, so as to conduct cost analysis and monitor the resources used by each task of the target data asset. Fine-grained monitoring under the cost dimension is provided to track and manage the cost of the entire data asset during the construction and operation process, optimize its resource usage, reduce operating costs, and avoid resource waste and reduced economic benefits.
[0069] In the embodiments of this specification, the data asset management platform constructs a fine-grained cost monitoring by capturing and analyzing various indicators related to the cost dimension, which can not only include the cost of cloud resource usage, but also cover the costs related to data processing and storage, and can timely optimize resource allocation and usage, thereby maximizing cost-effectiveness.
[0070] In an optional implementation of this embodiment, after tracking the monitoring indicators in at least one dimension for the full-link data and obtaining the indicator tracking information corresponding to each monitoring indicator, the method further includes:
[0071] Determine whether there is an abnormal indicator according to the indicator tracking information corresponding to each monitoring indicator;
[0072] If there is an abnormal indicator, the notification event of the cloud service is used to push an indicator abnormality message to the management role corresponding to the abnormal indicator.
[0073] It should be noted that after tracking the monitoring indicators under at least one dimension and obtaining the indicator tracking information corresponding to each monitoring indicator, an abnormal alarm can be issued. Specifically, it is possible to determine whether there are abnormal indicators based on the indicator tracking information corresponding to each monitoring indicator according to the set monitoring rules. If so, the notification event of the cloud service is used to push the indicator abnormality message to the management role corresponding to the abnormal indicator. The abnormal message is used to alert the abnormal indicator. Among them, the management role is a role that manages and monitors abnormal indicators. Different abnormal indicators can correspond to different management roles, or different abnormal indicators can be pushed to the same management role, and then the management role notifies the person in charge of the project to handle it.
[0074] For example, on a certain day, a task under the target data asset runs for too long and is not within an optimized range, or a task has a small amount of data but takes a particularly long time to run, and the calculations involved are not too complicated, then the running indicator is determined to be abnormal; or, for the code integrity indicator, you can also track and determine whether the upstream and downstream codes are complete. If not, it is determined that the integrity indicator is abnormal; or, for various indicators under the data governance dimension, assuming that some data is not updated, some assets are not generated, and data governance for a certain field does not meet the requirements of the data steward or the upstream and downstream data, then the data change indicator is determined to be abnormal.
[0075] In the embodiments of this specification, a fine-grained monitoring and early warning mechanism is designed, which can perform real-time analysis and provide timely early warnings based on various monitoring indicators in multiple dimensions such as development specifications and cost indicators, so that enterprises can quickly respond to abnormal indicators and take corresponding measures.
[0076] In addition, if a problem occurs on the upstream platform of the data asset management platform, the corresponding abnormal indicators can be detected, which can automatically trigger events and notify the upstream platform to achieve end-to-end linkage notification.
[0077] Step 106: Determine the monitoring role corresponding to each monitoring indicator, and use the notification event of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role.
[0078] It should be noted that the data asset management platform can utilize services provided by the public cloud AWS, such as AWS Event Bridge (event bridge), AWS Lambda (computing service) and Amazon SNS (message push), to create a tightly integrated data asset monitoring environment, allowing real-time tracking of data pipelines, cloud resource utilization and application performance directly within the AWS ecosystem.
[0079] In specific implementation, when it is necessary to notify the monitoring role corresponding to each monitoring indicator, AWS Event Bridge can be used to automatically detect and notify operation events, so as to push information to the corresponding monitoring role through notification events.
[0080] Of course, in actual implementation, in addition to the services provided by the public cloud AWS mentioned above, other cloud services can also be used to implement message push, such as Azure cloud services or scenarios involving hybrid cloud services to cope with complex and changing business needs. Optionally, you can use the "Azure Eventhub" cloud service. Azure Event Hubs is a highly scalable data ingestion service that is suitable for processing large-scale data streams. "Ingestion" usually refers to the process of "data ingestion" or "data acquisition", which involves collecting, receiving and importing data assets from various source systems into a centralized system, database or data processing platform for further processing, analysis or storage.
[0081] In an optional implementation of this embodiment, determining the monitoring role corresponding to each monitoring indicator includes:
[0082] Identify data monitoring requirements for at least one set of roles;
[0083] According to the data monitoring requirements and the monitoring content of each monitoring indicator, the monitoring role corresponding to each monitoring indicator is determined.
[0084] In specific implementation, the data asset management platform can be pre-configured with at least one type of set roles and the data monitoring needs of each type of set roles, that is, which type of role needs to obtain which type of data. The data monitoring needs of different roles may or may not overlap, that is, one monitoring indicator can correspond to this monitoring role, and one monitoring role can correspond to at least one monitoring indicator.
[0085] In addition, the user lists corresponding to different roles can be configured or input in advance. When a push is formed, the user list corresponding to the corresponding role can be called to push information to each user in the user list, thereby realizing information push within a specific group of people or organization.
[0086] In actual implementation, the monitoring roles corresponding to each monitoring indicator can be determined according to the data monitoring requirements and the monitoring content of each monitoring indicator. Specifically, for the first monitoring indicator, it is determined that data monitoring needs to include the roles of the monitoring content of the first monitoring indicator, and the determined roles are used as the monitoring roles corresponding to the first monitoring indicator. Any monitoring indicator can be used as the first monitoring indicator to determine the corresponding monitoring roles. For example, the monitoring content of the resource usage indicator is the usage resources and usage time of each task. The data monitoring of data engineers needs to include the usage resources of data assets, so the monitoring role corresponding to the resource usage indicator is determined to be data engineer.
[0087] For example, Figure 2bThis is a schematic diagram of data monitoring requirements for setting roles provided by an embodiment of this specification, such as Figure 2b As shown in the figure, the data monitoring requirements of upstream operation and maintenance personnel are whether the metadata is abnormal; the data monitoring requirements of data engineers are the use, optimization and reuse of data assets; the data monitoring requirements of data stewards are the asset table readiness time and asset additional metadata; the data monitoring requirements of downstream business users are data quality and operation errors; the data monitoring requirements of second-line operation and maintenance personnel are asset operation configuration information and operation status information. Figure 2b The relevant stakeholders shown share or push corresponding information to maximize the asset benefits of asset governance.
[0088] In the embodiments of this specification, the data asset management platform provides a push notification service, which can send customized alerts and information to different roles from employees to business analysts, thereby realizing personalized information push.
[0089] In an optional implementation of this embodiment, the data asset management method further includes:
[0090] Determine attribute information of each data asset in the data asset management platform, wherein the attribute information includes but is not limited to dependency, update frequency and / or quota information of a resource pool to be deployed;
[0091] In case of receiving a development instruction of a new data asset, determining the location of the new data asset to be deployed according to the attribute information;
[0092] The new data asset is deployed based on the to-be-deployed location.
[0093] It should be noted that a cloud room may produce hundreds of data assets, and each data asset may involve three or four hundred work tasks. These data assets involve some libraries and tables at various levels in the metadata system. They are related to each other, and there is a dependency relationship between them when configuring scheduling. In addition, different data assets have different update frequencies, and the quota information of the resource pool to be deployed is also different. Therefore, the dependency relationship, update frequency and / or quota information of the resource pool to be deployed of each data asset can be obtained to determine the location to be deployed of the new data asset, so as to deploy the new data asset based on the dependency relationship, update frequency and / or quota information of the resource pool to be deployed.
[0094] In actual implementation, there are multiple data assets managed in the data asset management platform, more than 90% of which are dependent on each other, forming a complex network structure. When receiving development instructions for new data assets, if they are created separately based on the current list of new data assets, it may cause a large waste of resources. The continuous creation of new data assets will cause a sharp increase in the cost of cloud services, and may conflict with the dependencies of some existing data assets.
[0095] Therefore, in the specific implementation, the location to be deployed of the new data asset can be determined based on the dependency relationship, update frequency and / or quota information of the resource pool to be deployed of each data asset, and then the new data asset can be deployed directly based on the location to be deployed. This is equivalent to directly splicing the upstream and downstream dependencies of the new data asset onto the dependencies of the existing data assets based on the update frequency when a new data asset is developed, which is equivalent to making a graft or adding a small branch. The deployed new data asset complies with the quota information of the resource pool to be deployed, thereby improving the deployment efficiency.
[0096] Specifically, the metadata of the target data asset can indicate the blood relationship between data assets, and thus the dependency relationship of each data asset can be determined based on metadata analysis, such as the scheduling of the previous layer and the scheduling of the next layer, the data of the previous layer and the data of the next layer, and a table in the data of the previous layer and other tables in the data of the next layer may have a data blood relationship between the upper and lower levels.
[0097] In the embodiments of this specification, after accessing the cloud service, the full-link data of the target data asset can be monitored, and the data that can be monitored and analyzed is more and more complete. For newly developed data assets or maintenance of old data assets, its upstream and downstream dependencies can be clearly known, including upstream scheduling, assets, tasks, execution nodes, etc. The data query speed is faster, which significantly improves the development, deployment and daily operation and maintenance efficiency of data assets, reduces the need for manual operations, ensures faster and more consistent asset updates and deployments, and accelerates the delivery cycle.
[0098] In an optional implementation of this embodiment, after determining the attribute information of each data asset in the data asset management platform, the method further includes:
[0099] In the case where the attribute information includes dependency relationships, determining whether there is a contradiction in the dependency relationships of the data assets;
[0100] If there is a contradiction, the abnormal data asset with the contradictory dependency is determined, the monitoring personnel of the abnormal data asset is determined, and the dependency alarm information is pushed to the monitoring personnel by using the notification event of the cloud service.
[0101] It should be noted that the dependency relationship between the data source table and the job scheduling in which different data assets are located may conflict. Therefore, after determining the dependency relationship between the source tables of each data asset in the data asset management platform, it is also possible to determine whether there is a contradiction in the dependency relationship of the job scheduling of each data asset. If there is a contradiction, the dependency alarm information can be pushed to the monitoring personnel.
[0102] In the embodiments of this specification, the data asset management platform can also proactively analyze whether there are any conflicts in the dependencies between existing data assets, and push them to corresponding monitoring personnel, such as second-line or third-line operation and maintenance personnel, to more comprehensively monitor the dependencies of data assets, thereby ensuring the efficiency of data asset development, deployment, and operation and maintenance.
[0103] In an optional implementation of this embodiment, the data asset management method further includes:
[0104] Identify downstream users or operators of the target data assets;
[0105] When the target data asset is updated, the update status of the target data asset is pushed to the downstream users or operators of the target data asset.
[0106] It should be noted that there may be data updates during the operation of the target data assets. The downstream users or operators of the target data assets can be determined. When the target data assets are updated, the update status of the target data assets can be pushed to the downstream users or operators of the target data assets to ensure that the downstream users or operators can timely know the update status of the target data assets, provide timely data insights, and ensure that the downstream users or operators can timely know the current update status of the target data assets.
[0107] In an optional implementation of this embodiment, the data asset management method further includes:
[0108] Receiving a query instruction initiated through a query interface, wherein the query interface is an interface corresponding to a query service provided by the cloud service, or the query interface is an application program interface for connecting to upstream and downstream platforms, and the query instruction carries a parameter to be queried;
[0109] Access the data lake corresponding to the cloud service according to the query parameters to obtain query results corresponding to the query parameters.
[0110] Specifically, the parameter to be queried is used to indicate the content that needs to be queried, such as the parameter to be queried may be a scheduling relationship, and the scheduling relationship of the target data asset is queried based on the query instruction; or, the parameter to be queried may be an asset status, and the asset status of the target data asset is queried based on the query instruction; or, the parameter to be queried may be a full-link monitoring data table, and the full-link monitoring data table of the target data asset is queried based on the query instruction.
[0111] In actual implementation, the query can be implemented based on the automatic query service of the cloud service, or the data asset management platform can also provide an external application programming interface (API interface) that can directly connect to the upstream and downstream platforms.
[0112] It should be noted that based on the data asset management platform, business users can efficiently and conveniently query information such as the status and scheduling relationships of data assets without having to deeply understand the underlying technical details. The user-friendly query service greatly improves the efficiency of cross-departmental collaboration, enabling business users to obtain the required information more quickly, thereby accelerating the decision-making process.
[0113] In the embodiments of this specification, by comprehensively monitoring every link of data asset construction, from code submission to asset deployment, it is possible to timely discover and warn of possible anomalies, thereby greatly reducing the occurrence of abnormal situations, which not only improves the stability of assets, but also ensures the continuous optimization of data quality, and provides reliable data support for decision-making. In addition, fine-grained cost monitoring and intelligent analysis can help to troubleshoot and optimize the configuration of data processing tasks, effectively reduce the waste of cloud computing resources, thereby achieving a dual reduction in cloud costs and operation and maintenance costs, and improving overall cost efficiency.
[0114] The following combination Figure 3 , taking the application of the data asset management method provided in this specification in the automobile manufacturing scenario as an example, the data asset management method is further explained. Figure 3 A schematic diagram of a data asset management method provided by an embodiment of this specification is shown, where a data asset management platform is connected to a cloud service. Figure 3 As shown, the data asset management platform uses cloud services to obtain the full-link data of the target data assets in the automobile manufacturing scenario. The full-link data is metadata, which may include code configuration information, work metadata, cloud resource subnet information, deployment information, three-line operation and maintenance information, etc. The data asset management platform provides report generation services, data query services, information push services, etc. based on the acquired full-link data.
[0115] In the embodiments of this specification, the data asset monitoring platform can be connected to the cloud service and use some basic services of the cloud service for personalized configuration to realize data monitoring services for data assets from development, deployment to operation and maintenance in the automobile manufacturing scenario, as well as message push services and data query services. For the automobile manufacturing industry that relies on complex data processing and analysis, it can improve operational efficiency and data quality, reduce costs, and provide a better user experience to meet the needs of different users.
[0116] Corresponding to the above method embodiment, this specification also provides a data asset management platform embodiment, Figure 4 A schematic diagram of the structure of a data asset management platform provided by an embodiment of the present specification is shown, and the data asset management platform is connected to a cloud service, such as Figure 4 As shown, the data asset management platform includes:
[0117] The data acquisition module 402 is configured to acquire the full-link data of the target data asset by using the cloud service, wherein the full-link data includes the relevant indicator data generated by the target data asset at each stage of construction and operation operation;
[0118] The indicator monitoring module 404 is configured to track monitoring indicators in at least one dimension for the full-link data and obtain indicator tracking information corresponding to each monitoring indicator;
[0119] The first push module 406 is configured to determine the monitoring role corresponding to each monitoring indicator, and use the notification event of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role.
[0120] Optionally, the first push module 406 is further configured to:
[0121] Identify data monitoring requirements for at least one set of roles;
[0122] According to the data monitoring requirements and the monitoring content of each monitoring indicator, the monitoring role corresponding to each monitoring indicator is determined.
[0123] Optionally, the data asset management platform further includes a second push module configured to:
[0124] Determine whether there is an abnormal indicator according to the indicator tracking information corresponding to each monitoring indicator;
[0125] If there is an abnormal indicator, the notification event of the cloud service is used to push an indicator abnormality message to the management role corresponding to the abnormal indicator.
[0126] Optionally, the data asset management platform further includes a data asset update module configured to:
[0127] Determine attribute information of each data asset in the data asset management platform, wherein the attribute information includes but is not limited to dependency, update frequency and / or quota information of a resource pool to be deployed;
[0128] Upon receiving a development instruction for a new data asset, determining a location to be deployed of the new data asset according to the attribute information;
[0129] The new data asset is deployed based on the to-be-deployed location.
[0130] Optionally, the data asset management platform further includes a third push module configured to:
[0131] In the case where the attribute information includes dependency relationships, determining whether there is a contradiction in the dependency relationships of the data assets;
[0132] If there is a contradiction, the abnormal data asset with the contradictory dependency is determined, the monitoring personnel of the abnormal data asset is determined, and the dependency alarm information is pushed to the monitoring personnel by using the notification event of the cloud service.
[0133] Optionally, the data asset management platform further includes a generation module configured to:
[0134] Generate and display a full-link monitoring data table of the target data asset based on the full-link data and set data analysis rules;
[0135] Push the full-link monitoring data table of the target data asset to the configured operation and maintenance role.
[0136] Optionally, the data asset management platform further includes a fourth push module configured to:
[0137] Identify downstream users or operators of the target data assets;
[0138] When the target data asset is updated, the update status of the target data asset is pushed to the downstream users or operators of the target data asset.
[0139] Optionally, the dimension is a cost dimension, and the monitoring indicator is a used resource; the indicator monitoring module 404 is further configured to:
[0140] Track the usage resources of each task in the full-link data to obtain the resource usage of each task under the target data asset;
[0141] According to the resource usage of each task under the target data asset, a resource usage list of the target data asset is generated, and the resource usage list is used as the indicator tracking information of the resource usage.
[0142] Optionally, the data asset management platform further includes a query module configured to:
[0143] Receiving a query instruction initiated through a query interface, wherein the query interface is an interface corresponding to a query service provided by the cloud service, or the query interface is an application program interface for connecting to upstream and downstream platforms, and the query instruction carries a parameter to be queried;
[0144] Access the data lake corresponding to the cloud service according to the query parameters to obtain query results corresponding to the query parameters.
[0145] An embodiment of the present specification provides a data asset management platform, including a data acquisition module, an indicator monitoring module and a first push module, which are deeply integrated with cloud services, and use cloud services to obtain full-link data of target data assets, so as to realize full-link monitoring of target data assets in various stages of construction and operation. By comprehensively monitoring each link of data assets, possible anomalies can be discovered in time, thereby greatly reducing the occurrence of abnormal situations, and provides tracking and monitoring of monitoring indicators in at least one dimension, so as to realize fine-grained monitoring of target data assets, and use notification events of cloud services to push indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role, respectively, provide customized information push capabilities for different roles, realize personalized information push, improve the efficiency of cross-team collaboration, and enable relevant roles to obtain the required monitoring information in time, which not only improves the stability of data assets, but also ensures the continuous optimization of data asset quality, provides reliable data support for decision-making, reduces operating costs and risks, and ensures the reliability of data asset management.
[0146] The above is a schematic scheme of a data asset management platform of this embodiment. It should be noted that the technical scheme of the data asset management platform and the technical scheme of the above data asset management method belong to the same concept, and the details of the technical scheme of the data asset management platform that are not described in detail can be found in the description of the technical scheme of the above data asset management method.
[0147] Figure 5 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0148] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0149] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0150] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.
[0151] The processor 520 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned data asset management method.
[0152] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data asset management method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned data asset management method.
[0153] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned data asset management method.
[0154] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data asset management method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned data asset management method.
[0155] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data asset management method when executed by a processor.
[0156] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned data asset management method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned data asset management method.
[0157] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0158] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0159] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0160] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0161] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A data asset management method, characterized in that: Applied to a data asset management platform, the data asset management platform is connected to a cloud service, and the method includes: Using the cloud service to obtain full-link data of the target data asset, wherein the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation; For the full-link data, tracking monitoring indicators in at least one dimension, and obtaining indicator tracking information corresponding to each monitoring indicator; The monitoring roles corresponding to the monitoring indicators are determined, and the indicator tracking information corresponding to the monitoring indicators is pushed to the corresponding monitoring roles respectively by using the notification events of the cloud service.
2. The data asset management method according to claim 1, characterized in that: The determining of the monitoring role corresponding to each monitoring indicator includes: Identify data monitoring requirements for at least one set of roles; According to the data monitoring requirements and the monitoring content of each monitoring indicator, the monitoring role corresponding to each monitoring indicator is determined.
3. The data asset management method according to claim 1, characterized in that: After tracking the monitoring indicators in at least one dimension for the full-link data and obtaining the indicator tracking information corresponding to each monitoring indicator, the method further includes: Determine whether there is an abnormal indicator according to the indicator tracking information corresponding to each monitoring indicator; If there is an abnormal indicator, the notification event of the cloud service is used to push an indicator abnormality message to the management role corresponding to the abnormal indicator.
4. The data asset management method according to claim 1, characterized in that: The method further comprises: Determine attribute information of each data asset in the data asset management platform, wherein the attribute information includes but is not limited to dependency, update frequency and / or quota information of a resource pool to be deployed; Upon receiving a development instruction for a new data asset, determining a location to be deployed of the new data asset according to the attribute information; The new data asset is deployed based on the to-be-deployed location.
5. The data asset management method according to claim 4, characterized in that: After determining the attribute information of each data asset in the data asset management platform, the method further includes: In the case where the attribute information includes dependency relationships, determining whether there is a contradiction in the dependency relationships of the data assets; If there is a contradiction, the abnormal data asset with the contradictory dependency is determined, the monitoring personnel of the abnormal data asset is determined, and the dependency alarm information is pushed to the monitoring personnel by using the notification event of the cloud service.
6. The data asset management method according to any one of claims 1 to 5, characterized in that: After the full-link data of the target data asset is obtained by using the cloud service, the method further includes: Generate and display a full-link monitoring data table of the target data asset based on the full-link data and set data analysis rules; Push the full-link monitoring data table of the target data asset to the configured operation and maintenance role.
7. The data asset management method according to any one of claims 1 to 5, characterized in that: The method further comprises: Identify downstream users or operators of the target data assets; When the target data asset is updated, the update status of the target data asset is pushed to the downstream users or operators of the target data asset.
8. The data asset management method according to any one of claims 1 to 5, characterized in that: The dimension is a cost dimension, and the monitoring indicator is a used resource; and for the full-link data, tracking the monitoring indicator under at least one dimension, and obtaining indicator tracking information corresponding to each monitoring indicator, includes: Track the usage resources of each task in the full-link data to obtain the resource usage of each task under the target data asset; According to the resource usage of each task under the target data asset, a resource usage list of the target data asset is generated, and the resource usage list is used as the indicator tracking information of the resource usage.
9. The data asset management method according to any one of claims 1 to 5, characterized in that: The method further comprises: Receiving a query instruction initiated through a query interface, wherein the query interface is an interface corresponding to a query service provided by the cloud service, or the query interface is an application program interface for connecting to upstream and downstream platforms, and the query instruction carries a parameter to be queried; Access the data lake corresponding to the cloud service according to the query parameters to obtain query results corresponding to the query parameters.
10. A data asset management platform, characterized in that: The data asset management platform is connected to the cloud service, and the platform includes: A data acquisition module is configured to use the cloud service to acquire full-link data of the target data asset, wherein the full-link data includes relevant indicator data generated by the target data asset at various stages of construction and operation; An indicator monitoring module is configured to track monitoring indicators in at least one dimension for the full-link data and obtain indicator tracking information corresponding to each monitoring indicator; The first push module is configured to determine the monitoring role corresponding to each monitoring indicator, and use the notification event of the cloud service to push the indicator tracking information corresponding to each monitoring indicator to the corresponding monitoring role.
11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data asset management method according to any one of claims 1 to 9 are implemented.
12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the data asset management method described in any one of claims 1 to 9.
13. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the steps of the data asset management method described in any one of claims 1 to 9.
Citation Information
Cited By
Databridge with dataflow studio
US20250209401A1
