Data asset management system and method

Abstract

The invention discloses a data asset management system and method, and relates to the field of data management, and the system comprises a data collection layer which is used for collecting multi-source heterogeneous data; the data integration layer is used for integrating the multi-source heterogeneous data and constructing a data warehouse; the data storage layer is used for carrying out structured storage on the data in the data warehouse; the data governance layer is used for performing standardization processing and quality management on the data stored in the data storage layer to obtain target data; and the data application layer is used for performing dynamic tracking inventory and classification level-to-level management on the data assets based on the target data. Through cooperative work of five layers of architectures, full-process closed-loop management of the multi-source heterogeneous data from collection to application is realized, and the problems of difficulty in integration of the multi-source data, low management efficiency, uncontrollable quality and the like are effectively solved by the overall scheme.

Description

Technical Field

[0001] This application relates to the field of data management, and in particular to a data asset management system and method. Background Technology

[0002] In the process of industrial digital transformation, data assets have become a key component of an organization's core competitiveness. Especially in the field of aero-engines, high-altitude simulation tests involve multiple types of sensors, control systems, and operational systems, and the resulting test data is characterized by multi-source heterogeneity, high real-time requirements, and large scale.

[0003] In existing technologies, data acquisition often relies on a single interface or fixed protocol, making it difficult to be compatible with data sources from different devices and systems, resulting in prominent data silos. Data integration lacks a unified framework, and format conversion and cleaning are inefficient, making it difficult to build standardized data warehouses. Storage architectures mostly use traditional relational databases, which cannot meet the needs of efficient reading, writing, and expansion of massive amounts of data. Data governance suffers from a lack of standards and imperfect quality verification mechanisms, leading to poor data consistency and low reliability. At the application level, there is a lack of dynamic tracking and hierarchical management capabilities, making it difficult to achieve full lifecycle management of data assets. These problems severely restrict the value mining of experimental data and affect product development efficiency and decision-making accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a data asset management system and method that can solve problems such as difficulty in integrating multi-source data, low management efficiency, and uncontrollable quality.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a data asset management system, comprising: The data acquisition layer is used to collect heterogeneous data from multiple sources. The data integration layer is used to integrate the multi-source heterogeneous data and build a data warehouse; A data storage layer is used for structured storage of data in the data warehouse; The data governance layer is used to standardize and manage the quality of the data stored in the data storage layer to obtain the target data. The data application layer is used for dynamic tracking, inventory, classification, and hierarchical management of data assets based on the target data.

[0006] Secondly, this application provides a data asset management method, including: Collect multi-source heterogeneous data through the data acquisition layer; A data warehouse is constructed by integrating the multi-source heterogeneous data through a data integration layer; The data in the data warehouse is stored in a structured manner through a data storage layer; The data governance layer performs standardization and quality management on the data stored in the data storage layer to obtain the target data. The data application layer performs dynamic tracking, inventory, classification, and hierarchical management of data assets based on the target data.

[0007] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a data asset management system and method. The data acquisition layer supports multi-source data access, breaking down data silos. The data integration layer can build a standardized data warehouse, improving data integration efficiency. The data storage layer adopts a structured storage scheme, ensuring efficient access to massive amounts of data. The data governance layer improves data credibility and security through standardization and quality management. The data application layer realizes dynamic tracking and hierarchical management, fully releasing the value of data assets. Through the collaborative work of these five layers, a closed-loop management system for multi-source heterogeneous data from acquisition to application is achieved. The overall solution effectively solves problems such as difficulty in integrating multi-source data, low management efficiency, and uncontrollable quality. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A schematic diagram of the overall architecture of a data asset management system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the data warehouse processing procedure in one embodiment of this application; Figure 3 This is a schematic diagram of the data governance layer architecture in one embodiment of this application; Figure 4 A schematic diagram of the technical architecture of a data asset management system provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a data asset management method provided in one embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] In one exemplary embodiment, such as Figures 1 to 4 As shown, a data asset management system is provided. This system is deployed in a computer device and includes a data acquisition layer (basic support), a data integration layer, a data storage layer, a data governance layer, and a data application layer. The functions and implementation principles of each layer are described in detail below.

[0013] (a) Data acquisition layer.

[0014] The data acquisition layer is used to collect multi-source heterogeneous data. In a specific application example, the multi-source heterogeneous data is high-altitude simulation test data of an aero-engine.

[0015] The data acquisition layer forms the foundational support, encompassing the network support environment, server storage devices, operating system platform, database management system, and various middleware. Employing a distributed architecture, the data acquisition layer incorporates multiple data connectors to support access and interaction with various data sources.

[0016] In this application, the data acquisition layer employs various data acquisition tools to achieve efficient acquisition of data from different sources and formats, supporting both batch and real-time acquisition to ensure data integrity and consistency. Specifically, the data acquisition layer acquires data from industrial interconnected devices in real time via the OPCUA protocol and performs batch acquisition of data from the business database to obtain multi-source heterogeneous data.

[0017] For batch data collection: For batch data collection of structured database tables stored in the TDM (Test Data Management) system, ETL (Extract, Transform, Load) jobs are configured and started on a schedule to collect data all at once. Incremental and full data collection functions are provided. Different strategies are adopted for different data sources according to specific business needs.

[0018] For real-time acquisition: Real-time acquisition, also known as streaming acquisition or stream processing, is implemented using Flink-CDC technology. Flink-CDC is used to capture database changes, monitor and stream data changes in real time. This application utilizes this technology to monitor database tables in real time, capture data changes such as insertions, updates, or deletions, and store these streaming data in a real-time analytics database such as Apache Doris.

[0019] In an exemplary embodiment, connection information for various data sources, including databases, file systems, and message queues, is first defined and maintained. Efficient access and management of different data sources are achieved through a unified interface and configuration management. Dynamic addition and deletion of data sources, as well as monitoring and maintenance of data source connections, are supported to ensure the stability and reliability of data acquisition. Then, data extraction, transformation, and loading operations are performed on the data sources. Data acquisition tasks, through scheduled tasks or real-time triggering mechanisms, ensure timely and accurate data collection, supporting multiple data formats and transmission protocols to adapt to different data acquisition needs while guaranteeing the security and efficiency of data transmission. Furthermore, the execution status of data acquisition tasks is monitored in real time, including task progress, error messages, and performance metrics. Task monitoring information is displayed through a visual interface, enabling administrators to promptly identify and handle anomalies during the data acquisition process, ensuring the smooth execution of data acquisition tasks.

[0020] This application's data acquisition layer, built upon a data acquisition tool, successfully achieves efficient data synchronization and processing between different heterogeneous data sources. Through an intuitive and easy-to-use visual configuration interface, it supports a data processing mode combining batch and stream processing, and can flexibly interface and integrate with Oracle, MySQL, and other systems.

[0021] Furthermore, through an intelligently optimized execution engine, it significantly improves throughput compared to traditional ETL tools under the same resource configuration. It also provides fine-grained access control to ensure data processing security and compliance. Whether for real-time or batch data processing, it offers efficient and reliable data integration solutions to meet the needs of various complex business scenarios.

[0022] (ii) Data integration layer.

[0023] The data integration layer is used to integrate the multi-source heterogeneous data and build a data warehouse. Data integration refers to the process of combining data scattered from different sources, in different formats, and with different structures into a unified, coherent, and valuable dataset through a series of technical means and processes. Its core objective is to eliminate data silos, resolve inconsistencies between data, and enable previously isolated data to work together to provide reliable data support for data analysis, business decision-making, and application system operation.

[0024] In a specific application example, the data integration layer uses Apache Seatunnel to integrate the multi-source heterogeneous data. The data integration layer supports both SQL (Structured Query Language) and DSL (Domain-Specific Language) development, providing the flexibility of dual-mode development. Furthermore, it integrates a CDC (Change Data Capture) mechanism to ensure end-to-end data consistency from the data source to the data asset management system. Simultaneously, it supports a combined batch and stream processing mode, allowing for flexible data configuration and management through an intuitive visual configuration interface.

[0025] Specifically, the data integration layer uses Apache Seatunnel to extract data from the business database to the data warehouse. The business database comprises various distributed business systems. For example, in high-altitude simulation testing of aero-engines, multiple test benches are used, each corresponding to a business system. Figure 2 As shown, a data warehouse comprises a raw data layer, a detailed data layer, a data aggregation layer, and a dimension layer. The data integration layer handles data processing including data cleaning, data transformation, data transmission, and data loading.

[0026] (1) The raw data layer (ODS) is used to store heterogeneous data from multiple sources.

[0027] (2) The Detailed Data Layer (DWD) is used to perform data cleaning, data transformation, dimension degradation and desensitization on the multi-source heterogeneous data stored in the original data layer to obtain detailed data.

[0028] Specifically, the structure of the detailed data layer is highly consistent with that of the original table. It is mainly responsible for cleaning the data in the original data layer (removing null values, dirty data, and data exceeding the limit range), data transformation, dimension degradation, and desensitization, thereby improving the quality and consistency of the data. It also uses dimension-based modeling, detailed wide tables, and reuses relational calculations to reduce data scanning.

[0029] Data cleaning involves filtering out data that does not meet the requirements, specifically including cleaning missing values, cleaning formatted content, and cleaning logical errors.

[0030] 1) Missing value cleaning. Remove unnecessary fields; fill missing content: fill missing fields with default values ​​or mean, median, mode, etc.; re-fetch data: if some field data is very important, but the data missing is severe, it is necessary to obtain the data again through other channels (such as data entry) and fill it in.

[0031] 2) Content Cleaning. Filter out inconsistent data regarding dates, times, values, and full-width / half-width characters, and process them into a consistent format. Content may contain unwanted characters, and some content may only contain a subset of valid characters (e.g., ID numbers should consist of numbers and letters). The most common problems include leading, trailing, and middle spaces; other issues include names containing numbers or symbols, and ID numbers containing Chinese characters. To address these problems, a combination of semi-automatic validation and semi-manual review should be used to identify potential issues and remove irrelevant characters.

[0032] 3) Logic error cleaning. Remove duplicate descriptions and values ​​that are outside the normal range.

[0033] The main tasks of data transformation are to transform inconsistent data, change the granularity of data, and calculate some business rules.

[0034] 1) Inconsistent data conversion is an integration process that unifies the same type of data from different business systems. For example, if the equipment failure occurred on January 15, 2012 at 12:20, but the reported failure date was January 5, 2012, the data will be extracted and converted into the system's specified format.

[0035] 2) Data granularity conversion: For example, if unit A reports flight time in minutes and unit B reports flight time in hours, the data will be uniformly converted into the system's specified format after extraction.

[0036] 3) Calculation of business rules: Different units have different business rules. For example, the failure rate is calculated by usage hours for unit A and by usage frequency for unit B. After extraction, it is recalculated according to the rules defined by the system.

[0037] (3) The data aggregation layer (DWS / ADS) is used to summarize and aggregate the detailed data to obtain the summary data.

[0038] Specifically, the data aggregation layer is primarily responsible for summarizing and aggregating the detailed data from the detailed data layer, forming aggregated data that can be directly used for analysis. Through aggregation operations, the amount of data can be reduced, query efficiency can be improved, and efficient data services can be provided to upper-layer applications.

[0039] (4) The Dimension Layer (DIM) is used to store the dimensional data of the multi-source heterogeneous data.

[0040] Specifically, the dimension layer focuses on storing dimensional data. It defines computational logic by adding dimensional attributes and associating dimensions, completing the attribute definition process and establishing consistent data analysis dimension tables. To avoid redundant attributes for related dimensions in the dimensional model, dimension tables are constructed based on star or snowflake schemas.

[0041] The data integration layer employs dimensional modeling to construct the data warehouse. Dimensional modeling is a classic modeling method in the data warehouse field. It starts with the needs of analysis and decision-making, enabling rapid requirements analysis. The following uses engine testing as an example to illustrate the model construction process: First, the relationships between tables in the original data layer are analyzed. Then, fact tables and corresponding dimension tables are extracted based on these relationships. The corresponding dimension tables are aggregated into a single fact table to form a wide table, serving as the data subject of the data aggregation layer. The data application layer can then retrieve data from this data subject for data querying, analysis, mining, and other applications. A data warehouse is established based on the different business application needs of the engine, as shown in Table 1.

[0042] Table 1 Data Warehouse Example

[0043] In addition, the data integration layer is also used for data fusion, data association, and data tagging.

[0044] Specifically, data fusion: Based on the characteristics of experimental data operations, a unified experimental data organization structure (XBOM) is established, creating a multi-level tree structure to display experimental data from various stages, specialties, and types. Data aggregation and fusion functions are provided, extracting data from various data sources and automatically summarizing and merging it according to the unified organizational structure (XBOM), facilitating the fusion and application of multiple data types.

[0045] Data Association: Data is retrieved from data sources, cleaned, transformed, and integrated, and relationships are established with the original data. Based on the data business hierarchy, a complete equipment data structure tree is built to display the relationships between data and business in a tree structure.

[0046] Data tagging: Tagging the data in the detailed data layer (data supplementation, subject identification, etc.). The process includes: tag definition, tag processing, tag management, and tag services.

[0047] (iii) Data storage layer.

[0048] The data storage layer is used for structured storage of data in the data warehouse.

[0049] In a specific application example, the data storage layer is based on Apache Doris, combining distributed storage and MPP (Massively Parallel Processing) architecture to perform structured storage of data in the data warehouse, achieving efficient storage and fast retrieval of massive amounts of data. Furthermore, local SSD (SolidState Drive) caching accelerates access to frequently accessed data, enabling flexible data storage and dynamic allocation. The data storage layer supports data ingestion within seconds, meeting the needs of real-time data analysis.

[0050] Apache Doris is a high-performance MPP database designed for real-time analytics scenarios. It boasts fast query performance, high concurrency, and real-time analysis capabilities, making it particularly suitable for large-scale data analysis and report generation. Apache Doris employs a distributed architecture, allowing for horizontal scaling to meet ever-growing data storage and analysis needs. It supports standard SQL query language, enabling users to easily query and analyze data. Furthermore, Apache Doris provides rich data import and export functions, supporting the conversion and transmission of various data formats, facilitating data integration and sharing.

[0051] This application uses Apache Doris to store data, which has the following advantages: (1) High Availability: Apache Doris uses multi-replica storage for both metadata and data, and synchronizes data logs via the Quorum protocol. Data is considered successfully written once most replicas have completed the write process, ensuring cluster availability even if a few nodes fail. Apache Doris supports both local and remote disaster recovery, enabling dual-cluster master-slave mode. When some nodes fail, the cluster can automatically isolate the faulty nodes, preventing impact on overall cluster availability.

[0052] (2) High compatibility: Apache Doris is highly compatible with the MySQL protocol, supports standard SQL syntax, and covers most MySQL and Hive functions. This high compatibility allows users to seamlessly migrate and integrate existing applications and tools. Apache Doris supports the MySQL ecosystem, allowing users to connect to Doris via MySQL client tools, making operation and maintenance more convenient. Simultaneously, the MySQL protocol can be used to adapt and be compatible with BI reporting tools and data transmission tools, ensuring efficiency and stability during data analysis and data transmission.

[0053] (3) Real-time data warehouse: Real-time data warehouse services can be built based on Apache Doris. Apache Doris provides sub-second data ingestion capabilities, and incremental changes in the upstream online transaction database can be captured in Doris within seconds. Relying on acceleration methods such as vectorization engine, MPP architecture and pipeline execution engine, it can provide sub-second data query capabilities, thereby building a high-performance, low-latency real-time data warehouse platform.

[0054] (4) Flexible modeling: Apache Doris offers a variety of modeling methods, such as wide table model, pre-aggregated model, star / snowflake model, etc. When importing data, you can use computing engines such as Flink and Spark to flatten the data into a wide table and write it into Doris, or you can directly import the data into Doris and perform data modeling operations through views, materialized views, or real-time multi-table joins.

[0055] (iv) Data governance layer.

[0056] The data governance layer is used to standardize and manage the quality of the data stored in the data storage layer to obtain the target data. Data governance can effectively ensure that the data construction process is carried out under a reasonable and efficient regulatory system, ultimately providing high-quality, secure, and traceable business data.

[0057] In a specific application example, such as Figure 3 As shown, the data governance layer includes: a metadata management module, a data standards management module, a data quality management module, and a data security management module. The functions and implementation schemes of each module are described in detail below.

[0058] (1) Metadata Management Module: This module manages the metadata of the data stored in the data storage layer. The metadata describes the attributes and context of the data. Metadata management is a core component of data governance, involving the management of information such as the definition, structure, source, quality, and security of data. The main objective of metadata management is to ensure the discoverability, understandability, and manageability of data, thereby improving the utilization efficiency and value of data assets.

[0059] This application provides metadata definition functionality covering the entire data domain and enables multi-level and different granularity metadata segmentation capabilities. Metadata is divided into three categories: business metadata (such as key features of scientific research tasks, such as model, stage, product number, etc.), management metadata (such as key features of data name, business department, version, permissions, etc.), and technical metadata (such as key features of data physical format, data size, storage location, etc.).

[0060] Business metadata describes the data from several aspects, including business requirements, business metrics, and business rules, as shown in Table 2.

[0061] Table 2 Examples of Business Metadata

[0062] Management metadata describes information such as data processing, data application, data status, and data security from the perspective of operation and maintenance management, as shown in Table 3.

[0063] Table 3 Example of managing metadata

[0064] Technical metadata provides a technical description of the data from the perspectives of data structure and data storage, as shown in Table 4.

[0065] Table 4 Examples of Technical Metadata

[0066] In a specific application example, the metadata management module works as follows: 1) Metadata Collection: Responsible for collecting metadata information from various data sources, including detailed descriptions of data name, type, format, relationship, and source. Automated tools are used to achieve real-time updates and synchronization of metadata, ensuring the accuracy and timeliness of the metadata information.

[0067] 2) Metamodel Management: Establish a centralized metadata repository to store and manage all collected metadata. The repository needs to have efficient data retrieval and management capabilities, support access control for different user roles, and version control and historical management of metadata.

[0068] 3) Metadata Maintenance: Regularly maintain and update metadata to ensure its accuracy and completeness. This includes cleaning up outdated metadata, updating metadata definitions and descriptions, and optimizing the performance of the metadata repository.

[0069] 4) Metadata Retrieval: Provides powerful metadata retrieval capabilities, allowing users to quickly locate the required metadata using keywords, data types, data sources, and other criteria. Retrieval results should include detailed information about the metadata, such as data definition, structure, and source, so that users can fully understand the context and purpose of the data.

[0070] (2) Data Standards Management Module: This module is used to formulate unified data standards and specifications, including data naming, format, and encoding rules, to ensure data uniformity and interoperability. The data standards management module provides support for data storage, ensuring the standardization and compliance of data storage.

[0071] Data standards include data definition standards (clarifying the meaning and purpose of data items), data format standards (specifying data encoding, units, precision, etc.), and data quality standards (setting requirements for data integrity, accuracy, and timeliness), such as naming conventions (e.g., DWD_Engine Model_Test Subject_Increment); and security classifications (e.g., "Turbine Temperature = Confidential"). In a specific application example, the data standards management module works as follows: 1) Scope of Data Standard Implementation: In this application, the data originates from multiple systems. To ensure data consistency and accuracy, the scope of the data standard implementation covers all key business processes and data exchange points. The scope of the data standard implementation includes: engine series, engine models, test benches, test tasks, test subject information, test data channel symbols, names, etc.

[0072] 2) Data Standards Release: This involves the creation, updating, and repealing of data standards. Key functions include: Data Standard Creation: Developing new data standards based on the organization's business needs and data governance strategies, including data naming rules, data formats, and data types; Data Standard Updates: Regularly evaluating and updating existing data standards as business grows and technology advances to ensure they continue to meet the organization's needs; Data Standard Repeal: Repealing data standards when they are no longer applicable or replaced by better standards, and notifying all relevant personnel and systems.

[0073] 3) Data standard mapping: This involves connecting and converting internal data standards with external standards, including identifying external data standards, establishing mapping rules, and ensuring the consistency and accuracy of data when exchanged between different systems.

[0074] 4) Data Standards Query: Ensures data standards are correctly implemented, including regular data checks to ensure compliance with established standards. Key functions include: conducting regular compliance assessments of data to ensure that data format, naming conventions, and types conform to organizational standards.

[0075] (3) Data Quality Management Module: Used to verify the quality of data, including verification of data accuracy, completeness, consistency, timeliness and availability. Specifically, it includes data cleaning (removing duplicate, erroneous or incomplete data), null value verification (checking according to the meta-model "required field" rules), data validation (checking whether the data meets the preset rules and standards), value range verification (checking whether the temperature value exceeds the limit according to the standard), and data monitoring (continuously tracking the data status and promptly discovering and solving problems).

[0076] The data quality management module supports quality verification, notification, and management services for various heterogeneous data sources, resolving various data quality issues during system operation, data center construction, and data governance. It relies on a computing engine middleware for data computation, providing functions such as data quality model building, data quality model execution, data quality task management, anomaly data discovery and storage, and data quality report generation.

[0077] In a specific application example, the data quality management module works as follows: 1) Rule Management. Configurable rule types include: null values; data duplication; referential integrity; data consistency; data fluctuation (year-on-year, month-on-month); data distribution audit; data format validation (commonly including time, ID number, phone number, etc.); and logical validation between different business data, including record-level and field-level comparisons.

[0078] 2) Data quality inspection. Establish quality inspection tasks to address potential quality issues that may arise during the storage and use of various types of data.

[0079] Specifically, the computing engine middleware performs periodic quality checks on the data. During the checks, it automatically identifies anomalies in the data, such as duplicate records, format errors, and data inconsistencies, and records this information in a data quality report.

[0080] 3) Rectification of data quality issues. Supports rectification of data object quality issue detection results.

[0081] Specifically, based on the analysis results in the data quality report, a data quality improvement plan is developed and implemented. This includes measures such as correcting data sources, data cleaning, data integration, and updating data validation rules to continuously improve the overall quality of the data.

[0082] (4) Data Security Management Module: Used to configure security rules, identify sensitive data, and perform data anonymization. Through data security management, it strengthens data access control, encrypted storage, and transmission security measures, and conducts regular data security audits and risk assessments to ensure the security of data during collection, storage, processing, transmission, and use.

[0083] Data security management includes key functions such as the formulation and enforcement of security rules, the identification and protection of sensitive data, and data anonymization, in order to ensure the confidentiality, integrity, and availability of data.

[0084] In a specific application example, the data security management module works as follows: 1) Security rule configuration: Establish a complete data security rule system, clarify data access permissions, encryption requirements, storage specifications, etc., and ensure that these rules are strictly enforced to prevent data leakage, tampering and other security incidents.

[0085] 2) Sensitive data identification: Based on security rules, advanced algorithms and tools are used to comprehensively scan and analyze the data on the platform, and automatically identify data items containing sensitive information.

[0086] 3) Data desensitization: For identified sensitive data, desensitization is performed to replace key information in the sensitive data with invalid or fictitious data in order to protect the privacy and security of the data.

[0087] In addition, the data governance layer also includes a data asset management module, which specifically includes asset cataloging, asset retrieval, and asset services.

[0088] In one exemplary embodiment, the governance process for aero-engine test data is as follows: Standard Management: Define Field Standard: Rotational Speed ​​Value (required, range 0~120%).

[0089] Metadata Management: Create a meta-model: Experimental data table (including speed and temperature fields).

[0090] Quality verification: A 5% missing speed value was detected, triggering an alarm.

[0091] Safety Management: The "Speed ​​Rotation" value is marked as internal public level and dynamically anonymized.

[0092] (v) Data application layer.

[0093] The data application layer is used for dynamic tracking, inventory, classification, and hierarchical management of data assets based on the target data.

[0094] The data application layer is the most direct manifestation of data value, providing analysis results to upper-level applications, such as building thematic libraries and topic libraries, specifically including asset portals, asset catalog management, asset maps, asset analysis, comprehensive queries, visualization analysis, data mining, and data sharing.

[0095] (1) Asset Portal: This is a user interface that provides a centralized entry point for users to access and manage assets. Its functions include asset search, asset details display, and asset usage statistics. The asset portal is designed to simplify the discovery, access, and use of data assets, while providing the necessary tools and services to support the lifecycle management of data assets.

[0096] (2) Asset Catalog Management: Construct a unified data sharing catalog to classify and manage shareable data resources according to business themes, data types, and other dimensions. Users can quickly find the data they need through the catalog, improving data acquisition efficiency. At the same time, it supports access control for the shared catalog to ensure that only authorized users can access specific data.

[0097] Asset catalog management includes: First, creating thematic catalogs based on asset topics, defining the catalog's structure and classification rules, and updating and maintaining the catalog as business develops and data resources change to ensure accuracy and timeliness. Version control is also provided to record the catalog's change history for easy tracking and management. Next, a registration portal is provided for data resources within the organization, allowing data owners to add data resources to the asset catalog according to established rules. During registration, a description of the data resource is required, including key information such as the data's name, type, source, and purpose. Finally, a catalog search function is provided, allowing users to quickly find the data resources they need based on different criteria.

[0098] (3) Asset Map: Provides an intuitive view of the distribution of data resources, helping users understand the physical and logical location of data assets. Through the data asset map, users can intuitively see the location of data storage, the path of data flow, and the relationships between data, which helps to optimize the data architecture, improve data access efficiency, and provide decision support for data governance.

[0099] The asset map provides both a panoramic view and search functionality. The panoramic view offers an overview of all data assets within the organization, including data type, quantity, storage location, and data flow, helping users fully understand the distribution and status of data assets and providing a basis for developing data management strategies. Asset search allows users to quickly locate desired data resources using keywords, tags, or other attributes, supporting various search criteria to ensure users can efficiently find the data they need. Furthermore, data asset search supports sorting, filtering, and analyzing search results, enabling users to obtain the information they require more precisely.

[0100] (4) Asset Analysis: By collecting and analyzing data usage, organizations can understand the usage patterns and value of their data assets. This includes analytical indicators such as data access frequency, data usage efficiency, and data value assessment. Through these analyses, data hotspots and coldspots can be identified, data resource allocation can be optimized, and the overall value of data can be improved.

[0101] Asset analysis provides asset overview and asset catalog analysis and statistics. The asset overview offers a holistic view of data assets, including their usage, storage, and distribution. Through the asset overview, organizations can quickly grasp the overall status of their data assets, identify key data resources, and formulate corresponding data management strategies. The asset catalog distribution statistics display the distribution of data assets across different catalogs, helping organizations understand which catalogs are the primary storage locations for data resources. This statistical information allows for the assessment of the rationality of the existing catalog structure, the identification of centralized or decentralized trends in data resources, and the optimization of the catalog structure to ensure the rational distribution and efficient utilization of data resources.

[0102] In addition, asset analysis also helps experimenters quickly build various data applications, such as data queries, data reports, data charts, and data analysis reports, by providing rich data service interfaces, data visualization tools, and data analysis models.

[0103] (5) Comprehensive Query: Allows users to retrieve and filter data using natural language or structured query language through a user-friendly interface. Users can sort, group, and summarize data as needed to gain deeper insights, such as querying data by test project, test subject, test time, engine model, etc.

[0104] Data querying includes multi-source business data aggregation queries and multi-dimensional data correlation queries. Multi-source business data aggregation queries support data aggregation queries across multiple data sources, enabling data integration and correlation analysis. Users can aggregate data from different systems and databases into a unified view for comprehensive queries. Multi-dimensional data correlation queries support multi-dimensional data correlation queries, using OLAP (Online Analytical Processing) technology to perform multi-angle and multi-level queries on data. For example, test data can be queried and analyzed according to multiple dimensions such as time, project, and engine model to discover correlations and trends between data.

[0105] (6) Visualization Analysis: Based on the specific business needs of whole-machine test result analysis, relevant business analysis tools are provided to analyze and apply the test data. A rich set of built-in data visualization components allows users to quickly construct data visualization charts through configuration. Customizable chart query conditions are provided; chart types, X-axis, Y-axis, colors, and styles can be customized based on the test data to intuitively display the data distribution and trends. Chart parameters can be adjusted to meet different display needs. For example, a line chart can be used to display the trend of test data over time, or a bar chart can be used to compare the performance indicators of different engine models.

[0106] (7) Data Mining: By introducing advanced technologies such as machine learning and deep learning, complex data analysis models can be constructed, such as engine profiling and predictive models. These models can help testers better understand the trends and patterns behind the data, thereby making more accurate predictions and decisions. For example, by integrating StarRing data mining tools and analyzing historical test data, engine performance prediction models can be constructed to predict engine performance under different conditions, providing data support for the development of new engines. In addition, data mining functions can also help identify potential risks and anomalies, ensuring test safety.

[0107] In this application, such as Figure 4 As shown, Apache Seatunnel is used to extract data from the business database into the data warehouse, and Apache Doris is used as the storage and computing platform. It can return query results for massive amounts of data with only sub-second response time, and can support high concurrency and high throughput query scenarios. Apache Doris can well meet the use cases of report analysis, real-time query, and unified data warehouse construction. Its features include MPP architecture, columnar storage, compressed storage, standard SQL, vectorized execution, storage-compute separation, simple architecture, excellent performance, easy operation and maintenance, and online-offline integration.

[0108] Furthermore, Apache DolphinScheduler is used to manage the workflow task scheduling of the entire system, resolving complex big data task dependencies and providing data and various OPS orchestration relationships for the application. Intelligent data analysis and mining are achieved through engines such as Apache Flink, supporting real-time data processing and complex analysis to meet the data analysis needs of different business scenarios. This application also provides a distributed scheduling center to achieve unified scheduling and management of tasks, support the resolution of complex big data task dependencies, and ensure the efficiency and reliability of data processing.

[0109] This application also provides an application scenario where the aforementioned data asset management system is applied. Specifically, the data asset management system provided in this embodiment can be applied to the data asset management scenario of high-altitude simulation tests for aero-engines. High-altitude simulation test data for aero-engines is scattered across various test bench systems. A data integration layer breaks down data silos, unifying multi-source heterogeneous data. A data storage layer constructs a traceable and reliable data asset repository. A data governance layer ensures data quality and security, supporting reliable analysis. A data application layer enables data-driven decision-making and knowledge accumulation, allowing for data visualization analysis and data mining. This solves the pain points of data dispersion, low utilization, and delayed analysis in high-altitude aero-engine tests, saving test costs and improving R&D efficiency.

[0110] Based on the same inventive concept, such as Figure 5 As shown in the embodiment of this application, a data asset management method is also provided, including the following steps 501 to 505.

[0111] Step 501: Collect multi-source heterogeneous data through the data acquisition layer. Specifically, collect data from industrial interconnected devices in real time via the OPCUA protocol; and perform batch data collection from the business database using SQL statements. The collected data is stored in the time-series database of the data hybrid storage module.

[0112] Step 502: Integrate the multi-source heterogeneous data through the data integration layer to build a data warehouse. Specifically, extract data from databases (such as MySQL, Oracle, DM, etc.) through a JDBC connector; feed the data into Apache Doris through a Kafka connector; and perform scheduled data synchronization through batch tasks. This achieves real-time data collection and batch synchronization, ensuring data timeliness and consistency.

[0113] Step 503: The data in the data warehouse is stored in a structured manner through the data storage layer. Specifically, Doris distributed storage is used; hot data is cached using local SSDs; columnar storage and vectorized execution of data are implemented through a distributed computing engine; and high-concurrency access and scalability of data are achieved through sharding and replication mechanisms.

[0114] Step 504 involves standardizing and quality-managing the data stored in the data storage layer through the data governance layer to obtain the target data. Specifically, the metadata management module automatically collects data metadata; the data standard management module formulates data standards; the data quality management module performs data quality testing; and the data security management module classifies, grades, and encrypts the data for protection. This achieves standardized and secure data management.

[0115] Step 505 involves the data application layer dynamically tracking, inventorying, and classifying data assets based on the target data. Specifically, the data development module performs data cleaning and transformation; the data analysis module performs real-time analysis and complex calculations; the data service management module enables data sharing and retrieval; and the data lifecycle management module facilitates automatic archiving and destruction. This achieves efficient data analysis and maximizes data utilization.

[0116] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the method embodiments described above.

[0117] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps described in the method embodiments above.

[0118] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the method embodiments described above.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0120] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0122] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data asset management system, characterized in that, The system includes: The data acquisition layer is used to collect heterogeneous data from multiple sources. The data integration layer is used to integrate the multi-source heterogeneous data and build a data warehouse; A data storage layer is used for structured storage of data in the data warehouse; The data governance layer is used to standardize and manage the quality of the data stored in the data storage layer to obtain the target data. The data application layer is used for dynamic tracking, inventory, classification, and hierarchical management of data assets based on the target data.

2. The data asset management system according to claim 1, characterized in that, The multi-source heterogeneous data refers to high-altitude simulation test data of aero-engines.

3. The data asset management system according to claim 1, characterized in that, The data acquisition layer supports both batch acquisition and real-time acquisition.

4. The data asset management system according to claim 1, characterized in that, The data acquisition layer collects data from industrial interconnected devices in real time via the OPCUA protocol and performs batch acquisition of data from the business database to obtain multi-source heterogeneous data.

5. The data asset management system according to claim 1, characterized in that, The data integration layer uses Apache Seatunnel to integrate the multi-source heterogeneous data.

6. The data asset management system according to claim 1, characterized in that, The data warehouse includes a raw data layer, a detailed data layer, a data aggregation layer, and a dimension layer; The raw data layer is used to store multi-source heterogeneous data; The detailed data layer is used to clean, transform, de-dimensionalize, and de-identify the multi-source heterogeneous data stored in the original data layer to obtain detailed data. The data aggregation layer is used to summarize and aggregate the detailed data to obtain summary data; The dimension layer is used to store the dimension data of the multi-source heterogeneous data.

7. The data asset management system according to claim 1, characterized in that, The data integration layer uses dimensional modeling to construct the data warehouse.

8. The data asset management system according to claim 1, characterized in that, The data storage layer is based on Apache Doris, and combines distributed storage and massive parallel processing architecture to perform structured storage of data in the data warehouse.

9. The data asset management system according to claim 1, characterized in that, The data governance layer includes: The metadata management module is used to manage the metadata of the data stored in the data storage layer; the metadata is used to describe the attributes and background of the data. The data standards management module is used to develop unified data standards and specifications to ensure data consistency and interoperability. The data quality management module is used to verify the quality of data; The data security management module is used to configure security rules, identify sensitive data, and perform data anonymization.

10. A data asset management method, characterized in that, The method applies the data asset management system according to any one of claims 1-9, and the method includes: Collect multi-source heterogeneous data through the data acquisition layer; A data warehouse is constructed by integrating the multi-source heterogeneous data through a data integration layer; The data in the data warehouse is stored in a structured manner through a data storage layer; The data governance layer standardizes and manages the quality of the data stored in the data storage layer to obtain the target data. The data application layer performs dynamic tracking, inventory, classification, and hierarchical management of data assets based on the target data.

Images