Big data asset management methods, devices, equipment, media and products
By adopting the data warehouse layering concept and asset flow management, the problem of low efficiency in big data asset management under the ODS-DWD approach has been solved, realizing clear management and self-service analysis throughout the entire lifecycle, and improving asset management efficiency and decision support capabilities.
Patent Information
- Application Number
- CN202410874072.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-07-01
AI Technical Summary
Existing methods for maintaining big data asset information through ODS-DWD are insufficient for managing big data assets quickly, efficiently, and clearly, especially when the inflow and outflow characteristics of asset management are not effectively combined, resulting in low management efficiency.
Adopting a data warehouse layering concept, big data assets are managed in layers based on their flow, including data inflow and outflow layers. Wide tables are generated and linked through unique asset identifiers to facilitate data asset analysis.
It enables clear management of big data assets throughout their entire lifecycle, improves asset management efficiency, allows for flexible self-service analysis, generates product operation analysis reports, and provides important decision-making basis such as return on investment.
Smart Images

Figure CN118861141B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a big data asset management method, a big data asset management device, a big data asset management equipment, a computer-readable storage medium, and a computer program product. Background Technology
[0002] Currently, big data asset information is generally maintained through a data warehouse (ODS) - DWD (Data Warehouse Detail Layer). The ODS is a database or data storage area used to store operational data from business systems. The DWD is an important part of the data warehouse, used to store detailed data that has been cleaned, integrated, and processed.
[0003] In typical data warehouse construction, the Operating Data Store (ODS) and Data Warehouse Development (DWD) are usually crucial links in the data flow. First, data is extracted from business systems and stored in the ODS as raw, real-time operational data. Then, after cleaning, processing, and integration, the data is loaded into the DWD, forming a detailed data layer suitable for analysis and application. Finally, applications such as data marts, data reports, and data visualizations can be built from the DWD to support enterprise decision-making and business optimization.
[0004] However, with the significant increase in the types and quantities of big data assets, as well as the increase in allocation objects and circulation links, maintaining asset information through the hierarchical aggregation method of ODS-DWD has become difficult to maintain big data assets quickly, efficiently, and clearly because it fails to take into account the inflow and outflow characteristics of asset management. There is an urgent need to build a big data asset management system based on the new data warehouse concept. Summary of the Invention
[0005] The main purpose of this application is to provide a big data asset management method, a big data asset management device, a big data asset management equipment, a computer-readable storage medium, and a computer program product, which aim to improve the efficiency of big data asset maintenance and management.
[0006] To achieve the above objectives, this application provides a big data asset management method, the method comprising:
[0007] Acquire asset information data of big data assets, wherein the big data assets include mainframes, computing resources, data analysis tools, and data model assets;
[0008] Based on the asset flow in the asset information data of the big data asset, the asset information data of the big data asset is layered into a data warehouse to obtain the data warehouse table of the big data asset. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data asset.
[0009] By merging the data warehouse tables of the data inflow layer and data outflow layer of the big data asset, a wide table of the big data asset is obtained;
[0010] Data asset analysis is performed based on the wide table of the big data assets.
[0011] For example, when the big data asset is a host, the step of performing data warehouse layering on the asset information data of the big data asset to obtain the data warehouse table of the big data asset includes:
[0012] The asset information data of the host is layered into data warehouses to obtain a first data warehouse table, a second data warehouse table, and a third data warehouse table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0013] For example, when the big data asset is a host, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0014] The first data warehouse table, the second data warehouse table, and the third data warehouse table of the host are associated with each other through a unique asset identifier to generate a wide table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0015] For example, when the big data asset is a computing resource, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0016] The data warehouse table storing the inflow information of the computing resources in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the computing resources.
[0017] For example, when the big data asset is a data analysis tool, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0018] The data warehouse table storing the inflow information of the data analysis tool, located in the data inflow layer, is associated with the data warehouse table containing all big data assets, located in the data outflow layer, through a unique asset identifier, to generate a wide table for the data analysis tool.
[0019] For example, when the big data asset is a data model, the step of fusing the data warehouse tables of the data inflow layer and the data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0020] The data warehouse table storing the data model inflow information in the data inflow layer is associated with the data warehouse table containing all big data assets in the data outflow layer through a unique asset identifier to generate the wide table of the data model.
[0021] This application also provides a big data asset management device, the big data asset management device comprising:
[0022] The acquisition module is used to acquire asset information data of big data assets, wherein the big data assets include host, computing resources, data analysis tools and data model assets;
[0023] The layering module is used to perform data warehouse layering on the asset information data of the big data asset according to the asset flow direction in the asset information data of the big data asset, so as to obtain the data warehouse table of the big data asset. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data asset.
[0024] The fusion module is used to merge the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset.
[0025] The analysis module is used to perform data asset analysis based on the wide table of the big data assets.
[0026] This application also provides a big data asset management device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the big data asset management method described above.
[0027] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the big data asset management method described above.
[0028] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the big data asset management method described above.
[0029] This application proposes a big data asset management method, device, equipment, computer-readable storage medium, and computer program product. The method acquires asset information data of big data assets, wherein the big data assets include host computers, computing resources, data analysis tools, and data model assets. Based on the asset flow in the asset information data, the method performs data warehouse layering on the asset information data of the big data assets to obtain a data warehouse table for the big data assets. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data assets. The method merges the data warehouse tables from the data inflow and data outflow layers of the big data assets to obtain a wide table for the big data assets. Data asset analysis is then performed based on the wide table of the big data assets.
[0030] In this application, based on the asset flow in the asset information data of big data assets, the data warehouse is layered into data inflow and data outflow layers for big data assets, including mainframes, computing resources, data analysis tools, and data models, resulting in a data warehouse table for big data assets. By merging the data warehouse tables from the data inflow and data outflow layers, a wide table for big data assets is obtained, and data asset analysis is performed based on this wide table. Therefore, based on the new concepts of data governance and data warehouse management, a layered data warehouse table is designed according to the inflow and outflow of big data assets, managing the entire lifecycle of all big data assets (including mainframes, computing resources, data analysis tools, and data model assets) from access and allocation to failure. Furthermore, online asset management can also assist in the cost and benefit management of big data assets. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of the structure of the operating device of the hardware operating environment involved in the embodiments of this application;
[0032] Figure 2 This is a flowchart illustrating an embodiment of the big data asset management method involved in the present application.
[0033] Figure 3 This is a schematic diagram of the design architecture of an embodiment of the big data asset management method involved in the present application.
[0034] Figure 4 This is a schematic diagram of the big data asset management device involved in the embodiments of this application.
[0035] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0036] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0037] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment structure involved in the embodiments of this application.
[0038] like Figure 1 As shown, the operating device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0039] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the operating equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0040] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.
[0041] exist Figure 1In the illustrated operating device, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the operating device of this application can be disposed in the operating device, and the operating device calls the computer program stored in the memory 1005 through the processor 1001 and performs the following operations:
[0042] Acquire asset information data of big data assets, wherein the big data assets include mainframes, computing resources, data analysis tools, and data model assets;
[0043] Based on the asset flow in the asset information data of the big data asset, the asset information data of the big data asset is layered into a data warehouse to obtain the data warehouse table of the big data asset. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data asset.
[0044] By merging the data warehouse tables of the data inflow layer and data outflow layer of the big data asset, a wide table of the big data asset is obtained;
[0045] Data asset analysis is performed based on the wide table of the big data assets.
[0046] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:
[0047] When the big data asset is the host, the step of performing data warehouse layering on the asset information data of the big data asset to obtain the data warehouse table of the big data asset includes:
[0048] The asset information data of the host is layered into data warehouses to obtain a first data warehouse table, a second data warehouse table, and a third data warehouse table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0049] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:
[0050] When the big data asset is the host, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain the wide table of the big data asset includes:
[0051] The first data warehouse table, the second data warehouse table, and the third data warehouse table of the host are associated with each other through a unique asset identifier to generate a wide table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0052] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:
[0053] When the big data asset is a computing resource, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table for the big data asset includes:
[0054] The data warehouse table storing the inflow information of the computing resources in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the computing resources.
[0055] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:
[0056] When the big data asset is a data analysis tool, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0057] The data warehouse table storing the inflow information of the data analysis tool, located in the data inflow layer, is associated with the data warehouse table containing all big data assets, located in the data outflow layer, through a unique asset identifier, to generate a wide table for the data analysis tool.
[0058] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:
[0059] When the big data asset is a data model, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0060] The data warehouse table storing the data model inflow information in the data inflow layer is associated with the data warehouse table containing all big data assets in the data outflow layer through a unique asset identifier to generate the wide table of the data model.
[0061] This application provides a big data asset management method, referring to... Figure 2 In one embodiment of the big data asset management method, the method includes:
[0062] Step S10: Obtain asset information data of big data assets, wherein the big data assets include host, computing resources, data analysis tools and data model assets;
[0063] In one embodiment, reference is made to Figure 3 The first step involves managing the purchase and allocation of all big data infrastructure mainframes. This includes managing mainframe equipment codes, IP addresses, capacity information, activation dates, allocation dates, and expiration dates, thus obtaining mainframe allocation information. The second step involves managing big data storage and computing resources and data analysis tools, managing the construction and cost flow of all big data platform and tool resources, thus obtaining storage and computing resource and tool allocation information. The third step is the management of the data capability construction area (middle platform data capability area). Resources from the first step, such as bare metal mainframes, and resources from the second step, such as storage and computing resources and data analysis tools, are allocated to the production domain. The production domain is used to develop big data infrastructure and application layer models, resulting in data model assets. The fourth step involves allocating all big data assets to tenant domains. The bare metal mainframes from the first step, the storage and computing resources and tools from the second step, and the data assets from the third step are all allocated to tenant domains. Tenants utilize these all big data assets to develop further application services for internal and external markets.
[0064] Step S20: Based on the asset flow direction in the asset information data of the big data asset, perform data warehouse layering on the asset information data of the big data asset to obtain the data warehouse table of the big data asset. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data asset.
[0065] A data warehouse is a dedicated database used to integrate, store, and manage various types of data within an enterprise. Data warehouses are typically used to support an enterprise's decision analysis and business intelligence needs; they are subject-oriented, integrated, and non-volatile data storage areas.
[0066] A data warehouse table is a tabular structure used to store and manage data in a data warehouse system. A data warehouse is a system for integrating, storing, and analyzing large amounts of data, and data warehouse tables are a fundamental component of it.
[0067] In this embodiment, the data warehouse for big data assets is layered. Unlike conventional data warehouse layering approaches, this method layers the data warehouse according to asset flow, defining the data responsibilities for each layer:
[0068] 1. Data Inflow Layer (DI): The DI layer records the data of assets that are accessed from external sources to the current management. This can be raw data accessed from upstream systems through interfaces or basic data maintained on the front end.
[0069] The data inflow layer can be further divided into a Data Inflow Detail layer (DID layer) and a Data Inflow Light Summary layer (DIM layer) as needed. The DID layer records the most detailed level of data inflow information, while the DIM layer performs light summary on the DID layer as needed, aggregating and merging key field data. This layer can create a DIM layer zipper table based on the DID.
[0070] 2. Data Outflow (DO) layer: The DO layer records data that is transferred from the current management to new objects.
[0071] The data outflow layer can be further divided as needed into a Data Outflow Details layer, a DOD layer (the outflow layer can be further divided according to outflow level, such as first-level outflow layer (DOD1) and second-level outflow layer (DOD2), etc.), and a Data Outflow Merge layer (DOM). The DOD layer records the most detailed level of data outflow information, while the DOM layer performs light aggregation on the DOD layer as needed, aggregating and merging key field data. This layer can create a DOM layer linked list based on the DOD.
[0072] (3) Wide Surface Layer (WT):
[0073] The WT layer is a wide table model that integrates data from multiple DIM and DOM layers, containing information on asset inflows and outflows. This wide table model allows for convenient drag-and-drop data processing using self-service data analysis tools, quickly generating diverse analytical reports (such as asset return rate information, asset completion rate information, etc.).
[0074] Step S30: Merge the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain the wide table of the big data asset;
[0075] Step S40: Perform data asset analysis based on the wide table of the big data assets.
[0076] A wide table is a table in a database or data storage system where each row contains a large number of columns. Compared to a narrow table, a wide table has more columns and is typically used to store entity data with multiple attributes or fields.
[0077] In this embodiment, the data warehouse tables of the data inflow layer and data outflow layer of big data assets are merged to obtain a wide table of big data assets, and data asset analysis is performed based on the wide table of big data assets.
[0078] Compared with existing technologies, this application proposes for the first time to use the concept of data warehouse for the lifecycle management of big data assets, and designs a data warehouse layered architecture suitable for big data asset management. It manages assets in layers according to asset flow. This method can better combine the attribute characteristics of big data assets themselves to carry out structured asset management. At the same time, the layered management concept of data warehouse can effectively help data governance, ensure the integrity and dynamic statistical nature of asset data maintenance, and realize more intelligent asset management.
[0079] Compared with existing technologies, this application proposes for the first time a data warehouse model based on full-scale big data assets (hosts, computing resources, tools, and data), and integrates wide tables, including wide tables for host access allocation, computing resource construction and allocation, tool construction and allocation, and data construction and allocation. Based on the wide tables, flexible self-service analysis can be performed to analyze information such as asset product usage and return on investment, helping to generate product operation analysis reports.
[0080] In this embodiment, the big data asset management method can clearly grasp the full lifecycle information of all big data assets, such as access and allocation, which helps to calculate the cost, revenue and profit of each type of detailed asset. It also facilitates the statistical analysis of the input-output ratio of each type of asset, thereby providing an important basis for product market investment selection.
[0081] In this embodiment, the big data asset management method can be used in unified operation and maintenance tools for managing host assets of big data tenant projects across the entire network. In metering and billing systems, it provides comprehensive asset metering statistics for the billing system. In open platform operation and maintenance management, it supports data on the completion status of daily operation and maintenance asset construction, generating product operation analysis reports and product return-to-output reports.
[0082] This application provides a big data asset management method. In another embodiment of the big data asset management method, when the big data asset is a host, the step of performing data warehouse layering on the asset information data of the big data asset to obtain the data warehouse table of the big data asset includes:
[0083] The asset information data of the host is layered into data warehouses to obtain a first data warehouse table, a second data warehouse table, and a third data warehouse table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0084] Based on the above embodiments, a data warehouse DI layer, DO layer and WT layer model for big data asset management is constructed.
[0085] When big data assets are the mainframes, an asset management model is designed for mainframe access and allocation.
[0086] (1) Host Access Management (Data Warehouse DI Layer): This involves information such as unique host identifier (including bare metal, container, and virtual machine), host access party, host source, host category, host IP, and host capacity, and maintains quota time, creation time, host status, and the current record's expiration time. Based on this, the first data warehouse table containing storage host access information located in the data inflow layer is obtained.
[0087] (2) Host allocation management (first layer allocation of the data warehouse DO layer, DOD1 layer): Host resources are allocated and managed, and the purpose of each host allocation (production or external tenant), delivery unit, delivery responsible person, secondary project and secondary purpose of delivery (whether it is for building storage computing resources such as Hadoop, Gbase, DACP, etc.) are recorded. Based on the latest host access information and current allocation status in (1), a daily full-database zipper table of host information is established. In this way, the second data warehouse table of storage host allocation information located in the data outflow layer is obtained.
[0088] (3) The host receiver maintains specific usage information (the first layer of the data warehouse DO layer allocates DOD2 layer): The host receiver needs to fill in the specific usage of the resource, including the name of the resource to be deployed, the detailed component name, whether it is a management node or a compute node, etc., to keep track of the actual flow of each host. In this way, the third data warehouse table of storage host usage information located in the data outflow layer is obtained.
[0089] For example, when the big data asset is a host, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0090] The first data warehouse table, the second data warehouse table, and the third data warehouse table of the host are associated with each other through a unique asset identifier to generate a wide table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0091] The first, second, and third data warehouse tables of the host are linked by the unique asset identifier, i.e. the unique host identifier, to generate a daily zipper table (wide table WT layer) showing the current status of all hosts in use.
[0092] This application provides a big data asset management method. In another embodiment of the big data asset management method, the step of performing data warehouse layering on the asset information data of the big data asset to obtain a data warehouse table of the big data asset includes:
[0093] Based on the unique identifier of each type of big data asset and the unique identifier of the allocation object, a data warehouse table of all big data assets located in the data outflow layer is generated.
[0094] In this embodiment, a big data full-scale asset allocation management model (DO layer) is designed to manage the allocation of host, computing resources, tools, and data assets. Based on the unique identifier of each asset type and the unique identifier of the allocation object, a full-scale big data asset allocation management model is generated. This results in a data warehouse table of full-scale big data assets located at the data outflow layer.
[0095] Next, the latest data for each day is retrieved, and wide tables for host access allocation, resource construction and allocation, and tool construction and allocation are generated respectively.
[0096] For example, when the big data asset is a computing resource, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0097] The data warehouse table storing the inflow information of the computing resources in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the computing resources.
[0098] When big data assets are used as computing resources, a computing resource asset management model is designed.
[0099] First, Hadoop and Gbase computing resources are built based on the various host resources that have already been delivered, and a new inflow management library model (data warehouse DI layer) is formed for this asset. This results in a data warehouse table located in the data inflow layer that stores the inflow information of the computing resources.
[0100] Then, the data warehouse table storing the inflow information of the computing resources located in the data inflow layer is associated with the data warehouse table of the full big data assets located in the data outflow layer, according to the unique asset identifier, to generate a construction and allocation model of the computing resources by cluster and component (Wide Table WT layer). Through this model, the cost input, revenue and profit of the cluster assets can be flexibly statistically analyzed.
[0101] For example, when the big data asset is a data analysis tool, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0102] The data warehouse table storing the inflow information of the data analysis tool, located in the data inflow layer, is associated with the data warehouse table containing all big data assets, located in the data outflow layer, through a unique asset identifier, to generate a wide table for the data analysis tool.
[0103] When big data assets are used as data analysis tools, a tool asset management model should be designed.
[0104] First, based on the various hosts and software resources already delivered, data acquisition tools, data development tools, data governance tools, and data reporting and analysis tools are built, forming a management model (data warehouse DI layer) for the inflow of these tool assets. This results in a data warehouse table located in the data inflow layer, storing the inflow information of the data analysis tools.
[0105] Then, the data warehouse table storing the data analysis tool inflow information located in the data inflow layer is associated with the data warehouse table of the full big data assets located in the data outflow layer, according to the unique asset identifier, to generate a daily zipper table (wide table WT layer) for tool construction and allocation. Through this model, the cost input, revenue and profit of the tool asset can be flexibly statistically analyzed.
[0106] For example, when the big data asset is a data model, the step of fusing the data warehouse tables of the data inflow layer and the data outflow layer of the big data asset to obtain a wide table of the big data asset includes:
[0107] The data warehouse table storing the data model inflow information in the data inflow layer is associated with the data warehouse table containing all big data assets in the data outflow layer through a unique asset identifier to generate the wide table of the data model.
[0108] When big data assets are data models, design a data capability building and management model (middle platform data capability area).
[0109] First, data assets are developed and generated based on Hadoop and MPP resources, as well as tools for data acquisition, development, governance, and report analysis. These assets are then applied to big data production operations, and a management model (data warehouse DI layer) is established for the inflow of these data assets. This results in a data warehouse table located in the data inflow layer, storing the inflow information of the aforementioned data model.
[0110] Then, the data warehouse table storing the data model inflow information located in the data inflow layer is associated with the data warehouse table of the full big data assets located in the data outflow layer, according to the unique asset identifier, to generate a daily data construction and allocation zipper table (wide table WT layer). Through this model, the cost input, revenue and profit of the data asset can be flexibly statistically analyzed.
[0111] The conventional data warehouse approach of aggregating data layer by layer fails to take into account the characteristics of asset inflows and outflows and the multi-stage flow of assets to manage the entire asset lifecycle. This includes difficulties in managing the flow of asset costs and application management. For example, among the tens of thousands of hosts allocated, it is difficult to determine whether a particular host is allocated to build Hadoop HBase components, GBASE, or tool assets. Furthermore, the lifecycle management of these newly created assets, such as creation, allocation, and decommissioning, lacks effective online management.
[0112] In the application scenario of the big data asset management method proposed in this application, unlike the previous management methods of manual offline maintenance or system registration of asset inbound, outbound, and offline management, a data warehouse management architecture for the entire life cycle of big data assets and an architecture scheme for asset cost flow management are designed. The entire process from big data host application to host allocation for building computing resources, host allocation for building tool resources, and allocation of computing resources and tools for production and construction of data assets is managed in a data warehouse. It also differs from the common data warehouse ODS-DWD layer design pattern. It standardizes the design method of each layer of the big data asset data warehouse model according to the inflow and outflow of assets. From DI (Data Asset Access Layer) -> DO (Data Asset Outflow Layer) -> WT (Data Wide Table Layer), the responsibilities of each layer of data are defined, making the data flow more orderly.
[0113] Taking hosts as an example: host application access model (DI layer) and host allocation model (DO layer); taking storage and computing resource construction as an example: storage and computing resource construction model (DI layer) and storage and computing resource allocation (DO layer); taking tool construction as an example: tool construction model (DI layer) and tool allocation (DO layer); taking data construction as an example: data construction model (DI layer) and data model allocation (DO layer); finally, based on the fusion of the DI layer and DO layer, four wide tables are generated, including a host access allocation wide table, a storage and computing resource construction and allocation wide table, a tool construction and allocation wide table, and a data construction and allocation wide table, so as to carry out flexible self-service analysis.
[0114] Reference Figure 4 Furthermore, this application embodiment also provides a big data asset management device, the big data asset management device comprising:
[0115] The acquisition module M1 is used to acquire asset information data of big data assets, wherein the big data assets include host, computing resources, data analysis tools and data model assets;
[0116] The layering module M2 is used to perform data warehouse layering on the asset information data of the big data asset according to the asset flow direction in the asset information data of the big data asset, so as to obtain the data warehouse table of the big data asset. The data warehouse layering includes a data inflow layer and a data outflow layer of the asset information data of the big data asset.
[0117] The fusion module M3 is used to fuse the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain the wide table of the big data asset;
[0118] The analysis module M4 is used to perform data asset analysis based on the wide table of the big data assets.
[0119] For example, when the big data asset is a host, the layering module is further used to:
[0120] The asset information data of the host is layered into data warehouses to obtain a first data warehouse table, a second data warehouse table, and a third data warehouse table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0121] For example, when the big data asset is a host, the fusion module is further configured to:
[0122] The first data warehouse table, the second data warehouse table, and the third data warehouse table of the host are associated with each other through a unique asset identifier to generate a wide table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
[0123] For example, when the big data asset is a storage and computing resource, the fusion module is further used to:
[0124] The data warehouse table storing the inflow information of the computing resources in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the computing resources.
[0125] For example, when the big data asset is a data analysis tool, the fusion module is further used to:
[0126] The data warehouse table storing the inflow information of the data analysis tool, located in the data inflow layer, is associated with the data warehouse table containing all big data assets, located in the data outflow layer, through a unique asset identifier, to generate a wide table for the data analysis tool.
[0127] For example, when the big data asset is a data model, the fusion module is further used to:
[0128] The data warehouse table storing the data model inflow information in the data inflow layer is associated with the data warehouse table containing all big data assets in the data outflow layer through a unique asset identifier to generate the wide table of the data model.
[0129] The big data asset management device provided in this application adopts the big data asset management method in the above embodiments, aiming to improve the efficiency of big data asset maintenance and management. Compared with conventional technology, the beneficial effects of the big data asset management device provided in this application are the same as those of the big data asset management method provided in the above embodiments, and other technical features in the big data asset management device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0130] Furthermore, this application embodiment also provides a big data asset management device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the big data asset management method described above.
[0131] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the big data asset management method described above.
[0132] Furthermore, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the big data asset management method described above.
[0133] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-described big data asset management method, and will not be repeated here.
[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to conventional technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0136] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A big data asset management method, characterized in that, The method includes: Acquire asset information data of big data assets, wherein the big data assets include mainframes, computing resources, data analysis tools, and data model assets; Based on the asset flow in the asset information data of the big data assets, the asset information data of the big data assets is layered into a data warehouse to obtain a data warehouse table for the big data assets. The data warehouse layer includes a data inflow layer and a data outflow layer for the asset information data of the big data assets. The data inflow layer records the data of assets accessed from external sources to the current management party, which is the original data accessed from the upstream system through an interface or the basic data maintained by the front end. The data outflow layer records the data of assets flowing out from the current management party and allocated to new objects. Based on the unique asset identifier and the unique identifier of the allocation object for each type of big data asset, a data warehouse table for all big data assets located in the data outflow layer is generated. By merging the data warehouse tables of the data inflow layer and data outflow layer of the big data asset, a wide table of the big data asset is obtained; Data asset analysis is performed based on the wide table of the big data assets.
2. The big data asset management method as described in claim 1, characterized in that, When the big data asset is the host, the step of performing data warehouse layering on the asset information data of the big data asset to obtain the data warehouse table of the big data asset includes: The asset information data of the host is layered into data warehouses to obtain a first data warehouse table, a second data warehouse table, and a third data warehouse table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
3. The big data asset management method as described in claim 1, characterized in that, When the big data asset is the host, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain the wide table of the big data asset includes: The first data warehouse table, the second data warehouse table, and the third data warehouse table of the host are associated with each other through a unique asset identifier to generate a wide table for the host. The first data warehouse table is a data warehouse table for storage host access information located in the data inflow layer, the second data warehouse table is a data warehouse table for storage host allocation information located in the data outflow layer, and the third data warehouse table is a data warehouse table for storage host usage information located in the data outflow layer.
4. The big data asset management method as described in claim 1, characterized in that, When the big data asset is a computing resource, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table for the big data asset includes: The data warehouse table storing the inflow information of the computing resources in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the computing resources.
5. The big data asset management method as described in claim 1, characterized in that, When the big data asset is a data analysis tool, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes: The data warehouse table storing the inflow information of the data analysis tool, located in the data inflow layer, is associated with the data warehouse table containing all big data assets, located in the data outflow layer, through a unique asset identifier, to generate a wide table for the data analysis tool.
6. The big data asset management method as described in claim 1, characterized in that, When the big data asset is a data model asset, the step of fusing the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset includes: The data warehouse table storing the inflow information of the data model assets in the data inflow layer is associated with the data warehouse table of the full big data assets in the data outflow layer through a unique asset identifier to generate a wide table of the data model assets.
7. A big data asset management device, characterized in that, The big data asset management device includes: The acquisition module is used to acquire asset information data of big data assets, wherein the big data assets include host, computing resources, data analysis tools and data model assets; The layering module is used to perform data warehouse layering on the asset information data of the big data assets based on the asset flow direction in the asset information data of the big data assets, thereby obtaining a data warehouse table for the big data assets. The data warehouse layering includes a data inflow layer and a data outflow layer for the asset information data of the big data assets. The data inflow layer records data of assets accessed from external sources to the current management entity, including raw data accessed from upstream systems via interfaces or basic data maintained on the front end. The data outflow layer records data of assets flowing out from the current management entity and allocated to new objects. Based on the unique asset identifier and the unique identifier of the allocated object for each type of big data asset, a data warehouse table for all big data assets located in the data outflow layer is generated. The fusion module is used to merge the data warehouse tables of the data inflow layer and data outflow layer of the big data asset to obtain a wide table of the big data asset. The analysis module is used to perform data asset analysis based on the wide table of the big data assets.
8. A big data asset management device, characterized in that, The big data asset management device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the big data asset management method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the big data asset management method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the big data asset management method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Data asset estimation method, system and device and data medium station
CN116341956A
Multi-bin management system for maintenance data of pavement regular inspection full life cycle
CN118134457A
