Data asset management system based on drilling full-service data center

Abstract

The invention discloses a data asset management system based on a drilling full-service data platform, and the system comprises four-transverse three-longitudinal frame division, four transverse represents the process of data capitalization, and three longitudinal represents a support system for the construction of the drilling data platform, through the construction of the data platform, the construction of the drilling data asset management system is realized, and the construction efficiency of the drilling data asset management system is improved. Unified acquisition and standard storage of data are formed, a unified data sharing service and a unified application architecture system are provided, the problem of information islands is effectively solved, and closed-loop management of data generated by business and data service business is promoted. The advanced big data technology is adopted, mass data are converted into high-quality data assets, informatization is promoted to be converted from management and control to an enabling role, deeper business insight is provided for the drilling industry, and high-quality development of businesses is assisted.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data asset management system based on a drilling full-business data middle platform. Background Art

[0002] With the accelerated development of the informatization process in the drilling industry, the amount of data related to production, technology, safety, finance, equipment, instruments, manpower, etc. involved in the drilling business has increased dramatically. On the one hand, enterprises need to effectively organize and manage a large amount of diversified data, and on the other hand, they need to provide data sharing, technical support, management decision-making and statistical analysis and other auxiliary support for technical personnel and management decision-makers. The isolated business system that used to improve work efficiency for enterprises has become a hindrance to the further development of the drilling industry.

[0003] The data middle platform aggregates and governs cross-domain data, abstracts and encapsulates data into services, and provides the logical concept of business value to the front desk. It is born to solve data integration, data standardization, data service, and data operation. The introduction of the theory and related technologies of the data middle platform hopes to optimize the data asset management model of the entire organization, activate the value of data, guide service innovation, and further strengthen the closeness of "data + business". Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the lack of appropriate and effective management means for the massive data assets generated in the existing drilling process, and to provide a data asset management method and system based on a drilling full-business data middle platform.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a data asset management system based on the drilling full business data middle platform, including a data assetization system and a support system. The data assetization system is a management platform that converts all objectively existing data involved in the existing drilling business into data assets and opens them for sharing. The support system provides data standards, data management, monitoring and application tools for the data middle platform.

[0006] The data assetization system includes:

[0007] The acquisition layer includes a data production environment module and a data access module. The data production environment module is used to perceive all objective data sources and types involved in the existing drilling business. The data access module is used to access data from different sources, different forms, and different protocols generated in the data production environment and allow the data to enter the data warehouse.

[0008] The data warehouse layer includes the original data domain module, the data processing module and the standard data domain module. The original data domain module is used to store original data. The data processing module is used to process the original data in various forms in a pattern-based manner according to business needs to form standardized subject data. The standard data domain module is used to store the standardized data assets formed.

[0009] The data intelligent development layer includes the application data domain module and the data service module, which is used to integrate the high-quality basic data generated by the data warehouse layer into data for specific scenarios, forming data services and data asset service-oriented processes. The application data domain module is used to intelligently develop the data assets in the data warehouse in three ways: data modeling, data development, and data calculation. The data service module is used to convert the data generated by data applications into security services for different types of data, and provide services for data assets as needed;

[0010] The data consumption layer is used for open sharing of data services and building a data resource directory for self-service consumption by consumers.

[0011] The support system includes:

[0012] Standard system, using existing national standards, enterprise standards, and industry standards to regulate the entire life cycle of data assets;

[0013] One-stop data operation, used to manage, monitor and support the operation and maintenance of the data center;

[0014] Tool support system, application tools for data middle platform.

[0015] The data production environment module includes data streams, databases, logs, business applications, web pages, files, and manual perception; the data access module provides global data access capabilities generated by the drilling data production environment, including data stream access, database collection, log collection, application data collection, web page data crawling, file data collection, and manual reporting. Data stream access can achieve real-time access to streaming data; database collection realizes the extraction and analysis of data stored in the database; log collection realizes incremental data collection based on database transaction logs; application data collection realizes data collection from external business systems or internal legacy business systems; web page data crawling realizes the collection of data in web pages; file data collection realizes the collection of unstructured data recorded in text; manual reporting realizes the collection of manually reported types of data.

[0016] The data access module can realize the access of data in various storage forms, multiple storage locations, and multiple structures at the source to the data middle platform.

[0017] The data processing module completes the extraction of data assets through cleaning and transformation, quality verification, consistency comparison, and data labeling operations, and carries out the process of data assetization.

[0018] The data modeling constructs a subject data model according to the different needs of each business theme; the data development includes machine learning and data mining. The machine learning uses statistical technology to find out the rules in the historical data of a business and learn them, and then applies them to new data for prediction. The data mining extracts information that cannot be directly obtained from massive data, integrates the information through various data sources, and explores its internal relationships. The data modeling provides offline computing engine, real-time computing engine, and database query engine services according to the different needs of each business theme.

[0019] The data services include data release management, security management, encryption and decryption control, and dashboard management;

[0020] The data publishing management publishes data resources into a standard REST interface for data users to call, and provides complete interface authority management;

[0021] The security management can control IP address access rights in the form of a whitelist in the system configuration function, further enhancing the security of the system;

[0022] The encryption and decryption control allows users to encrypt and decrypt returned data, filter columns and interface parameters in the form of custom classes, ensuring the security of interface use and returned data and avoiding leakage of sensitive information;

[0023] The dashboard management provides an overview of the data service system through the dashboard, including data on "number of data sources", "number of interfaces" and "number of users".

[0024] The standard system includes platform data standards, data collection standards, data application standards, data management standards, data security management standards, master data and management standards, and metadata and management standards;

[0025] The platform data standards include the scope and specifications of data entering the data governance application platform, including the core data scope, shared data scope, and collected data scope, including data entities, entity attributes, associated attributes, attribute value ranges, and attribute encoding methods;

[0026] The data collection standards, data governance application platform sharing data methods and data content, specifically including the data collection interface technology form, collection protocol, collection cycle or trigger mechanism, the collection data content including data scope, collection data model, data carrier form and encoding method;

[0027] The data application standards, data sharing technology methods, interaction processes, data service primitives, data service parameters, return result structure and encoding methods provided by the data governance application platform. If it is a file method, it is necessary to define the file content structure, encoding method, file naming format, and data ready time agreement;

[0028] The data management standard, the data lifecycle management standard of the data governance application platform, defines management roles, management processes and form contents;

[0029] The data security management standards, data security, and privacy protection agreements;

[0030] The master data and management standards, master data scope, data model, and management principles;

[0031] The metadata and management standards, metadata scope, data model, and management principles.

[0032] The one-stop data operation platform includes the following aspects: business terminology management, data standard management, data asset management, data quality management, life cycle management, data security management, master data management, and data sharing management.

[0033] The beneficial effects of the present invention are: through the construction of the data middle platform, the construction of the drilling data asset management system is realized, the unified data collection and standardized storage are formed, and a unified data sharing service and a unified application architecture system are provided, which effectively solves the problem of information islands, promotes the closed-loop management of business data generation and data service business. The use of advanced big data technology can transform massive data into high-quality data assets, promote the transformation of informatization from control to empowerment, provide deeper business insights for the drilling industry, and help the high-quality development of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural diagram of the data asset management system based on the drilling full business data middle platform of the present invention. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0036] like Figure 1As shown, the data asset management system based on the drilling full business data middle platform of the present invention includes a four-horizontal and three-vertical framework division, wherein the first of the four horizontals includes the data production environment and data access; the second of the four horizontals is the data warehouse including the original data domain, data processing and standard data domain; the third of the four horizontals is the data intelligent development layer including the application data domain and data asset service; the last horizontal is data consumption. The three verticals include the standard system, one-stop data operation and tool support system.

[0037] Example:

[0038] The data asset management system based on the drilling full business data center includes a four-horizontal and three-vertical framework division. The four horizontals represent the process of data assetization, and the three verticals refer to the support system for the construction of the drilling data center.

[0039] The first of the four horizontals includes the data production environment and data access

[0040] Data production environment refers to the perception of all objective data sources and types involved in existing drilling business

[0041] Data access is used to access data from different sources, in different forms, and with different protocols generated in the data production environment, allowing the data to enter the data warehouse.

[0042] The second of the four horizontal lines is the data warehouse, which includes three parts: original data domain, data processing and standard data domain.

[0043] The original data domain is used to store the original data

[0044] Data processing is used to pattern-process raw data in various forms according to business needs to form standardized subject data.

[0045] The standard data domain is used to store the data assets of the normative standards.

[0046] The third of the four horizontal layers is the data intelligence development layer, which includes the application data domain and data services. This layer integrates the high-quality basic data formed above into data for specific scenarios to form data services and the process of data asset service.

[0047] The application data domain is used for intelligent development of data assets in the data warehouse in three ways: data modeling, data development, and data computing.

[0048] Data services transform the data generated by data applications into security services for different types of data, and provide services for data assets as needed.

[0049] The last horizontal line represents data consumption, which is the open sharing of data services and the construction of a data resource directory for self-service consumption by consumers.

[0050] The three verticals include standard system, one-stop data operation and tool support system

[0051] The standard system uses existing national standards, enterprise standards, and industry standards to regulate the entire life cycle of data assets.

[0052] One-stop data operations are used to provide management, monitoring and support for the operation and maintenance of the data center.

[0053] Tool support system, application tools for data middle platform.

[0054] The data production environment refers to all objective data sources and types involved in the existing drilling business, including data streams, databases, logs, interfaces, business applications, web pages, files, and manual perception.

[0055] The data access provides global data access capabilities generated by the drilling data production environment, including data stream access, database collection, log collection, application data collection, web crawling, file transfer and manual reporting. Data stream collection can realize real-time access to streaming data; database inventory data extraction can realize the collection of data stored in the database; transaction log collection can realize incremental data collection based on database transaction logs; application data crawling can realize data collection from external business systems or internal legacy business systems; web data crawling can realize data collection from web pages; file data collection can realize data collection recorded in text; user behavior data collection can realize data collection of manually reported types.

[0056] In order to access data in various forms, it is necessary to provide real-time data transmission, batch data transmission and file transmission capabilities to meet the needs of data applications with different timeliness and forms. Through the data access module, it is possible to access data in various storage forms, storage locations and structures at the source to the data center.

[0057] The data warehouse provides storage capabilities for data of various types and forms, including three parts: original data domain, data processing and standard data domain.

[0058] The original data domain is used to store the original form data

[0059] The data processing is used to process the original data in various forms into standardized subject data according to business needs. This process requires operations such as cleaning and transformation, quality verification, consistency comparison, and data labeling to complete the extraction of data assets, that is, the process of data assetization.

[0060] The standard data domain is used to store data assets that have been processed and integrated to form standardized standards.

[0061] The data intelligence development layer includes two parts: application data domain and data service. This layer integrates the high-quality basic data formed above into data for specific scenarios to form data services and data asset service process.

[0062] The application data domain is used to perform intelligent development of data assets in the data warehouse in three ways: data modeling, data development, and data calculation.

[0063] The data modeling can construct a subject data model according to the different requirements of each business subject.

[0064] The data development includes machine learning and data mining

[0065] The machine learning method uses statistical techniques to find patterns in historical business data and then applies them to new data for prediction.

[0066] Data mining is to extract information that cannot be directly obtained from massive data, integrate information through various data sources, and discover its internal relationships.

[0067] The data modeling can provide offline computing engine, real-time computing engine, and database query engine services according to the different needs of various business topics.

[0068] The data service converts the data generated by data applications into security services for different types of data, and provides services for data assets according to business needs, including data release management, security management, encryption and decryption control, and dashboard management.

[0069] The data publishing management can publish data resources into a standard REST interface for data users to call, and provide complete interface permission management. Set the interface type: fully public, application required, private. For fully public interfaces, users can call directly without the need for administrator approval; for interfaces that require application, users must first submit an application for use on the client, and can only use it normally after the administrator approves it on the management side; for private interfaces, users cannot view them. Set the interface request type: all, query, add, modify, delete According to the interface request operation type set by the administrator, the caller can perform the corresponding request operation. Set whether the data resource item is hidden: For a certain item in the data resource, the administrator can set it to be public or hidden. When the administrator sets a data resource item to be hidden, the data resource obtained by the data caller will not be visible to the information of this item. On the contrary, the data caller can obtain it normally and see it. Set query conditions: The administrator can return only the data resources that meet the query conditions, rather than all the data, by customizing the where query conditions to meet the data security control requirements.

[0070] The security management can control the IP address access rights in the form of a whitelist in the system configuration function, further enhancing the security of the system. The access address control function provides two-level settings: the global setting is that after the user enables the whitelist function, the IP address allowed to be accessed can be added in the global setting. The IP address in the whitelist can smoothly access the client application end to perform interface calls, maintenance and other operations. The whitelist setting of the user list After selecting a user in the user list, the system administrator can add a whitelist that allows the user to access on the right. In subsequent use, when the user tries to log in to the system, his IP address must be within the whitelist range of the user list to successfully log in, otherwise the login fails. User-level data permission control supports setting different data permission controls according to user levels, that is, for the same interface, the administrator can set different field columns to be returned according to different applicants; it can also be set through the where query condition to only return data resources that meet the query conditions, so as to achieve user-level data permission control and ensure data isolation and security. The specific process is as follows: when the interface user applies for the interface, he must first apply for the columns to be obtained. When the administrator approves, the specific information of the user's application can be viewed through the detailed information page, including the return columns of the application. When the administrator views the detailed information of the user's application, he can set the specific return columns according to the application user's situation. At the same time, he can also use the where query condition to set only the data resources that meet the query condition to be returned, so as to achieve user-level data permission control and ensure data isolation and security.

[0071] The encryption and decryption control allows users to encrypt and decrypt returned data, filter columns and interface parameters in the form of custom classes, ensuring the security of interface use and returned data and avoiding leakage of sensitive information.

[0072] The dashboard management provides an overview of the data service system through the dashboard, including data such as "number of data sources", "number of interfaces", and "number of users". Users can quickly access the data source, interface list, and user list through the dashboard to perform maintenance operations such as adding, modifying, and deleting corresponding data. In addition, the dashboard also provides analytical data such as "interface access trend chart", "user access TOP", and "interface access TOP", and allows users to customize time periods for statistical analysis of linkage data. The interface access trend chart can help users view the number of times the system interface has been accessed within a specific time period. At the same time, users can also view the TOP5 most visited users and the TOP5 most called interfaces within a specific time period, so that users can quickly grasp management information.

[0073] The data consumption described above refers to the open sharing of data services and the construction of a data resource directory for self-service consumption by consumers.

[0074] The data directory is oriented to scenarios such as open sharing of data resources, data exchange, and data analysis for various business parties. It provides a platform for sharing data between data providers and data demanders, and provides a unified entry for registration, discovery, query, positioning, and acquisition of data resources to see the full picture of the data.

[0075] The data directory includes directory services, browsing retrieval, management monitoring and service interfaces

[0076] The directory service provides a unified metadata management function for all data resources. At the same time, the module provides full-process management functions such as registration, publishing, review, application and subscription of data resources to meet the daily management needs of data resources.

[0077] The browsing retrieval provides global retrieval of data resources and data map browsing functions.

[0078] The management monitoring provides directory management and statistical functions.

[0079] The service interface system provides a wide variety of third-party service interfaces, including management integration service interfaces, download service interfaces, etc. Through the above service interfaces, users can integrate and perform secondary development of products in third-party systems to meet the functional requirements of users in different business scenarios.

[0080] The support system includes a standard system, a one-stop data operation and a tool support system

[0081] The standard system uses existing national standards, enterprise standards, and industry standards to regulate the entire life cycle of data assets, and can also ensure the rigor of the data asset circulation process within the entire system. It includes platform data standards, data collection standards, data application standards, data management standards, data security management standards, master data and management standards, and metadata and management standards.

[0082] The platform data standard defines the scope and specifications of data entering the data governance application platform, including the core data scope, shared data scope, and collected data scope. It includes data entities, entity attributes, associated attributes, attribute value ranges, attribute encoding methods, etc.

[0083] The data collection standard defines the data sharing method and data content of the data governance application platform, including the data collection interface technology form, collection protocol, collection cycle or trigger mechanism, etc. The collected data content includes data scope, collection data model, data carrier form and encoding method, etc.

[0084] The data application standard defines the data sharing technology, interaction process, data service primitives, data service parameters, return result structure and encoding method provided by the data governance application platform. If it is a file method, it is necessary to define the file content structure, encoding method, file naming format, data ready time and other agreements.

[0085] The data management standard defines the data lifecycle management standard of the data governance application platform, defines management roles, management processes, form content, etc.

[0086] The data security management standard defines the agreements on data security and privacy protection.

[0087] The master data and management standards define the master data scope, data model, management principles, etc.

[0088] The metadata and management standards define metadata scope, data model, management principles, etc.

[0089] The one-stop data operation is used to provide management, monitoring and support for the operation and maintenance of the data middle platform. The operation of organizational data assets is inseparable from a rigorous operation system. It adheres to the system principles of "safe and controllable, full coverage, full life cycle, and full process closed loop", and is guaranteed by the production organization system to achieve sustainable, high-quality, and valuable operations. In order to ensure the specific implementation and implementation of the above strategies, it is an indispensable and important link to establish a management and control system that focuses on the highest dimension of the overall situation, including the data supply process of departments and organizations at all levels, the development process of central data assets, and the data consumption process based on digital operations + digital innovation. All need to be included in a scientific management framework.

[0090] The one-stop data operation platform should provide a multi-role and multi-level authority management mechanism that fits the characteristics of the data management business to meet the participation of multiple types of personnel in the data business, thereby realizing the construction of a data governance application platform system at the organizational level, ensuring that data can guide marketing, decision-making and innovation behaviors on the demand side, and can also guide production on the supply side, optimize the supply chain, realize intelligent operation and control, and optimize organizational resource allocation and risk warning.

[0091] The one-stop data operation platform should include the following aspects: business terminology management, data standard management, data asset management, data quality management, lifecycle management, data security management, master data management, and data sharing management.

[0092] In summary, the content of the present invention is not limited to the above-mentioned embodiments, and people with insight in the same field can easily propose other embodiments within the technical guiding ideology of the present invention, but such embodiments are all included in the scope of the present invention.

Claims

1. A data asset management system based on the drilling full business data center, characterized in that: It includes a data assetization system and a support system. The data assetization system is a management platform that converts all objectively existing data involved in the existing drilling business into data assets and opens them up for sharing. The support system provides data standards, data management, monitoring and application tools for the data middle platform. The data assetization system includes: The acquisition layer includes a data production environment module and a data access module. The data production environment module is used to perceive all objective data sources and types involved in the existing drilling business. The data access module is used to access data from different sources, different forms, and different protocols generated in the data production environment and allow the data to enter the data warehouse. The data warehouse layer includes the original data domain module, the data processing module and the standard data domain module. The original data domain module is used to store the original data. The data processing module is used to process the original data in various forms in a pattern-based manner according to business needs to form standardized subject data. The standard data domain module is used to store the standardized data assets formed. The data intelligent development layer includes the application data domain module and the data service module, which is used to integrate the high-quality basic data generated by the data warehouse layer into data for specific scenarios, forming data services and data asset service-oriented processes. The application data domain module is used to intelligently develop the data assets in the data warehouse in three ways: data modeling, data development, and data calculation. The data service module is used to convert the data generated by data applications into security services for different types of data, and provide services for data assets as needed; The data consumption layer is used for open sharing of data services and building a data resource directory for self-service consumption by consumers. The support system includes: Standard system, using existing national standards, enterprise standards, and industry standards to regulate the entire life cycle of data assets; One-stop data operation, used to manage, monitor and support the operation and maintenance of the data center; Tool support system, application tools for data middle platform.

2. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The data production environment module includes data streams, databases, logs, business applications, web pages, files, and manual perception; the data access module provides global data access capabilities generated by the drilling data production environment, including data stream access, database collection, log collection, application data collection, web page data crawling, file data collection, and manual reporting. Data stream access can achieve real-time access to streaming data; database collection realizes the extraction and analysis of data stored in the database; log collection realizes incremental data collection based on database transaction logs; application data collection realizes data collection from external business systems or internal legacy business systems; web page data crawling realizes the collection of data in web pages; file data collection realizes the collection of unstructured data recorded in text; manual reporting realizes the collection of manually reported types of data.

3. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The data access module can realize the access of data in various storage forms, multiple storage locations, and multiple structures at the source to the data middle platform.

4. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The data processing module completes the extraction of data assets through cleaning and transformation, quality verification, consistency comparison, and data labeling operations, and carries out the process of data assetization.

5. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The data modeling constructs a subject data model according to the different needs of each business theme; the data development includes machine learning and data mining. The machine learning uses statistical technology to find out the rules in the historical data of a business and learn them, and then applies them to new data for prediction. The data mining extracts information that cannot be directly obtained from massive data, integrates the information through various data sources, and explores its internal relationships. The data modeling provides offline computing engine, real-time computing engine, and database query engine services according to the different needs of each business theme.

6. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The data services include data release management, security management, encryption and decryption control, and dashboard management; The data publishing management publishes data resources into a standard REST interface for data users to call, and provides complete interface authority management; The security management can control IP address access rights in the form of a whitelist in the system configuration function, further enhancing the security of the system; The encryption and decryption control allows users to encrypt and decrypt returned data, filter columns and interface parameters in the form of custom classes, ensuring the security of interface use and returned data and avoiding leakage of sensitive information; The dashboard management provides an overview of the data service system through the dashboard, including data on "number of data sources", "number of interfaces", and "number of users".

7. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The standard system includes platform data standards, data collection standards, data application standards, data management standards, data security management standards, master data and management standards, and metadata and management standards; The platform data standards include the scope and specifications of data entering the data governance application platform, including the core data scope, shared data scope, and collected data scope, including data entities, entity attributes, associated attributes, attribute value ranges, and attribute encoding methods; The data collection standards, data governance application platform sharing data methods and data content, specifically including the data collection interface technology form, collection protocol, collection cycle or trigger mechanism, the collection data content including data scope, collection data model, data carrier form and encoding method; The data application standards, data sharing technology methods, interaction processes, data service primitives, data service parameters, return result structure and encoding methods provided by the data governance application platform. If it is a file method, it is necessary to define the file content structure, encoding method, file naming format, and data ready time agreement; The data management standard, the data lifecycle management standard of the data governance application platform, defines management roles, management processes and form contents; The data security management standards, data security, and privacy protection agreements; The master data and management standards, master data scope, data model, and management principles; The metadata and management standards, metadata scope, data model, and management principles.

8. According to claim 1, the data asset management system based on the drilling full business data middle platform is characterized in that: The one-stop data operation platform includes the following aspects: business terminology management, data standard management, data asset management, data quality management, life cycle management, data security management, master data management, and data sharing management.

Images