Intelligent data system based on cloud platform and data medium station

Abstract

The invention discloses an intelligent data system based on a cloud platform and a data medium station, and relates to the technical field of information. Comprising a cloud platform, a data source, data aggregation, data governance, data organization, data application, data security and data mining and innovation. The system aims at achieving comprehensive collection, efficient storage, intelligent analysis, flexible management and safe application of data through highly integrated data processing and analysis capacity, and providing accurate data support for various decisions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and particularly relates to an intelligent data system based on a cloud platform and a data middle platform. BACKGROUND

[0002] With the advent of the big data era, data has become an important strategic resource. It has important significance for national security, military strategy, commercial competition and other fields. Data systems have made significant progress in informatization, cross-departmental cooperation and technology application, but there are still problems such as insufficient information collection, limited data analysis capability, insufficient data sharing and utilization, and challenges such as massive, heterogeneous data sources, complex data processing requirements, and data security and compliance. The traditional data processing mode is difficult to meet the needs of modern analysis. Therefore, building an intelligent data system based on a cloud platform and a data middle platform has become the key to solving the above problems. SUMMARY

[0003] The present application aims to provide an intelligent data system based on a cloud platform and a data middle platform, which aims to realize comprehensive data collection, efficient storage, intelligent analysis, flexible management and safe application through highly integrated data processing and analysis capabilities, and provide accurate data support for various decisions.

[0004] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0005] The present application provides an intelligent data system based on a cloud platform and a data middle platform, which includes a cloud platform, a data source, data aggregation, data governance, data organization, data application, data security and data mining and innovation, wherein:

[0006] The cloud platform includes network resource management, computing resource management, storage resource management, cloud platform management, and big data platform.

[0007] The data source includes databases, documents, web pages, pictures, audio, and video, which realizes the acquisition of various data sources.

[0008] The data aggregation includes access management, policy management, resource management, and job management.

[0009] The data governance includes metadata management, metadata modeling, data quality management, data operation and maintenance, and data lifecycle management.

[0010] The data organization includes data organization cataloging, data resource directory, and global data retrieval.

[0011] The data application includes data customization, intelligence, asset management service platform, and data permission management.

[0012] Optionally, the access management implements multiple forms of data type connection capabilities, and provides but is not limited to database synchronization capabilities;

[0013] The policy management provides integrated and fusable data adaptation analyzers and data conversion functions according to data processing steps such as extraction, synchronization and integration, has automatic data source heterogeneous data analysis capabilities, and is configured according to business scenarios to realize intelligent, visual and component data aggregation and integration task construction, complete adaptive data analysis and process handling.

[0014] The resource management realizes maintenance, update, deletion, query and display functions of various resources such as database resources.

[0015] The task management realizes a multi-scene task scheduling mechanism, and supports multi-angle task monitoring functions for data access tasks.

[0016] Optionally, the metadata modeling is to visually construct a data warehouse model and standardize the model.

[0017] The data operation and maintenance is to display the overall data situation through visualization or report forms.

[0018] Optionally, the data organization and cataloging is to classify and organize data according to different processing levels and purposes, ensure the structuring, ordering and efficient use of data, and divide data into original layer, detail layer, summary layer and application layer.

[0019] The data resource directory realizes system analysis, classification and cataloging of various data resources in the organization, provides clear, accurate and comprehensive data resource views to facilitate users to find, understand and use data.

[0020] The global data provides various types of data retrieval functions.

[0021] Optionally, the network resource management realizes management of network resources, the computing resource management realizes management of computing resources, the storage resource management realizes management of storage resources, the cloud platform management realizes resource system on-demand expansion, hot upgrade, physical topology scheduling, cloud disk, encryption, load balancing and tenant isolation functions, and the big data platform realizes distributed resource and task scheduling, big data computing, database deployment management, database upgrade, large-scale data processing and high-performance offline analysis framework functions of massive data.

[0022] Based on the above technical solutions, the application can obtain the following technical effects:

[0023] By integrating the efficient computing power of the cloud platform, the unified management advantage of the data center, and the advanced data processing technology, a comprehensive, intelligent, and secure data system is constructed. This system not only greatly improves the efficiency and quality of data processing, but also promotes the deep development and wide application of data resources, providing strong data support for government decision-making, enterprise operation, scientific research exploration, and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the overall architecture diagram provided by an embodiment of the present application;

[0025] Figure 2 is the workflow diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below in combination with the drawings and specific embodiments. The advantages and features of the present application will be more apparent according to the following description and claims. It should be noted that the drawings are greatly simplified and are not of precise scale, and are only used to facilitate and clarify the purpose of illustrating the embodiments of the present application.

[0027] It should be noted that in order to clearly illustrate the content of the present application, the present application specifically raises multiple embodiments to further explain different implementation manners of the present application, wherein the multiple embodiments are enumerative rather than exhaustive. In addition, in order to illustrate simply, the content mentioned in the previous embodiments is often omitted in the later embodiments, therefore, the content not mentioned in the later embodiments can be correspondingly referred to the previous embodiments.

[0028] Embodiment 1

[0029] As Figure 1 shown is the overall architecture diagram of the intelligent data system based on the cloud platform and the data center provided by the present embodiment. It includes a cloud platform 1, a data source 2, a data aggregation 3, a data governance 4, a data organization 5, a data application 6, a data security 7, and a data mining and innovation 8.

[0030] The cloud platform 1 includes network resource management 11, computing resource management 12, storage resource management 13, cloud platform management 14, and big data platform 15. The network resource management 11 realizes the management of network resources, the computing resource management 12 realizes the management of computing resources, the storage resource management 13 realizes the management of storage resources, the cloud platform management 14 realizes the functions of on-demand capacity expansion resource system, hot upgrade, physical topology scheduling, cloud disk, encryption, load balancing, tenant isolation, etc. The big data platform 15 realizes the functions of distributed resource and task scheduling, big data computing, database deployment management, database upgrade, large-scale data processing, high-performance offline analysis framework of massive data, etc.

[0031] The data source 2 includes databases 21, documents 22, web pages 23, images 24, audio 25, and video 26. It enables the acquisition of various data sources.

[0032] The data aggregation 3 includes access management 31, policy management 32, resource management 33, and job management 34. Access management 31 enables the reception of various data types, providing, but not limited to, database synchronization capabilities. Policy management 32, tailored to the characteristics of different data types and following data processing steps such as extraction, synchronization, and integration, provides an integrated, fusion-compatible data adapter and data transformation function. It possesses automatic parsing capabilities based on heterogeneous data sources and allows for customized configuration according to business scenarios, enabling intelligent, visual, and component-based data aggregation and integration task construction, completing adaptive data parsing and process-oriented processing. Resource management 33 enables functions such as maintenance, updating, deletion, querying, and displaying various resources, including database resources. Job management 34 implements a multi-scenario task scheduling mechanism and supports multi-faceted task monitoring for data access tasks.

[0033] The data governance 4 includes metadata management 41, metadata modeling 42, data quality management 43, data operation and maintenance 44, and data lifecycle management 45. Metadata modeling 42 involves visually constructing a data warehouse model and standardizing and reducing the model. Data operation and maintenance 44 involves displaying the overall status of the governed data through visualization or reports.

[0034] The data organization 5 includes data organization and cataloging 51, data resource catalog 52, and global data retrieval 53. Data organization and cataloging 51 classifies and organizes data according to different processing levels and uses, ensuring the structured, orderly, and efficient use of data. Data is divided into raw, detailed, summary, and application layers. The data resource catalog 52 provides a tool for systematically sorting, classifying, and cataloging various data resources within the organization, offering a clear, accurate, and comprehensive view of data resources to facilitate user searching, understanding, and use of data. Global data retrieval 53 provides retrieval functions for various types of data.

[0035] The data application 6 includes data customization 61, intelligent BI 62, asset management service platform 63, and data access control 64.

[0036] like Figure 2 The diagram shown is a flowchart of the workflow provided in this embodiment, which specifically includes the following:

[0037] The infrastructure is provided by cloud platform 1, and the data is acquired by data source 2. The acquired data is then connected to the data middle platform, and through data access, aggregation, processing, governance, mining, organization and management, a data warehouse is formed, which provides a unified data interface service to the outside world to achieve resource sharing.

[0038] In summary, by integrating the high-efficiency computing power of the cloud platform, the unified management advantages of the data middleware, and advanced data processing technologies, a comprehensive, intelligent, and secure data system has been constructed. This system not only significantly improves the efficiency and quality of data processing but also promotes the in-depth development and widespread application of data resources, providing strong data support for multiple fields such as government decision-making, enterprise operations, and scientific research.

[0039] Example 2

[0040] This embodiment describes an intelligent data system based on a cloud platform and a data middleware. It mainly includes the following eight core modules:

[0041] 1. Infrastructure:

[0042] Build a cloud-based big data platform to achieve unified management and flexible allocation of network, computing, and storage resources.

[0043] By adopting a microservice architecture, the scalability, flexibility, and reliability of the system can be improved.

[0044] By leveraging containerization technologies (such as Docker) and Kubernetes orchestration tools, dynamic allocation and efficient utilization of resources can be achieved.

[0045] 2. Data source

[0046] Data is obtained from multiple sources, including the Internet, social media, and professional databases, using distributed web crawling technology.

[0047] By adopting cloud storage technology, it enables massive, high-speed, and low-cost data storage, while supporting hot and cold backup strategies to ensure data security and availability.

[0048] 3. Data aggregation

[0049] It aggregates and connects various metadata and raw data materials, including structured, semi-structured, and unstructured data such as documents, web pages, images, audio, and video, providing comprehensive and unified data access capabilities.

[0050] 4. Data Governance

[0051] Integrate ETL (Extract, Transform, Load) tools to clean, transform, and load data, thereby improving data quality.

[0052] Machine learning algorithms, such as cluster analysis and association rule mining, are used to perform in-depth analysis of data and reveal hidden relationships and patterns between data.

[0053] Through metadata management, metadata modeling, data quality control, and data operation and maintenance, we achieve full lifecycle management of data.

[0054] Design a unified data model and metadata standard to ensure data standardization and consistency.

[0055] Implement data lifecycle management, including data creation, storage, use, archiving, and destruction, to optimize the efficiency of data resource utilization.

[0056] 5. Data Organization

[0057] The system employs a hierarchical and layered data organization model to reduce data redundancy and improve data processing efficiency. It provides multi-dimensional heterogeneous data storage management capabilities based on hybrid storage, supporting the distributed storage of various data types.

[0058] 6. Data Application

[0059] It provides API interfaces and SDKs to support rapid data access and customized application development.

[0060] Build a data visualization platform to intuitively display data analysis results in the form of charts, dashboards, etc., to assist in decision-making.

[0061] Establish a data resource catalog to enable data classification, labeling, and retrieval, thereby improving the efficiency of data discovery and utilization.

[0062] It supports data access control to ensure secure data sharing among different users and departments.

[0063] 7. Data security

[0064] Implement security measures such as data encryption, access control, and audit logs to protect data from unauthorized access and leakage.

[0065] Establish a data governance framework, including data quality monitoring, compliance review, and data privacy protection, to ensure the legal and compliant use of data.

[0066] 8. Data Mining and Innovation

[0067] By combining advanced technologies such as deep learning and natural language processing, we can conduct data mining and innovation research to uncover new value points in data.

[0068] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. An intelligent data system based on a cloud platform and a data middleware, characterized in that, This includes cloud platforms, data sources, data aggregation, data governance, data organization, data application, data security, and data mining and innovation, among which: The cloud platform includes network resource management, computing resource management, storage resource management, cloud platform management, and big data platform; The data sources include databases, documents, web pages, images, audio, and video, enabling the acquisition of various data sources; The data aggregation includes access management, policy management, resource management, and job management; The data governance includes metadata management, metadata modeling, data quality management, data operation and maintenance, and data lifecycle management; The data organization includes data organization cataloging, data resource catalog, and global data retrieval; The data applications include data customization, intelligence, asset management service platforms, and data access control.

2. The system according to claim 1, characterized in that, Access management enables the reception of various data types and provides, but is not limited to, database synchronization capabilities. The strategy management system provides an integrated and fusionable data adaptation parser and data transformation function based on the characteristics of different data types and data processing steps such as extraction, synchronization, and integration. It has the ability to automatically parse heterogeneous data based on data sources and can be customized according to business scenarios to realize intelligent, visual, and component-based data aggregation and integration task construction, and complete adaptive data parsing and process-oriented processing. Resource management enables the maintenance, updating, deletion, querying, and display of various resources, including database resources. The task management system implements a multi-scenario task scheduling mechanism and supports multi-angle task monitoring for data access tasks.

3. The system according to claim 1, characterized in that, Metadata modeling involves visually constructing a data warehouse model and then standardizing and constraining that model. Data operations and maintenance involves presenting the overall status of the managed data through visualization or reporting methods.

4. The system according to claim 1, characterized in that, Data organization and cataloging involves classifying and organizing data according to different processing levels and uses to ensure that the data is structured, orderly, and used efficiently; the data is divided into the raw layer, detailed layer, summary layer, and application layer. The data resource catalog is a tool for systematically sorting, classifying, and cataloging various data resources within an organization, providing a clear, accurate, and comprehensive view of data resources to facilitate users in finding, understanding, and using data; The full-domain data provides search functions for various types of data.

5. The system according to claim 1, characterized in that, Network resource management enables the management of network resources, computing resource management enables the management of computing resources, storage resource management enables the management of storage resources, and cloud platform management enables on-demand expansion of resource systems, hot upgrades, physical topology scheduling, cloud disks, encryption, load balancing, and tenant isolation functions; the big data platform enables distributed resource and task scheduling, big data computing, database deployment management, database upgrades, large-scale data processing, and a high-performance offline analysis framework for massive amounts of data.

Images