A private cloud data transmission method and data management platform

Abstract

The application discloses a private cloud data transmission method and a data management platform, and relates to the technical field of big data processing.The method mainly comprises the following steps: collecting heterogeneous data by using flume components, sqoop components and datax components, and uniformly transmitting the heterogeneous data to kafka components through canal components, while storing the heterogeneous data in hive components and hbase components respectively; performing offline calculation on the data stored in the hive components by using MapReduce components and Spark components; adding secondary indexes to the data stored in the hbase components by using phoenix components, and performing real-time calculation on the data by using Spark components and flink components; and providing consistent access interfaces for the hive components and the hbase components by using DBapi components, so as to realize the output of data.In the scheme, offline data is uniformly stored in the hive components, real-time data is transmitted to the hbase components in real time after being written ETL by the flink components or the Spark components, and the hbase components use the phoenix components as client to connect the DBapi components, so that the unified output storage of multi-source data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and particularly relates to a private cloud data transmission method and a data management platform. BACKGROUND

[0002] The relevant laws and regulations of the government for operating a cloud are not very perfect at present, and group enterprises generally have obvious competitors and important data that need to be kept secret. If a public cloud is used, it is difficult to investigate the responsibility when data problems occur. Meanwhile, the charging standards of cloud operators are not unified, and the traffic charging is different. Sometimes, enterprises need independent dedicated lines, and there is no standard for who to bear the cost. At present, there are not many operators that can provide cloud services for group enterprises, and they are seriously isolated from each other. Once the cloud service provider is changed, data migration is a big problem (data standards, and the enterprise data volume is generally large).

[0003] The deployment cycle of the traditional private cloud component service is long, the data service layer for outputting data to the outside involves complex data structures and multiple source databases, and more developers are needed. At the same time, the developers are required to have rich skills. Therefore, how to realize data output to meet various offline computing, real-time computing and complex data analysis scenarios requires a multifunctional data service layer to realize the unification of multi-source data. The data service layer structure of the data lake function developed by large companies is complex, involves many technologies, and consumes a lot of resources for deployment. There are few open sources, and it is difficult to develop again to meet the needs of enterprises. SUMMARY

[0004] The private cloud data transmission method provided by the present application aims to solve the problem that the private cloud cannot realize unified output storage of multi-source data and long offline data output time in the prior art.

[0005] To achieve the above-mentioned purpose, the following technical solutions are adopted in the present application:

[0006] The private cloud data transmission method of the present application comprises:

[0007] Heterogeneous data is collected by using flume components, sqoop components and datax components, and the heterogeneous data is uniformly transmitted to a kafka component through a canal component;

[0008] The kafka component receives the heterogeneous data, and stores the heterogeneous data in a hive component and a hbase component according to data types, wherein the data types include offline data and real-time data;

[0009] MapReduce components and Spark components are used to perform offline computing on the data stored in the hive component;

[0010] The phoenix component is used to add a secondary index for the data stored in the hbase component, and the Spark component and the flink component are used for real-time calculation of the data;

[0011] The DBapi component is used to provide a consistent access interface for the hive component and the hbase component to realize the output of data.

[0012] Preferably, the DBapi component provides a consistent access interface for the hive component and the hbase component to realize the output of data, which further includes connecting the phoenix component through the DBapi component to write SQL statements to realize real-time output of data in the hbase component.

[0013] Preferably, the DBapi component provides a consistent access interface for the hive component and the hbase component to realize the output of data, which further includes using a java springboot framework, encapsulating a jdbc driver, parsing json parameters, splicing dynamic SQL, using SQL language development, connecting a database and writing a data interface.

[0014] Preferably, the DBapi component provides a consistent access interface for the hive component and the hbase component to realize the output of data, which further includes using token verification to perform black and white list verification on an ip accessing the hive component or the hbase component.

[0015] Preferably, the heterogeneous data includes log data, relational data and buried point data.

[0016] A private cloud data management platform, comprising:

[0017] A data platform monitoring system for collecting heterogeneous data and classifying and processing the heterogeneous data;

[0018] A data service module for providing a unified access interface for data processed by the platform monitoring system to realize the output of data, and verifying an ip accessing the interface;

[0019] A data application module for performing personalized application processing on data output by the data service module.

[0020] Preferably, the data platform monitoring system comprises:

[0021] A data collection module for collecting heterogeneous data using a flume component, a sqoop component and a datax component;

[0022] The data import module is used to transfer the heterogeneous data collected by the data acquisition module to the Kafka component using the Canal component;

[0023] The data storage module is used by the Kafka component to receive the heterogeneous data and store the heterogeneous data in the Hive component and the HBase component respectively according to the data type, wherein the data type includes offline data and real-time data;

[0024] The offline processing module is used to perform offline computations on data stored in the Hive component using the MapReduce and Spark components.

[0025] The real-time processing module is used to add secondary indexes to data stored in the HBase component using the Phoenix component, and to perform real-time computation on the data using the Spark and Flink components.

[0026] As a preferred option, a user profile library is used to create user profiles based on user behavior;

[0027] The real-time dashboard module is used to display the data processing process in real time.

[0028] The reporting submodule is used to compile various data into reports for data analysis.

[0029] The monitoring and early warning submodule is used to monitor the operating status of the management platform and issue an alarm before a fault occurs;

[0030] Other data application submodules are used to perform other personalized processing on the data.

[0031] Preferably, the management platform architecture consists of Apache DolphinScheduler, CDH, and DBapi components.

[0032] Preferably, the management platform also includes Flink, Phoenix, Sqoop, Kafka, and DataX components.

[0033] The present invention has the following beneficial effects:

[0034] In this solution, database data is monitored by the Flink component, and parsing scripts are written to synchronize CRUD operations to the HBase component in real time. Log data, event tracking data, and other data are transmitted to the HBase component in real time after ETL is written using the Flink or Spark component. The HBase component uses the Phoenix component as a client to connect to the DBapi component, thus achieving unified output and storage of multi-source data. At the same time, the Kafka component, DataX component, and DBapi component are used to produce offline data at the minute or hour level, and rapid analysis is achieved by writing SQL interfaces. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a first flowchart of Embodiment 1 of this application implementing a private cloud data transmission method;

[0037] Figure 2 This is a flowchart illustrating a private cloud data transmission method as described in Embodiment 2 of this application.

[0038] Figure 3 This is a block diagram of a private cloud data management platform implemented in Embodiment 3 of this application;

[0039] Figure 4 This is a schematic diagram of the structure of a private cloud data management platform implemented in Embodiment 4 of this application. Detailed Implementation

[0040] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] The terms "first," "second," etc., used in the claims and description of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate. This is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] Example 1

[0043] like Figure 1 As shown, a private cloud data transmission method includes the following steps:

[0044] S110. Collect heterogeneous data using Flume, Sqoop, and DataX components, and transmit the heterogeneous data to Kafka component through Canal component.

[0045] S120, the Kafka component receives the heterogeneous data and stores the heterogeneous data in the Hive component and the HBase component respectively according to the data type, the data type including offline data and real-time data;

[0046] S130. Use the MapReduce and Spark components to perform offline computation on the data stored in the Hive component.

[0047] S140. Use the Phoenix component to add a secondary index to the data stored in the HBase component, and use the Spark and Flink components to perform real-time computation on the data.

[0048] S150. Use the DBapi component to provide a consistent access interface for the hive component and the hbase component to achieve data output.

[0049] In Example 1, firstly, heterogeneous data such as external log data, relational data, event tracking data, and other data are collected using Flume, DataX, and Sqoop components to synchronize metadata. Then, the synchronized data is uniformly imported into the Kafka messaging system using the Canal component for easy data transfer. Next, these heterogeneous data are stored separately according to whether they are offline or real-time data. Offline data is stored in the Hive component, and real-time data is stored in the HBase component. At the same time, the HBase component uses the Phoenix client to implement secondary indexes to improve query efficiency. Then, the MapReduce and Spark components are used to perform offline calculations and analysis on the data stored in the Hive component, and the Spark and Flink components are used to perform real-time calculations on the data stored in the HBase component. Finally, the DBapi component is used to achieve real-time output of the data. In this application, during real-time computation, even with multi-source data, only the Flink component is needed to write the ETL process to implement CRUD operations for different sinks and special scenario development, ensuring no data loss or duplication. Simultaneously, data is stored in real-time in the HBase component, which uses the Phoenix component to improve query efficiency. Finally, the data service DBapi component connects to Phoenix to write SQL statements, truly achieving real-time data output. Furthermore, ordinary offline data is implemented on a T+1 basis, meaning only data from the previous day is currently available, and daily runs are scheduled on a daily basis. This application, using the Kafka component + DataX component + DBapi component, enables offline data to be output at the minute or hour level, and rapid analysis is achieved through SQL interface writing.

[0050] Example 2

[0051] like Figure 2 As shown, a private cloud data transmission method includes the following steps:

[0052] S210. Collect heterogeneous data using Flume, Sqoop, and DataX components, and transmit the heterogeneous data to Kafka component through Canal component.

[0053] Log data is collected using Flume components, data migration between databases is performed using Sqoop and DataX components, and then Canal-Kafka components are used, that is, Kafka components are embedded into Canal components as clients to achieve the connection between Canal and Kafka components, so as to achieve massive message transmission and synchronization. In this application, all collected data is imported into the Kafka component.

[0054] The Sqoop component can import data from relational databases into Hadoop components such as HDFS, Hive, or HBase, and vice versa. Sqoop fully utilizes the MapReduce computing framework during data import and export, generating a MapReduce job based on input conditions and running it within the Hadoop cluster. Using the MapReduce framework to perform import or export operations simultaneously on multiple nodes is faster than running multiple parallel import / export operations on a single node, while also providing good concurrency and fault tolerance. DataX enables data exchange between heterogeneous databases and file systems, built using a Framework+plugin architecture. The Framework handles most of the technical challenges of high-speed data exchange, such as buffering, flow control, concurrency, and context loading, providing simple interfaces for plugin interaction. Plugins only need to access the data processing system, and the open framework allows developers to quickly create new plugins to support new databases / file systems.

[0055] S220 and the Kafka component receive the heterogeneous data and store the heterogeneous data in the Hive component and the HBase component respectively according to the data type, which includes offline data and real-time data;

[0056] By storing data in a unified database in real time, that is, storing real-time data in the HBase component and offline data in the Hive component, multi-source data statistical storage can be achieved.

[0057] The Hive component is a data warehouse tool based on Hadoop. It can map structured data files to a database table and provide complete SQL query functionality. It can convert SQL statements into MapReduce tasks for execution. Its advantages include low learning cost, the ability to quickly implement simple MapReduce statistics using SQL-like statements without developing dedicated MapReduce applications, making it very suitable for statistical analysis in data warehouses. At the same time, all Hive data is stored in Hadoop-compatible file systems (e.g., Amazon S3, HDFS). Hive does not modify the data during the data loading process; it only moves the data to the directory set by Hive in HDFS. Therefore, Hive does not support data modification or addition; all data is determined at the time of loading.

[0058] S230. Use the MapReduce and Spark components to perform offline computation on the data stored in the Hive component.

[0059] The MapReduce component is a parallel computing and execution software framework. It provides a large but well-designed parallel computing framework that automatically parallelizes computational tasks, automatically partitions computational data and tasks, automatically allocates and executes tasks on cluster nodes, and collects computation results. It handles many of the complex underlying details of parallel computing, such as data distribution and storage, data communication, and fault tolerance, making it particularly suitable for offline processing of petabyte-scale (PB) or larger datasets, significantly reducing the burden on software developers. The Spark component, on the other hand, is a memory-based big data parallel computing framework that improves the real-time performance of data processing in big data environments while ensuring high fault tolerance and scalability. It allows users to deploy Spark components on a large number of inexpensive hardware devices to form a cluster. Combining the MapReduce and Spark components can improve the efficiency of offline computing.

[0060] S240. Use the Phoenix component to add a secondary index to the data stored in the HBase component, and use the Spark and Flink components to perform real-time computation on the data.

[0061] The Phoenix component is a SQL layer built on HBase that allows us to create tables, insert data, and query HBase data using standard JDBC APIs instead of HBase client APIs. It can only query HBase data. This application uses the Phoenix component to improve the efficiency of querying data in HBase.

[0062] Flink is a distributed processing engine for streaming and batch data. The code is mainly implemented in Java, with some code in Scala. Its main application is streaming data, i.e., real-time computing.

[0063] The S250 and DBapi components use the Java Spring Boot framework to encapsulate JDBC drivers, parse JSON parameters, construct dynamic SQL, and are developed using the SQL language to connect to the database and write data interfaces.

[0064] DBapi is a database application programming interface that provides a consistent access interface for different databases, enabling code portability between different databases. In traditional technologies, specific coding is required at the application layer for specific databases. If the underlying database is changed, the database-related programs in the application layer must be changed accordingly. DBapi components, on the other hand, use drivers to replace the underlying database technology. After connecting to the database, all databases can be operated in the same way.

[0065] S260. Connect to the Phoenix component via the DBapi component and write SQL statements to achieve real-time output of data from the HBase component;

[0066] S270, the DBapi component sends a token for verification, and performs blacklist / whitelist checks on the IPs accessing the hive component or the hbase component.

[0067] Token verification technology is used to perform blacklist and whitelist verification on IPs that request access to Hive and HBase to ensure the secure output of data.

[0068] Example 3

[0069] like Figure 3 As shown, a private cloud data management platform includes: a data platform monitoring system, a data service module, and a data application module; wherein,

[0070] The data platform monitoring system 10 is used to collect heterogeneous data and classify the heterogeneous data.

[0071] The data service module 20 is used to provide a unified access interface for the data processed by the platform monitoring system to realize data output, and to verify the IP address accessing the interface;

[0072] The data application module 30 is used to perform personalized application processing on the data output by the data service module.

[0073] As illustrated in Example 3, a private cloud refers to computing services provided exclusively to selected users (rather than the general public) via the Internet or a dedicated internal network. Also known as an internal cloud or corporate cloud, private cloud computing offers enterprises many advantages of public clouds (including self-service, scalability, and elasticity). It provides additional control and customization capabilities through dedicated resources, far surpassing locally hosted computing infrastructure. Furthermore, private clouds offer a higher level of security and privacy through corporate firewalls and internal hosting, ensuring that third-party providers cannot access operational and sensitive data. A drawback is that the company's IT department bears the costs and management responsibilities of the private cloud. Therefore, private clouds require the same staffing, management, and maintenance costs as traditional data center ownership.

[0074] Furthermore, the traditional private cloud architecture is Apache DolphinScheduler + CDH + BI. This framework cannot implement multi-source data query services. Therefore, the BI service requires programmers to develop separate programs to query and match data from multiple sources, and finally merge the data and distribute it to the front end for display. This process involves querying and matching between multiple services. There are many types of databases, and the real-time nature of the data they store varies greatly. Queries are constrained by multiple databases, making it impossible to achieve truly real-time output results. In addition, offline data output is generally achieved on a T+1 basis, i.e., on a daily basis. That is, you can only view the data from the previous day. Running at midnight every day can only be scheduled on a daily basis.

[0075] Based on this, the private cloud in this solution uses an Apache DolphinScheduler + CDH + DBapi component architecture, combined with components such as Flink, Phoenix, and DataX. By using Flink + DBapi, there is no intermediate matching process between different sources, thus achieving true real-time response and providing services to the outside world in real time. Only those familiar with SQL can develop it, reducing the number of developers, shortening the development time of the entire process, and reducing development costs. At the same time, Kafka + DataX + DBapi are used to achieve minute-level or hour-level data output for offline data.

[0076] Because components such as CDH, DS, Flink, Phoenix, and DataX have numerous versions that need to be matched one by one for proper use, it's necessary to check the supported version ranges for each component through the official open-source code. A script is used to automatically add Cloudera's Maven dependencies, compile and package the corresponding Flink, Phoenix, and other component versions, create parcels and CSD packages, start a local HTTP service, integrate it with the CDH and DS services, and finally, test version compatibility by writing test cases. After compilation and testing, connecting to the compiled Flink client via DS allows for visual submission of Flink jobs.

[0077] To meet the characteristics and requirements of a private cloud platform, this solution divides the private cloud platform into different functional modules, using different functional modules to manage different functions, thereby achieving unified management of distributed data. At the same time, the datax component is installed on each machine to match the distributed characteristics of the CDH cluster and enable batch data collection from any host.

[0078] Example 4

[0079] like Figure 4 As shown, a private cloud data management platform includes: a data platform monitoring system, a data service module, and a data application module.

[0080] The data platform monitoring system (CM) is responsible for the initial processing of all data, including data acquisition, data storage, data migration, ETL, and data computation, enabling offline and real-time computation of distributed data. The data service module provides DBapi interface services, offering a consistent access interface for different databases, enabling code portability between different databases, reducing development code, and also performing IP address blacklist / whitelist verification to ensure secure data output. The data application module performs different data application processing according to enterprise needs, such as real-time dashboards, reports, monitoring and alerts, realizing customized processing for enterprise requirements.

[0081] Furthermore, the data platform monitoring system includes:

[0082] The data acquisition module is used to collect heterogeneous data using Flume, Sqoop, and DataX components.

[0083] The data import module is used to transfer the heterogeneous data collected by the data acquisition module to the Kafka component using the Canal component;

[0084] The data storage module is used by the Kafka component to receive the heterogeneous data and store the heterogeneous data in the Hive component and the HBase component respectively according to the data type, wherein the data type includes offline data and real-time data;

[0085] The offline processing module is used to perform offline computations on data stored in the Hive component using the MapReduce and Spark components.

[0086] The real-time processing module is used to add secondary indexes to data stored in the HBase component using the Phoenix component, and to perform real-time computation on the data using the Spark and Flink components.

[0087] Database data is monitored by writing parsing scripts to perform operations such as adding, deleting, and modifying data in binlogs via Flink and synchronized to HBase in real time. Log data, event tracking data, and other data are transmitted to HBase in real time after ETL is written using Flink or Spark. Phoenix is ​​used as the client, thus achieving unified output and storage of multi-source data.

[0088] In real-time computing, even with multi-source data, you can ensure that the data is not lost or duplicated simply by using Flink to write the ETL process to add, delete, and modify different sinks and develop special scenarios.

[0089] Offline data and data analysis are still implemented using traditional Hive. The difference is that Hive is used to associate with HBase or separate data cleaning logic is written. Data is stored in a unified data source, Hive, and DataX implements minute-level or hour-level scheduling. This allows offline data to be more readily available, thus achieving a unified output data result library.

[0090] Furthermore, the data application module includes:

[0091] User profile library, used to build user profiles based on user behavior;

[0092] The real-time dashboard module is used to display the data processing process in real time.

[0093] The reporting submodule is used to compile various data into reports for data analysis.

[0094] The monitoring and early warning submodule is used to monitor the operating status of the management platform and issue an alarm before a fault occurs;

[0095] Other data application submodules are used to perform other personalized processing on the data.

[0096] In this solution, both offline and real-time computations are queried and analyzed under their respective unified data sources, avoiding the problems of slow queries, inability to perform cascading queries, and data silos caused by multi-source data. The versions of Flink and Phoenix components can be freely selected and upgraded within a certain range according to the developers' characteristics. Custom data services achieve unified data from multiple sources, with separate offline and real-time data storage. For computation results, SQL interfaces can be easily developed on the data service interface for online data quality verification, allowing for timely adjustments to custom code logic and facilitating self-testing. Secure management of the data service includes token verification and IP blacklists / whitelists, enabling enterprises to securely output data externally.

[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A private cloud data transmission method, characterized in that, Includes the following steps: Heterogeneous data is collected using Flume, Sqoop, and DataX components, and then transmitted to Kafka via Canal. The Kafka component receives the heterogeneous data and stores it in the Hive and HBase components according to its data type, including offline data and real-time data. Use MapReduce and Spark components to perform offline computations on data stored in the Hive component; We use the Phoenix component to add secondary indexes to the data stored in the HBase component, and use the Spark and Flink components to perform real-time computation on the data. The DBapi component provides a consistent access interface for the Hive and HBase components to enable data output. The provision of a consistent access interface for the Hive and HBase components using the DBapi component to enable data output further includes: Connect to the Phoenix component using the DBapi component to write SQL statements to achieve real-time output of data from the HBase component; Using the Java Spring Boot framework, we encapsulate a JDBC driver, parse JSON parameters, construct dynamic SQL, develop using SQL language, connect to the database, and write data interfaces. The private cloud architecture includes Apache DolphinScheduler, CDH, and Dbapi components, as well as Flink, Phoenix, Sqoop, Kafka, and DataX components.

2. The private cloud data transmission method according to claim 1, characterized in that, The provision of a consistent access interface for the Hive and HBase components using the DBapi component to enable data output also includes: using token verification to perform blacklist and whitelist checks on the IPs accessing the Hive or HBase components.

3. The private cloud data transmission method according to claim 1, characterized in that, The heterogeneous data includes log data, relational data, and event tracking data.

4. A private cloud data management platform, implementing the private cloud data transmission method of claim 1, characterized in that, include: A data platform monitoring system is used to collect heterogeneous data and classify and process the heterogeneous data. The data service module is used to provide a unified access interface for the data processed by the platform monitoring system to realize data output, and to verify the IP address accessing the interface; The data application module is used to perform personalized application processing on the data output by the data service module.

5. A private cloud data management platform according to claim 4, characterized in that, The data platform monitoring system includes: The data acquisition module is used to collect heterogeneous data using Flume, Sqoop, and DataX components. The data import module is used to transfer the heterogeneous data collected by the data acquisition module to the Kafka component using the Canal component; The data storage module is used by the Kafka component to receive the heterogeneous data and store the heterogeneous data in the Hive component and the HBase component respectively according to the data type, wherein the data type includes offline data and real-time data; The offline processing module is used to perform offline computations on data stored in the Hive component using the MapReduce and Spark components. The real-time processing module is used to add secondary indexes to the data stored in the HBase component using the Phoenix component, and to perform real-time computation on the data using the Spark and Flink components. The management platform architecture consists of Apache DolphinScheduler, CDH, and DBapi components, and also includes Flink, Phoenix, Sqoop, Kafka, and DataX components.

6. A private cloud data management platform according to claim 4, characterized in that, The data application module includes: User profile library, used to build user profiles based on user behavior; The real-time dashboard module is used to display the data processing process in real time. The reporting submodule is used to compile various data into reports for data analysis. The monitoring and early warning submodule is used to monitor the operating status of the management platform and issue an alarm before a fault occurs; Other data application submodules are used to perform other personalized processing on the data.

Images