A cloud computing big data platform system based on microservices

Abstract

The present invention discloses a cloud computing big data platform system based on microservices, including a big data platform and a cloud computing platform connected by a microservice module, wherein the cloud computing platform includes a virtual machine module, a container module, a CloudAgent module, a computing module, a mirror module, and a network module, and the big data platform includes a database module, a distributed file system module, a big data management module, a big data development module, and a big data application module. The present invention integrates the installation, configuration, and use of various open source big data technologies Hadoop, HBase, Hive, Spark, Zookeeper, Sqoop, Pig, Kafka, Flume, and Mahout, as well as program codes, sample programs, and running scripts into a virtual machine or container, and in this virtual machine or container, the CloudAgent program is installed and configured to realize functions such as configuring a big data cluster service, starting a big data cluster service, stopping a big data cluster service, and running a big data operation service. The present invention can realize centralized deployment of big data resources and on-demand use.

Description

Technical Field

[0001] The present invention relates to the management of big data cluster services in the field of exploration and development, and the operation of big data operation services, and in particular to a cloud computing big data platform system based on microservices. Background Art

[0002] In the field of exploration and development, petroleum big data processing technology has been widely used, and there are mature application examples in reservoir prediction, sand body connectivity, sedimentary microfacies analysis, oil and gas well operating condition warning, drilling engineering warning, etc. The application of petroleum big data processing technology requires a stable and reliable big data analysis platform.

[0003] Hadoop, a big data platform, is a distributed system infrastructure developed by the Apache Foundation. It can fully utilize the power of clusters for high-speed computing and storage. Hadoop big data processing technology has a history of more than ten years and is regarded as the preferred solution for big data cluster processing. Spark is a fast and general computing engine designed for large-scale data processing. It enables memory-distributed data sets. In addition to providing interactive queries, it can also optimize iterative workloads. HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. Hive is a data warehouse tool based on Hadoop that can map structured data files to a database table and provide simple SQL query functions. It can also convert SQL statements into MapReduce tasks. ZooKeeper is a distributed application coordination service and an important component of Hadoop and HBase. It provides Java and C interfaces. Sqoop is a tool under Apache that transfers data between Hadoop and relational database servers. It has two core functions: importing and exporting data. Importing data is to import MySQL and Oracle data into Hadoop's HDFS, Hive, HBase and other data storage systems; exporting data is to export data from the Hadoop file system to relational databases such as MySQL.

[0004] The big data analysis platform runs on the Linux operating system, is distributed and has a complex structure. Business personnel need to manually complete system management and platform operation and maintenance through command lines or system built-in tools, and cannot focus on solving business problems, which seriously affects work efficiency. Summary of the invention

[0005] The purpose of the present invention is to solve the deficiencies in the prior art and provide a cloud computing big data platform system based on microservices.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] A cloud computing big data platform system based on microservices includes a big data platform and a cloud computing platform connected by a microservice module, the cloud computing platform includes a virtual machine module, a container module, a CloudAgent module, a computing module, a mirror module, and a network module, and the big data platform includes a database module, a distributed file system module, a big data management module, a big data development module, and a big data application module. Among them:

[0008] The computing module of the cloud computing platform is used to provide virtual machine or container operation management services, including creating, starting, and stopping virtual machine or container services;

[0009] The image module of the cloud computing platform is used to provide virtual machine or container image management services, including compiling, uploading, synchronizing, and deleting virtual machine or container image services;

[0010] The network module of the cloud computing platform is used to provide virtual machine or container network services, including creating, editing, and deleting network / subnet services;

[0011] The virtual machine module of the cloud computing platform includes virtual machines for various big data software;

[0012] The container module of the cloud computing platform includes containers for various big data software;

[0013] The CloudAgent module of the cloud computing platform represents the cloud agent service program running in a virtual machine or container;

[0014] The microservice module is used to provide microservice interfaces, including creating virtual machines / containers, creating big data clusters, starting big data cluster services, and creating big data job services;

[0015] The big data management module of the big data platform is used to provide big data cluster management, including creation, deployment, configuration, service startup, and deletion of big data cluster services;

[0016] The big data development module of the big data platform is used to provide big data job creation and operation services;

[0017] The big data application module of the big data platform is used to provide big data application modeling and analysis services;

[0018] The database module of the big data platform is used to provide code, operation results, and operation status services for big data jobs;

[0019] The distributed file system module of the big data platform is used to provide services for saving big data job code files and operation result files.

[0020] Furthermore, the big data management module of the big data platform and the specific steps of the front-end portal website for performing big data cluster deployment are as follows:

[0021] Step 1: The system administrator edits the big data platform service template file and specifies the big data cluster parameters;

[0022] Step 2: The system administrator publishes the big data platform service;

[0023] Step 3: The tenant administrator applies for a big data cluster and fills in the big data cluster application parameters;

[0024] Step 4: The approval administrator approves the big data cluster and checks the application parameters of the big data cluster;

[0025] Step 5: The system administrator deploys the big data cluster according to the big data cluster application parameters. The cloud computing platform will create a virtual machine / container for the big data cluster.

[0026] Step 6: The tenant administrator / ordinary user views the big data cluster, and checks the big data cluster machines and configuration information;

[0027] Step 7: The tenant administrator / ordinary user modifies / starts / stops the big data cluster service. The virtual machines / containers of the big data platform start and stop the big data cluster service according to the specified configuration.

[0028] Furthermore, the steps of the big data management module of the big data platform and the CloudAgent program processing of the big data cluster deployment virtual machine / container agent are as follows:

[0029] Step 1: Create a virtual machine or container based on the big data cluster deployment;

[0030] Step 2: Initialize the CloudAgent program on the running big data cluster virtual machine / container;

[0031] Step 3: The CloudAgent program reads the big data configuration information from the database, obtains the big data cluster host name, IP address, whether it is configured, and other parameters, and configures the big data cluster. If the local machine is the master node of the big data cluster, the configuration file must also be copied to other machines in the big data cluster.

[0032] Step 4: The CloudAgent program listens to port 5995;

[0033] Step 5: The CloudAgent program waits for the client's connection request;

[0034] Step 6: If a connection request comes in, accept the connection and receive data;

[0035] Step 7: Parse command parameters;

[0036] Step 8: If the connection request command is requestSetupBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, the number of machines in the big data cluster; whether Hadoop, HBase, Hive, Spark, and Zookeeper are configured; and configure the big data cluster;

[0037] Step 9: If the connection request command is requestStartBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, whether to start hadoop, whether to start hbase, whether to start hive, whether to start spark, whether to start zookeeper, and start the big data cluster service;

[0038] Step 10: If the connection request command is requestStopBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, whether to stop hadoop, whether to stop hbase, whether to stop hive, whether to stop spark, whether to stop zookeeper; and stop the big data cluster service;

[0039] Step 11: Return the result.

[0040] Furthermore, the specific steps of the big data development module of the big data platform to perform big data operation operations are as follows:

[0041] Step 1: Ordinary users create a big data job and select the big data job type;

[0042] Step 2: Ordinary users edit big data jobs online, rewrite the code according to the sample program, and save it to the database and distributed file system;

[0043] Step 3: Ordinary users run big data jobs. If there are multiple big data analysis platforms, select the big data analysis platform server and run the big data job.

[0044] Step 4: Ordinary users view the running results of the big data job, or re-run the big data job to view the results.

[0045] Furthermore, the steps of processing big data jobs of the big data development module of the big data platform and the big data cluster virtual machine / container agent CloudAgent program are as follows:

[0046] Step 1: Create a virtual machine or container based on the big data cluster deployment;

[0047] Step 2: Initialize the CloudAgent program on the running big data cluster virtual machine / container;

[0048] Step 3: The CloudAgent program listens to port 5995;

[0049] Step 4: The CloudAgent program waits for the client's connection request;

[0050] Step 5: If a connection request comes in, accept the connection, receive data, and parse command parameters;

[0051] Step 6: If the connection request command is requestRunBigdata, further parse other parameters and obtain the big data job file from the distributed file system;

[0052] Step 7: Run the big data job;

[0053] Step 8: Return the result.

[0054] Compared with the prior art, the present invention integrates the installation, configuration, use, program code, sample program, and running script of various open source big data technologies Hadoop, HBase, Hive, Spark, Zookeeper, Sqoop, Pig, Kafka, Flume, and Mahout into a virtual machine or container. In this virtual machine or container, the CloudAgent program is installed and configured to realize functions such as configuring big data cluster services, starting big data cluster services, stopping big data cluster services, and running big data job services. The present invention can realize centralized deployment of big data resources and on-demand use. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a system architecture diagram of the microservice-based cloud computing big data platform of the present invention.

[0056] Figure 2 Automatically deploy system architecture diagram for big data platform cluster.

[0057] Figure 3 Run processes for big data platform jobs. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0059] The present embodiment is a cloud computing big data platform system based on microservices, the system architecture design is as follows Figure 1 As shown, including:

[0060] (1) The computing module (C101) of the cloud computing platform provides virtual machine or container operation management services, including creating, starting, and stopping virtual machine or container services;

[0061] (2) The cloud computing platform image module (C102) provides virtual machine or container image management services, including compiling, uploading, synchronizing, and deleting virtual machine or container image services;

[0062] (3) The cloud computing platform network module (C103) provides virtual machine or container network services, including creating, editing, and deleting network / subnet services;

[0063] (4) Cloud computing platform virtual machine module (C104) represents a virtual machine containing various big data software;

[0064] (5) Cloud computing platform container module (C105) represents a container containing various big data software;

[0065] (6) Cloud computing platform CloudAgent module (C106) represents the cloud agent service program running in the virtual machine or container;

[0066] (7) Microservice module S201 provides microservice interfaces, including creating virtual machines / containers, creating big data clusters, starting big data cluster services, and creating big data job services;

[0067] (8) The big data platform big data management module B301 provides big data cluster management, including creation, deployment, configuration, service startup, and deletion of big data cluster services;

[0068] (9) The big data platform big data development module B302 provides big data job creation and operation services;

[0069] (10) Big data platform big data application module B303 provides big data application modeling and analysis services;

[0070] (11) The big data platform database module B304 provides code, operation results, and operation status services for big data jobs;

[0071] (12) The big data platform distributed file system B305 provides services such as saving big data job code files and operation result files.

[0072] In the big data management module of the big data platform, the big data cluster automatic deployment system architecture design is as follows Figure 2 As shown, the front-end portal website of the cloud computing big data platform performs big data cluster deployment operations, including the following steps:

[0073] Step 1: The system administrator edits the big data platform service template file and specifies the big data cluster parameter D101;

[0074] Step 2: The system administrator releases the big data platform service D102;

[0075] Step 3: The tenant administrator applies for a big data cluster and fills in the big data cluster application parameter D103;

[0076] Step 4: The approval administrator approves the big data cluster and checks the big data cluster application parameter D104;

[0077] Step 5: The system administrator deploys the big data cluster. The big data cluster D105 is deployed according to the big data cluster application parameters. The cloud computing platform will create a virtual machine / container for the big data cluster.

[0078] Step 6: The tenant administrator / ordinary user views the big data cluster, and checks the big data cluster machines and configuration information D106;

[0079] Step 7: The tenant administrator / ordinary user modifies / starts / stops the big data cluster service D107. The virtual machines / containers of the big data platform start and stop the big data cluster service according to the specified configuration.

[0080] The big data management module in the big data platform, the big data cluster virtual machine / container agent CloudAgent program processing flow automatically deployed by the big data cluster, includes the following steps:

[0081] Step 1: Create a virtual machine or container D201 according to the big data cluster deployment;

[0082] Step 2: Initialize the CloudAgent program D202 of the running big data cluster virtual machine / container;

[0083] Step 3: The CloudAgent program reads the big data configuration information from the database, obtains the big data cluster host name, IP address, whether it is configured, and other parameters, and configures the big data cluster. If the local machine is the master node of the big data cluster, the configuration file must also be copied to other machines in the big data cluster D203;

[0084] Step 4: The CloudAgent program listens on port 5995D204;

[0085] Step 5: The CloudAgent program waits for the client's connection request D205;

[0086] Step 6: If a connection request comes in, accept the connection and receive data D206;

[0087] Step 7: Analyze command parameter D207;

[0088] Step 8: If the connection request command is requestSetupBigdata, further parse other parameters to obtain the machine name of the master node of the big data cluster, the IP address of the master node, the number of machines in the big data cluster; whether hadoop, hbase, hive, spark, and zookeeper are configured; and configure the big data cluster D208;

[0089] Step 9: If the connection request command is requestStartBigdata, further parse other parameters to obtain the machine name of the master node of the big data cluster, the IP address of the master node, whether to start hadoop, whether to start hbase, whether to start hive, whether to start spark, whether to start zookeeper, and start the big data cluster service D209;

[0090] Step 10: If the connection request command is requestStopBigdata, further parse other parameters to obtain the machine name of the master node of the big data cluster, the master node IP address, whether to stop hadoop, whether to stop hbase, whether to stop hive, whether to stop spark, whether to stop zookeeper; and stop the big data cluster service D210;

[0091] Step 11: Return result D211.

[0092] The big data development module in the big data platform, the big data platform operation process design is as follows Figure 3 As shown in the figure, the front-end portal of the big data analysis platform performs big data operation operations, including the following steps:

[0093] Step 1: Ordinary users create a big data job and select the big data job type J101;

[0094] Step 2: Ordinary users edit the big data job online, rewrite the code according to the sample program, and save it to the database and distributed file system J102;

[0095] Step 3: Ordinary users run big data jobs. If there are multiple big data analysis platforms, select the big data analysis platform server and run big data job J103.

[0096] Step 4: Ordinary users can view the running results of the big data job, or re-run the big data job to view the results J104.

[0097] The big data development module in the big data platform, the big data cluster virtual machine / container agent CloudAgent program big data job processing flow, includes the following steps:

[0098] Step 1: Create a virtual machine or container J201 according to the big data cluster deployment;

[0099] Step 2: Initialize the CloudAgent program J202 of the running big data cluster virtual machine / container;

[0100] Step 3: CloudAgent program listens on port 5995J203;

[0101] Step 4: The CloudAgent program waits for the client's connection request J204;

[0102] Step 5: If a connection request comes in, accept the connection, receive data, and parse command parameter J205;

[0103] Step 6: If the connection request command is requestRunBigdata, further parse other parameters and obtain the big data job file J206 from the distributed file system;

[0104] Step 7: Run big data job J207;

[0105] Step 8: Return result J208.

[0106] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A cloud computing big data platform system based on microservices, characterized in that: It includes a big data platform and a cloud computing platform connected by a microservice module. The cloud computing platform includes a virtual machine module, a container module, a CloudAgent module, a computing module, a mirror module, and a network module. The big data platform includes a database module, a distributed file system module, a big data management module, a big data development module, and a big data application module, wherein: The computing module of the cloud computing platform is used to provide virtual machine or container operation management services, including creating, starting, and stopping virtual machine or container services; The image module of the cloud computing platform is used to provide virtual machine or container image management services, including compiling, uploading, synchronizing, and deleting virtual machine or container image services; The network module of the cloud computing platform is used to provide virtual machine or container network services, including creating, editing, and deleting network / subnet services; The virtual machine module of the cloud computing platform includes virtual machines for various big data software; The container module of the cloud computing platform includes containers for various big data software; The CloudAgent module of the cloud computing platform represents the cloud agent service program running in a virtual machine or container; The microservice module is used to provide microservice interfaces, including creating virtual machines / containers, creating big data clusters, starting big data cluster services, and creating big data job services; The big data management module of the big data platform is used to provide big data cluster management, including creation, deployment, configuration, service startup, and deletion of big data cluster services; The big data development module of the big data platform is used to provide big data job creation and operation services; The big data application module of the big data platform is used to provide big data application modeling and analysis services; The database module of the big data platform is used to provide code, operation results, and operation status services for big data jobs; The distributed file system module of the big data platform is used to provide services for saving big data job code files and operation result files; The CloudAgent module of the big data platform and the steps of the CloudAgent program processing on the big data cluster deployment virtual machine / container agent are as follows: Step 1: Create a virtual machine or container based on the big data cluster deployment; Step 2: Initialize the CloudAgent program on the running big data cluster virtual machine / container; Step 3: The CloudAgent program reads the big data configuration information from the database, obtains the big data cluster host name, IP address, whether to configure parameters, and configures the big data cluster. If the local machine is the master node of the big data cluster, the configuration file must also be copied to other machines in the big data cluster. Step 4: The CloudAgent program listens to port 5995; Step 5: The CloudAgent program waits for the client's connection request; Step 6: If a connection request comes in, accept the connection and receive data; Step 7: Parse command parameters; Step 8: If the connection request command is requestSetupBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, the number of machines in the big data cluster; whether Hadoop, HBase, Hive, Spark, and Zookeeper are configured; and configure the big data cluster; Step 9: If the connection request command is requestStartBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, whether to start hadoop, whether to start hbase, whether to start hive, whether to start spark, whether to start zookeeper, and start the big data cluster service; Step 10: If the connection request command is requestStopBigdata, further parse to obtain the machine name of the master node of the big data cluster, the IP address of the master node, whether to stop hadoop, whether to stop hbase, whether to stop hive, whether to stop spark, whether to stop zookeeper, and stop the big data cluster service; Step 11: Return the result; The big data development module of the big data platform performs the following specific steps for big data operation: Step 1: Ordinary users create a big data job and select the big data job type; Step 2: Ordinary users edit big data jobs online, rewrite the code according to the sample program, and save it to the database and distributed file system; Step 3: Ordinary users run big data jobs. If there are multiple big data analysis platforms, select the big data analysis platform server and run the big data job. Step 4: Ordinary users view the results of the big data job, or re-run the big data job to view the results; The steps of the big data development module of the big data platform and the big data cluster virtual machine / container agent CloudAgent program big data job processing are as follows: Step 1: Create a virtual machine or container based on the big data cluster deployment; Step 2: Initialize the CloudAgent program on the running big data cluster virtual machine / container; Step 3: The CloudAgent program listens to port 5995; Step 4: The CloudAgent program waits for the client's connection request; Step 5: If a connection request comes in, accept the connection, receive data, and parse command parameters; Step 6: If the connection request command is requestRunBigdata, further parse other parameters and obtain the big data job file from the distributed file system; Step 7: Run the big data job; Step 8: Return the result.

2. The microservice-based cloud computing big data platform system according to claim 1, characterized in that: The big data management module of the big data platform and the specific steps of the front-end portal website to deploy the big data cluster are as follows: Step 1: The system administrator edits the big data platform service template file and specifies the big data cluster parameters; Step 2: The system administrator publishes the big data platform service; Step 3: The tenant administrator applies for a big data cluster and fills in the big data cluster application parameters; Step 4: The approval administrator approves the big data cluster and checks the application parameters of the big data cluster; Step 5: The system administrator deploys the big data cluster according to the big data cluster application parameters. The cloud computing platform will create a virtual machine / container for the big data cluster. Step 6: The tenant administrator / ordinary user views the big data cluster, and checks the big data cluster machines and configuration information; Step 7: The tenant administrator / ordinary user modifies / starts / stops the big data cluster service. The virtual machines / containers of the big data platform start and stop the big data cluster service according to the specified configuration.

Images