A big data platform deployment method and system for a trusted innovation environment

Abstract

The present invention discloses a big data platform deployment method and system for a trusted innovation environment, which belongs to the field of trusted innovation big data technology. The implementation of the method includes the following processes: 1) constructing an installation package according to the software and hardware information to be deployed; 2) media transmission of the installation package and startup of the management service; 3) initializing all nodes of the environment according to the script provided by the management service; 4) role division of the environment nodes; 5) automatic installation of the big data platform; 6) functional testing of the platform through the management service; 7) performance testing and performance tuning of the platform through the management service. The present invention can adapt to different domestically produced CPUs and operating systems, improve deployment efficiency, realize the rapid deployment and delivery of the big data platform, avoid the waste of human resources due to multiple adaptation scenarios, and circumvent the problems of uncontrollable, error-prone, and inconvenient management of the manual adaptation process.

Description

Technical Field

[0001] The present invention relates to the field of big data technology for ICT-based innovation, and specifically to a big data platform deployment method and system for ICT-based innovation environments. Background Art

[0002] The innovative development of information technology applications is currently a national strategy and a new driving force for national economic development under the current circumstances. The development of ICT is intended to fundamentally address security issues—that is, to make it something we can control, research, develop, and produce. The development of the ICT industry has become crucial for the digital transformation of the economy and the advancement of the industrial chain. By introducing technological systems, strengthening the industrial foundation, and enhancing security capabilities, we are promoting the localization of the ICT industry and driving the transformation of the traditional IT information industry. At the same time, with the advent of the big data era, data applications have penetrated into all walks of life. Big data technology has brought new perspectives to business analysis and industry development, fully stimulating the impact and promotion of data on social development.

[0003] An ICT environment refers to an environment independently developed based on domestically produced CPUs and operating systems within the context of innovative information technology applications. Because the ICT industry is still in its developmental stages and relevant standards have yet to be unified, various issues arise during the hardware and software adaptation process. Generally, adapting to an ICT environment requires human intervention and trial and error, compiling and debugging different big data components for different hardware and software environments. This process consumes significant human resources and is time-consuming.

[0004] Against the backdrop of the rapid development of big data and information technology, a big data platform deployment method that can quickly adapt to different information technology CPUs and operating systems is needed to improve delivery efficiency. Summary of the Invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a big data platform deployment method and system for the information and innovation environment, which can adapt to different domestic CPUs and operating systems, improve deployment efficiency, realize the rapid deployment and delivery of the big data platform, avoid the waste of human resources due to multiple adaptation scenarios, and circumvent the problems of uncontrollable, error-prone and inconvenient management of the manual adaptation process.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A big data platform deployment method for an information-based innovation environment, the implementation of which includes the following steps:

[0008] 1) Build an installation package based on the software and hardware information to be deployed;

[0009] 2) Installation package media transmission and management service startup;

[0010] 3) Initialize all nodes in the environment according to the script provided by the management service;

[0011] 4) Divide the roles of environmental nodes;

[0012] 5) Automated installation of big data platform;

[0013] 6) Perform functional testing of the platform through management services;

[0014] 7) Perform performance testing and performance tuning on the platform through management services.

[0015] This method enables automated deployment of big data platforms in trusted computing environments, encompassing everything from installation package construction, management service startup, node initialization, platform deployment, functional and performance testing, to delivery and commissioning. This significantly shortens the adaptation process and enables rapid delivery.

[0016] Preferably, the installation package is constructed according to the CPU, operating system, and operating system version information of the deployment environment; this process includes compiled large data packages of various types to avoid on-site adaptation during deployment.

[0017] You can compile the components that need to be compiled in advance according to different combinations. After compilation, they can be stored in the library for subsequent construction of installation packages, avoiding recompilation and debugging during the deployment process.

[0018] Preferably, the management services include instance management services, big data application services, task execution services, task management services, functional testing services, and performance testing services;

[0019] Instance management service, mainly responsible for instance management, that is, big data cluster management, supports multi-instance deployment and management, including service startup and shutdown, configuration, and monitoring alarms within the cluster;

[0020] Big data application services provide a visual operation interface to implement resource management, permission management, tenant management, backup and recovery management of components such as HDFS, HBase, Hive, and Kafka, simplifying user operations and providing an easy-to-use operation mode in addition to client operations.

[0021] Task execution service, which controls nodes through SSH to implement deployment operations on each node;

[0022] The task management service is responsible for task scheduling, task status, task logs, and task retry management, and ultimately deploys tasks by calling the task execution service;

[0023] Functional testing service, responsible for functional testing of the entire instance after instance deployment, using prepared automated test scripts to perform testing and issue a functional test report;

[0024] The performance testing service is responsible for performing performance testing on the entire instance after instance deployment is completed. It uses prepared automated test scripts to perform performance testing and issue performance test reports. The performance test scripts can be expanded and performance testing requirements can be gradually improved according to actual needs.

[0025] Preferably, all nodes in the environment are initialized according to the script provided by the management service, that is, after the management service is started, the initialization script is downloaded, the script is uploaded to all nodes in the environment, and then the script is executed on each node to complete the node initialization; the initialization process mainly includes the following steps:

[0026] Check the software and hardware information. If it is inconsistent with the information provided during the build, it will fail and prompt an error.

[0027] Installation package pre-installation check: Checks whether the installation packages of the major data components provided in the installation package can be installed normally on the node. If they cannot be installed normally, an error message will be displayed, and the patch package needs to be re-adapted and re-built to avoid errors during the actual installation process.

[0028] Data disk inspection: detect which data disks are on each node, and format and mount them according to unified processing specifications;

[0029] Add deployment-related system users, create a subsequent deployment-related user on each node, and initialize the public key for mutual trust operations.

[0030] Preferably, the environment nodes are divided into roles, including basic service nodes and big data service nodes;

[0031] The deployment components of basic service nodes include OpenLDAP, Kerberos, and database; the deployment components of big data service nodes include Hadoop, HBase, Hive, Kafka, Spark, Zookeeper, and Ranger.

[0032] Preferably, the big data platform is automatically installed and deployed through management services, node planning, task execution services, and task management services.

[0033] If a deployment fails, you can view the task log through the task management service and use the retry function provided by the task management service to complete the deployment task after manual intervention based on the log information.

[0034] After deployment is complete, you can use the instance through the instance management service.

[0035] Preferably, the platform is functionally tested through the management service, with built-in automated test scripts to test whether the functions of the deployed instances are normal, and a test report is issued after the test is completed;

[0036] The platform is performance tested and tuned through management services. It has built-in performance test scripts for major data components, performs performance tests based on deployed instances, and issues performance test reports. Performance tuning documents are also provided, and the performance of instances can be optimized by referring to the performance test reports.

[0037] This method supports a variety of software and hardware combinations, provides a visual operation page, automates the deployment of the big data platform, and improves deployment and operation efficiency.

[0038] The present invention also claims protection for a big data platform deployment system for an ICT-based innovation environment, which implements the above-mentioned big data platform deployment method for an ICT-based innovation environment;

[0039] The system includes a construction service module, a media transmission module and a management service module. The construction service module is used to build an installation package based on the software and hardware information to be deployed; the media transmission module is used to upload the built installation package to the management node of the information creation environment and start the management service module;

[0040] The system also includes a node initialization module, a platform deployment module, a function test module and a performance test module.

[0041] The present invention also claims protection for a big data platform deployment device for an information innovation environment, comprising: at least one memory and at least one processor;

[0042] The at least one memory is configured to store a machine-readable program;

[0043] The at least one processor is used to call the machine-readable program to execute the above-mentioned big data platform deployment method for the information technology innovation environment.

[0044] The present invention also requires protection of computer-readable media, which store computer instructions. When the computer instructions are executed by a processor, the processor executes the above-mentioned big data platform deployment method for the information innovation environment.

[0045] Compared with the prior art, the big data platform deployment method and system for the information innovation environment of the present invention have the following beneficial effects:

[0046] This method and system supports software and hardware combinations of different CPUs and operating systems. Based on different combinations, major data components are centrally compiled and adapted in advance, eliminating the need for manual adaptation in the delivery environment and saving R&D time.

[0047] The instance management service supports multi-instance deployment. One set of management services can complete the deployment and management of multiple environments, saving deployment resources, improving management efficiency, and reducing operation and maintenance costs.

[0048] It provides task management services. All operations are issued in the form of tasks, and each task is tracked and managed throughout its life cycle. The task status, task log, and task retry functions it provides can effectively assist in task execution.

[0049] Provides functional testing capabilities, which can be used to perform functional testing after the platform is deployed to ensure that the platform is normal and available after delivery;

[0050] Provide performance testing services, which can perform performance testing on the platform after deployment, and provide optimization suggestions while demonstrating the platform performance to ensure the quality of the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a diagram of the system architecture of a big data platform deployment for a trusted innovation environment provided by an embodiment of the present invention;

[0052] Figure 2 It is an overall deployment flow chart of the big data platform deployment method for the trusted innovation environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] A deployment system and method for a big data platform for an information and innovation environment. The method supports the combination selection of multiple CPU models and operating systems; multiple instance deployment can be achieved by deploying only one set of management services; supports automated deployment of the big data platform, and realizes full life cycle management of the deployment process through task management services; supports functional testing of the platform after deployment to ensure normal operation of the platform; and supports performance testing of the platform after deployment to ensure efficient operation of the platform.

[0055] The components that need to be compiled can be compiled in advance according to different combinations. After compilation, they can be stored in the database for subsequent construction of installation packages, avoiding recompilation and debugging during the deployment process. It also supports instance management to better manage deployment resources as a whole, improve operation and maintenance efficiency, and save deployment resources. The full life cycle management of the deployment process can be achieved through task management services.

[0056] Built-in automated testing scripts can realize automated functional testing, eliminating the need for manual testing and improving testing efficiency. It also has built-in performance testing scripts for major data components and supports extensions. It can perform personalized performance testing according to needs and perform performance tuning based on test results to ensure efficient operation of the platform.

[0057] The specific implementation process of this method is as follows:

[0058] 1. Build the installation package,

[0059] The installation package is built based on the CPU, operating system, and operating system version information of the deployment environment. This process includes compiled large data packages of various types, avoiding on-site adaptation during the deployment process.

[0060] 2. Installation package media transmission and management node service startup,

[0061] The built installation package needs to be uploaded to the management node of the trusted computing environment. After the upload is complete, you only need to execute the startup script to complete the startup of the management service. The startup process will complete the detection of software and hardware information. It can only be executed successfully if it matches the software and hardware information provided during the build.

[0062] Management services consist of instance management services, big data application services, task execution services, task management services, functional testing services, and performance testing services:

[0063] 1. Instance management service: This service is primarily responsible for instance management, specifically big data cluster management, supporting multi-instance deployment and management. This includes starting and stopping services within the cluster, configuring services, and monitoring alerts.

[0064] 2. Big data application services provide a visual operation page that can implement resource management, permission management, tenant management, backup and recovery management, and other operations for components such as HDFS, HBase, Hive, and Kafka. This greatly simplifies user operation difficulty and provides an easy-to-use operation mode in addition to client operations.

[0065] 3. Task execution service, which controls nodes through SSH to implement deployment operations on each node.

[0066] 4. Task management service, responsible for task scheduling, task status, task logs, task retry management, and ultimately deploying tasks by calling the task execution service.

[0067] 5. Functional testing service: responsible for functional testing of the entire instance after the instance deployment is completed, using the prepared automated test scripts to test and issue a functional test report.

[0068] 6. Performance testing service: responsible for performance testing the entire instance after instance deployment is completed, using prepared automated test scripts to perform performance testing and issue performance test reports. The performance test scripts can be expanded and performance testing requirements can be gradually improved according to actual needs.

[0069] 3. Initialize all nodes in the environment according to the script provided by the management service.

[0070] After the management service is started, you can download the initialization script, upload this script to all nodes in the environment, and then execute this script on each node to complete the node initialization.

[0071] The initialization process mainly includes the following steps:

[0072] S1. Check the software and hardware information. If it is inconsistent with the information provided during the build, the system will fail and prompt an error.

[0073] S2. Pre-installation check of the installation package: Checks whether the installation packages of the major data components provided in the installation package can be installed normally on the node. If they cannot be installed normally, an error message will be displayed, and the patch package needs to be re-adapted and re-built to avoid errors during the actual installation process.

[0074] S3. Data disk inspection: detect which data disks are on each node and format and mount them according to unified processing specifications.

[0075] S4. Add deployment-related system users. Each node creates a subsequent deployment-related user and initializes the public key for mutual trust operations.

[0076] 4. Divide the roles of environmental nodes into basic service nodes and big data service nodes.

[0077] This step mainly involves planning the node roles of the big data platform in advance. The basic service nodes mainly deploy components such as OpenLDAP, Kerberos, and databases; the big data service nodes mainly deploy components such as Hadoop, HBase, Hive, Kafka, Spark, Zookeeper, and Ranger.

[0078] 5. Automated installation of big data platform

[0079] Once the first few steps are complete, you can begin platform deployment. Using the management service, you can deploy the platform based on the pre-planned nodes, task execution services, and task management services. Under normal circumstances, platform automated deployment can be completed quickly. If deployment fails, you can view the task logs through the task management service and, based on the log information, manually intervene using the retry function provided by the task management service to complete the deployment task. After deployment is complete, you can use the instance management service to use it.

[0080] 6. Functional testing of the platform functions is performed through management services. The service has built-in automated testing scripts that can automatically test whether the functions of the deployed instances are normal. A test report will be issued after the test is completed.

[0081] 7. Performance testing and performance tuning of the platform are carried out through management services. The service has built-in performance testing scripts for each big data component, which can perform performance testing based on the actual deployment. After the test is completed, a performance test report is issued, and performance tuning documents are also provided. The performance of the instance can be optimized by referring to the performance test report.

[0082] The above steps enable automated deployment of the big data platform in a trusted computing environment, encompassing everything from building the installation package, starting the management service, initializing nodes, deploying the platform, performing functional and performance testing, and ultimately delivering it for use. This significantly shortens the adaptation process and enables rapid delivery.

[0083] The embodiment of the present invention also provides a big data platform deployment system for an information-based innovation environment, which includes a construction service module, a media transmission module, and a management service module. The construction service module is used to build an installation package according to the software and hardware information to be deployed; the media transmission module is used to upload the built installation package to the management node of the information-based innovation environment and start the management service module;

[0084] The system also includes a node initialization module, a platform deployment module, a function test module and a performance test module.

[0085] The system implements the big data platform deployment method for the information and innovation environment described in the above embodiments of the present invention.

[0086] An embodiment of the present invention further provides a big data platform deployment device for an information innovation environment, comprising: at least one memory and at least one processor;

[0087] The at least one memory is configured to store a machine-readable program;

[0088] The at least one processor is used to call the machine-readable program to execute the big data platform deployment method for the information technology innovation environment described in the above embodiment of the present invention.

[0089] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon, and when the computer instructions are executed by a processor, the processor is caused to execute the method for deploying a big data platform for a trust-based innovation environment as described in the above-mentioned embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes for implementing the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device is caused to read and execute the program code stored in the storage medium.

[0090] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0091] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0092] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0093] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0094] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A big data platform deployment method for a trusted innovation environment, characterized in that: The implementation of this method includes the following steps: 1) Build the installation package based on the software and hardware information to be deployed; 2) Installation package media transmission and management service startup; 3) Initialize all nodes in the environment according to the script provided by the management service; 4) Divide the roles of environmental nodes; 5) Automated installation of big data platform; 6) Conduct functional testing of the platform through management services; 7) Conduct performance testing and performance tuning of the platform through management services; The initialization of all nodes in the environment is performed according to the script provided by the management service. That is, after the management service is started, the initialization script is downloaded, the script is uploaded to all nodes in the environment, and then the script is executed on each node to complete the node initialization. The initialization process includes the following steps: Check the software and hardware information. If it is inconsistent with the information provided during the build, it will fail and prompt an error. Installation package pre-installation check: Checks whether the installation packages of the major data components provided in the installation package can be installed normally on the node. If they cannot be installed normally, an error message will be displayed, and the patch package needs to be re-adapted and re-built to avoid errors during the actual installation process. Data disk inspection: detect which data disks are on each node, and format and mount them according to unified processing specifications; Add deployment-related system users. Each node creates a subsequent deployment-related user and initializes the public key for mutual trust operations. The role division of the environment nodes includes basic service nodes and big data service nodes; The deployment components of basic service nodes include OpenLDAP, Kerberos, and database; the deployment components of big data service nodes include Hadoop, HBase, Hive, Kafka, Spark, Zookeeper, and Ranger.

2. A big data platform deployment method for a trust-based innovation environment according to claim 1, characterized in that The installation package is constructed based on the CPU, operating system, and operating system version information of the deployment environment; this process includes compiled large data packages of various types to avoid on-site adaptation during deployment; Compile the components that need to be compiled in advance according to different combinations, and put them into the library after compilation for subsequent construction of the installation package.

3. A big data platform deployment method for an information innovation environment according to claim 1 or 2, characterized in that: The management services include instance management services, big data application services, task execution services, task management services, functional testing services, and performance testing services; Instance management service, responsible for instance management, that is, big data cluster management, supports multi-instance deployment and management, including starting and stopping services within the cluster, configuration, and monitoring alarms; Big data application services provide a visual operation page to implement component resource management, permission management, tenant management, backup and recovery management, simplifying user operations and providing an easy-to-use operation mode in addition to client operations. Task execution service, which controls nodes through SSH to implement deployment operations on each node; The task management service is responsible for task scheduling, task status, task logs, and task retry management, and ultimately deploys tasks by calling the task execution service; Functional testing service, responsible for functional testing of the entire instance after instance deployment, using prepared automated test scripts to perform testing and issue a functional test report; The performance testing service is responsible for performing performance testing on the entire instance after instance deployment is completed. It uses prepared automated test scripts to perform performance testing and issue performance test reports. The performance test scripts can be expanded and performance testing requirements can be gradually improved according to actual needs.

4. A method for deploying a big data platform for an information innovation environment according to claim 3, characterized in that: The big data platform is automatically installed and deployed through management services, node planning, task execution services, and task management services. If a deployment fails, you can view the task log through the task management service and use the retry function provided by the task management service to complete the deployment task after manual intervention based on the log information. After deployment is complete, you can use the instance through the instance management service.

5. A big data platform deployment method for an information innovation environment according to claim 1 or 2, characterized in that: The platform is functionally tested through management services, with built-in automated test scripts to test whether the functions of the deployed instances are normal, and a test report is issued after the test is completed; The platform is performance tested and tuned through management services. It has built-in performance test scripts for major data components, performs performance tests based on deployed instances, and issues performance test reports. Performance tuning documents are also provided, and the performance of instances can be optimized by referring to the performance test reports.

6. A big data platform deployment system for a trusted innovation environment, characterized in that: The system implements the big data platform deployment method for the trust innovation environment as described in any one of claims 1-5; It includes a construction service module, a media transmission module, and a management service module. The construction service module is used to build an installation package based on the software and hardware information to be deployed; the media transmission module is used to upload the built installation package to the management node of the information creation environment and start the management service module; The system also includes a node initialization module, a platform deployment module, a function test module and a performance test module.

7. A big data platform deployment device for a trusted innovation environment, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 5.

8. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, enable the processor to perform the method according to any one of claims 1 to 5.

Images