A big data management deployment method applied to a localized operating system

Abstract

The application provides a big data management deployment method applied to a localized operating system, and is characterized in that the method comprises the following steps: downloading Ambari source code on a host installed with a localized operating system; modifying the Ambari source code; after the modification of the Ambari source code is completed, running a compilation command to compile the Ambari source code, obtaining a Yum source after the compilation is completed, then configuring the Yum source of the big data management software for use in installing the big data management software; and one-key installation and configuration of the big data management software. After the method is adopted, the big data management software can be installed on a machine running the localized operating system, and one-key installation and one-key configuration of the big data management software can be realized. Furthermore, a person skilled in the art can install other big data application services except for the big data application services provided in the kit according to actual project needs.

Description

Technical Field

[0001] This invention relates to a deployment scheme for big data management services applied to a domestically developed operating system. Background Technology

[0002] With the continuous advancement and development of domestically developed operating systems, compatibility with domestic systems such as Kylin has become increasingly urgent. Kylin is an Arm-based system (this architecture is favored for its low power consumption and economic advantages, and will see wider adoption in future service architecture designs). However, most ecosystems are deployed on x86 architecture systems, including big data management services and application services. Currently, most popular big data management platforms, both domestically and internationally, do not offer Arm-based compatible installation packages.

[0003] Meanwhile, existing big data management systems also have some significant shortcomings: First, the installation process is complex, requiring manual installation and configuration of many dependent services, which is difficult and requires personnel with a certain technical background, making it impossible to achieve a user-friendly installation or one-click configuration; second, the types of big data application services that can be hosted are limited, only those included within the suite can be installed, and services outside the suite cannot be installed; third, they only support installation on x86 architecture systems and do not support the installation and deployment of domestically developed systems. These shortcomings of big data management systems will restrict the development of domestically developed big data technologies. Summary of the Invention

[0004] One objective of this invention is to address the current lack of marketable big data management software solutions compatible with Arm architecture systems, and the fact that existing big data management software is complex to install and requires highly skilled maintenance personnel. A further objective of this invention is to address the issue of limited application service compatibility.

[0005] To achieve the above objectives, the technical solution of the present invention provides a big data management and deployment method applied to a domestically developed operating system, characterized by comprising the following steps:

[0006] Step 1: Download the Ambari source code on a host machine with a domestic operating system installed, and install the dependency environment required for compiling the Ambari source code at the same time;

[0007] Step 2: Modify the Ambari source code, including the following:

[0008] Modify the installation script and configuration files:

[0009] Add description information about the domestic operating system to the configuration file os_family.json in the $ambari-server / lib / ambari_commons / resources directory;

[0010] Modify the script file os_check.py in the $ambari-server / lib / ambari_commons directory to enable the script file os_check.py to recognize the domestic operating system and its version information, ensuring a smooth installation.

[0011] Modify the script files params.py, yum_manager.py, script.py, os_check.py, and system.py for the big data management service and agent service, including:

[0012] Modify the script file params.py to add logic code to obtain the base_url path of the repo file, so that the script for installing big data management software can recognize the Yum source path and URL information during the installation process, which is used for the installation of application service components;

[0013] Modify the `yum_check_package_available` method in the script file `yum_manager.py` to obtain the `hdp-select` version by reading the global configuration of the big data management software, and use it to verify whether the big data management software installation package has been installed;

[0014] Modify the STACK_VERSION_PLACEHOLDER parameter in the script file script.py to obtain the HDP version number information from the global configuration of the big data management software, so that the subsequent execution script can recognize the HDP version information;

[0015] Modify the logic of the os_distribution method in the script file os_check.py to read the corresponding operating system and operating system version information from the global configuration of the big data management software, so that the big data management software can identify the Kylin operating system;

[0016] Modify the `action_create` method in the script file `system.py` to determine whether the contents of the repo configuration file under the Yum repository are the same as the configuration file loaded when the big data management software starts. If they are different, the new Yum repository configuration file will overwrite the old configuration file.

[0017] Modify the *.pom description file and *.py executable file in the Ambari source code package, manually replacing the keywords amd64 and noarch that are recognized by x86 systems with the keyword aarch64 that is recognized by Arm architecture systems;

[0018] Modify the script files os_check.py and os_utils.py for the management service and proxy service:

[0019] Modify the configuration file of the script file os_check.py, and modify the ambari_repo_file attribute in the get_ambari_repo_file_full_name method to specify that it should be obtained from the global configuration of the big data management software, so as to correctly identify the configuration path of the Yum source corresponding to the big data management software;

[0020] Modify the operatingSystem attribute of the _get_os_type method in the two script files os_utils.py to set it to centos, and modify the type attribute of the _get_os_version method to set it to 7, so that it can be recognized by the big data management software installation script;

[0021] Create a new directory $ambari-server / resources / wdbdp / jdk. In the newly created $ambari-server / resources / wdbdp / jdk directory, add a script file named install.sh. Write the following logic in the install.sh script: check if JDK is already installed on the host. If not, automatically call the command yum install oracle-j2sdk1.8 to install JDK.

[0022] Modify the ambari-server file in the $ambari-server / sbin directory, and add a call to the install.sh script file in the $ambari-server / resources / wdbdp / jdk directory;

[0023] Create a new directory $ambari-server / resources / wdbdp / mysql. In this new directory, add the script file install.sh and the file mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz. Add logic to the install.sh script to check and install MySQL. If MySQL is already installed on the host, skip the installation steps; otherwise, install MySQL.

[0024] Create a new directory $ambari-server / resources / wdbdp / mysql / init_script, and add an Ambari-DDL-MySQL-CREATE.sql file to the newly created $ambari-server / resources / wdbdp / mysql / init_script directory;

[0025] Create a new directory `$ambari-server / src / main / python / wdb_dm`. In this new directory, create a file named `ambari-config.py`. Write code in `ambari-config.py` to configure the database IP address, port, username, password, JDK installation path, database type, MySQL driver package path, and driver package symbolic link path required for the big data management software installation.

[0026] Add a MySQL option to the ambari-server file in the $ambari-server / sbin directory, and call the script file install.sh in the $ambari-server / resources / wdbdp / mysql directory and the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the MySQL option to install and configure database information.

[0027] Step 3: After modifying the Ambari source code, run the compilation command to compile the Ambari source code. After compilation, extract the executable binary files ambari-server-2.xy0-0.aarch64.rpm and ambari-agent-2.xy0-0.aarch64.rpm, create a Yum repository, and then configure the Yum repository for the big data management software for installation.

[0028] Step 4: Execute the command `yum install ambari-server` to install the big data management software. Since the ambari-server file in the $ambari-server / sbin directory of the Ambari source code adds a call to the script file install.sh in the $ambari-server / resources / wdbdp / jdk directory, the JDK can be automatically loaded, detected, and installed during the installation of the big data management software.

[0029] Step 5: Execute the `ambari-server mysql` command for one-click configuration, completing the MySQL database check, installation, and configuration process. This includes the following steps:

[0030] Step 501: After adding the MySQL user group and MySQL user to the host, check if the installation directory / usr / local / mysql, the data directory / mysqldata / mysql_$PORT, and the log directory / mysqldata / binlog_3306 exist. If they do not exist, create them automatically.

[0031] Step 502: Delete the MariaDB database that comes with the domestic operating system to prevent file overwriting with the MySQL database that will be installed, which would cause a conflict that would prevent MySQL from working properly.

[0032] Step 503: Install the libaio library, set permissions for the installation directory, data directory, and log directory, and extract the mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz file under the $ambari-server / resources / wdbdp / mysql directory in the Ambari source code to complete the installation of the MySQL database;

[0033] Step 504: Configure the / etc / my.cnf file, start the MySQL service, modify the default password and access permissions of the MySQL database, and add permission for remote machines to access the MySQL database.

[0034] Step 505: Call the Ambari-DDL-MySQL-CREATE.sql file located in the $ambari-server / resources / wdbdp / mysql / init_script directory in the Ambari source code to initialize the MySQL database, tables, and data information required during the installation and configuration of the big data management software;

[0035] Step 506: Use the Config class in the Ambari source code package to obtain the default configuration of the big data management software. Then, use the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the Ambari source code to overwrite the default configuration with the user-defined configuration and generate a new configuration file, which is output to the system path / etc / ambari-server / conf / ambari.properties. At the same time, based on the MySQL database driver package path in the user-defined configuration, copy the MySQL database driver package under that path to the system's / usr / share / java directory, and generate a symbolic link for the driver package based on the symbolic link path in the configuration, so that the big data management software can be correctly recognized during the initialization process.

[0036] Step 6: After the big data management software is running, place the prepared third-party plugins into the big data management software's loading directory $ambari-server / resources / stacks / HDP / xy / services, and restart the big data management service. It will then be able to automatically recognize third-party application services and custom application services, and perform service installation, monitoring, and management.

[0037] Preferably, the script files params.py, yum_manager.py, script.py, os_check.py, and system.py for the management service are located in the following directories:

[0038] $ambari-server / resources / stack-hooks / before-INSTALL / scripts;

[0039] $ambari-server / lib / ambari_commons / repo_manager;

[0040] $ambari-server / lib / resource_management / libraries / script;

[0041] $ambari-server / lib / ambari_commons;

[0042] $ambari-server / lib / resource_management / core / providers;

[0043] The script files for the proxy service, params.py, yum_manager.py, script.py, os_check.py, and system.py, are located in the following directories:

[0044] $ambari-agent / resources / stack-hooks / before-INSTALL / scripts;

[0045] $ambari-agent / lib / ambari_commons / repo_manager;

[0046] $ambari-agent / lib / resource_management / libraries / script;

[0047] $ambari-agent / lib / ambari_commons;

[0048] $ambari-agent / lib / resource_management / core / providers.

[0049] Preferably, the script files os_check.py and os_utils.py for managing the service are located in the following directories:

[0050] $ambari-server / lib / ambari_commons;

[0051] $ambari-server / lib / ambari_commons;

[0052] The script files os_check.py and os_utils.py for managing the proxy service are located in the following directories:

[0053] $ambari-agent / lib / ambari_commons;

[0054] $ambari-agent / lib / ambari_commons.

[0055] Preferably, in step 6, the third-party plugin is an executable script, configuration file, installation package, or template, which automatically handles the distribution, installation, monitoring, and service dependencies of third-party big data application services through script code.

[0056] The method provided by this invention allows for the installation of big data management software on machines running domestic operating systems (such as Kylin), enabling one-click installation and configuration. Furthermore, this invention adds various open-source third-party application services and custom application services, allowing those skilled in the art to install other big data application services besides those included in the suite, according to actual project needs. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the overall compilation process of the present invention.

[0058] Figure 2 One-click configuration flowchart for big data management software;

[0059] Figure 3 Flowchart for loading services for third-party / custom applications. Detailed Implementation

[0060] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0061] The big data management and deployment method for domestically produced operating systems provided by this invention is based on the following technical terms known to those skilled in the art:

[0062] Ambari: A popular system software for managing and deploying big data application services.

[0063] Management service: This service is used to manage and deploy big data application services. It allows you to monitor the running status of each big data application service, its service status, and the system resources consumed by each service.

[0064] Agent service: A service used to report messages to the management service and execute commands issued by the management service.

[0065] Big data application services: Execution services with specific functions, such as storage services, computing services, task scheduling services, coordination services, etc.

[0066] A Yum repository is a storage repository description for resources accessed by a web application server. It describes the storage path of the resource files to be accessed, the access priority of the resource files, and their index names. After configuring a Yum repository, you can use the command `yum list xxx` to search for relevant resource files and perform corresponding operations on the searched resource files. For example, to check if the Java runtime environment package exists in the repository, you can use the command `yum list jdk`. After finding it, you can use the command `yum install jdk.aarch64` to install the JDK software.

[0067] This embodiment uses the Kylin operating system as an example to further illustrate the present invention:

[0068] Before implementing this invention, a machine with the Kylin operating system installed (referred to as the "host" in this invention) is required. The Ambari source code is downloaded on this machine, and the dependent environment required for compiling the Ambari source code is installed.

[0069] This embodiment discloses a big data management deployment method applied to a domestically developed operating system, including Kylin operating system identification, version identification, Yum source identification, and the creation of a big data management software package, specifically including the following steps:

[0070] The first step is to modify the Ambari source code to enable one-click installation and configuration of big data management software on machines running the Kylin operating system, including the following:

[0071] Modify the installation script and configuration files:

[0072] Add the description information of the Kylin system to the configuration file os_family.json in the $ambari-server / lib / ambari_commons / resources directory;

[0073] Modify the script file os_check.py in the $ambari-server / lib / ambari_commons directory to enable the script file os_check.py to recognize the Kylin operating system and its version information, ensuring a smooth installation.

[0074] Modify the script files for the big data management service and the agent service. The management service script files include:

[0075] The management service script file params.py is located in the $ambari-server / resources / stack-hooks / before-INSTALL / scripts directory;

[0076] The management service script file yum_manager.py is located in the $ambari-server / lib / ambari_commons / repo_manager directory;

[0077] The management service script file script.py is located in the $ambari-server / lib / resource_management / libraries / script directory;

[0078] The management service script file os_check.py is located in the $ambari-server / lib / ambari_commons directory;

[0079] The management service script file system.py is located in the $ambari-server / lib / resource_management / core / providers directory;

[0080] The proxy service script files include:

[0081] The proxy service script file params.py is located in the directory $ambari-agent / resources / stack-hooks / before-INSTALL / scripts;

[0082] The agent service script file yum_manager.py is located in the $ambari-agent / lib / ambari_commons / repo_manager directory;

[0083] The proxy service script file script.py is located in the $ambari-agent / lib / resource_management / libraries / script directory;

[0084] The agent service script file os_check.py is located in the $ambari-agent / lib / ambari_commons directory;

[0085] The proxy service script file system.py is located in the $ambari-agent / lib / resource_management / core / providers directory;

[0086] Modify the management service script file params.py and the proxy service script file params.py to add logic code to obtain the base_url path of the repo file. This allows the script that installs the big data management software to recognize the Yum source path and URL information during the installation process, which is then used for the installation of application service components.

[0087] Modify the `yum_check_package_available` method in the management service script file `yum_manager.py` and the agent service script file `yum_manager.py` to obtain the `hdp-select` version by reading the global configuration of the big data management software, and use it to verify whether the big data management software installation package has been installed;

[0088] Modify the STACK_VERSION_PLACEHOLDER parameter in the management service script file script.py and the agent service script file script.py to obtain the HDP version number information from the global configuration of the big data management software, so that the subsequent execution scripts can recognize the HDP version information;

[0089] Modify the logic of the os_distribution method in the management service script file os_check.py and the agent service script file os_check.py to read the corresponding operating system and operating system version information from the global configuration of the big data management software, so that the big data management software can identify the Kylin operating system;

[0090] Modify the `action_create` method in the management service script file `system.py` and the agent service script file `system.py` to determine whether the contents of the `repo` configuration file under the Yum repository are the same as the configuration file loaded when the big data management software starts. If they are different, the new Yum repository configuration file will overwrite the old configuration file.

[0091] After modifying the script files for the above big data management and agent services, ensure that the big data management software can execute correctly, and then proceed to the next installation step;

[0092] Modify the *.pom description file and *.py executable file in the Ambari source code package, manually replacing the keywords amd64 and noarch that are recognized by x86 systems with the keyword aarch64 that is recognized by Arm architecture systems;

[0093] Modify the script files for the management service and the agent service. The script files for the management service include:

[0094] The script file os_check.py is located in the $ambari-server / lib / ambari_commons directory;

[0095] The script file os_utils.py is located in the $ambari-server / lib / ambari_commons directory;

[0096] The script files for managing the agent service include:

[0097] The script file os_check.py is located in the $ambari-agent / lib / ambari_commons directory;

[0098] The script file os_utils.py is located in the $ambari-agent / lib / ambari_commons directory;

[0099] Modify the configuration files of the two script files os_utils.py, and modify the ambari_repo_file attribute in the get_ambari_repo_file_full_name method to specify that it should be obtained from the global configuration of the big data management software, so as to correctly identify the configuration path of the Yum source corresponding to the big data management software.

[0100] Modify the operatingSystem attribute of the _get_os_type method in the os_check.py script file to set it to centos, and modify the type attribute of the _get_os_version method to set it to 7, so that it can be recognized by the big data management software installation script;

[0101] To enable automatic JDK detection and installation, the following modifications need to be made to the Ambari source code:

[0102] Create a new directory $ambari-server / resources / wdbdp / jdk. In the newly created $ambari-server / resources / wdbdp / jdk directory, add a script file named install.sh. Write the following logic in the install.sh script: check if JDK is already installed on the host. If not, automatically call the command yum install oracle-j2sdk1.8 to install JDK.

[0103] Modify the ambari-server file in the $ambari-server / sbin directory, and add a call to the install.sh script file in the $ambari-server / resources / wdbdp / jdk directory;

[0104] After making the above modifications to the Ambari source code, the JDK will be automatically loaded, detected, and installed during the installation of big data management software.

[0105] To enable automatic installation and configuration of MySQL, the following modifications need to be made to the Ambari source code:

[0106] Create a new directory $ambari-server / resources / wdbdp / mysql. In this new directory, add the script file install.sh and the file mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz. Add logic to the install.sh script to check and install MySQL. If MySQL is already installed on the host, skip the installation steps; otherwise, install MySQL.

[0107] Create a new directory $ambari-server / resources / wdbdp / mysql / init_script, and add an Ambari-DDL-MySQL-CREATE.sql file to the newly created $ambari-server / resources / wdbdp / mysql / init_script directory;

[0108] Create a new directory `$ambari-server / src / main / python / wdb_dm`. In this new directory, create a file named `ambari-config.py`. Write code in `ambari-config.py` to configure the database IP address, port, username, password, JDK installation path, database type, MySQL driver package path, and driver package symbolic link path required for the big data management software installation.

[0109] Add a MySQL option to the ambari-server file in the $ambari-server / sbin directory, and call the install.sh script in the $ambari-server / resources / wdbdp / mysql directory and the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the MySQL option to install and configure the database information.

[0110] The second step is to modify the Ambari source code and then run the compilation command to compile the Ambari source code. After compilation, extract the executable binary files ambari-server-2.xy0-0.aarch64.rpm and ambari-agent-2.xy0-0.aarch64.rpm, create a Yum repository, and then configure the Yum repository for the big data management software for installation.

[0111] The third step is to execute the command `yum install ambari-server` to install the big data management software. Because the ambari-server file in the $ambari-server / sbin directory of the Ambari source code adds a call to the script file install.sh in the $ambari-server / resources / wdbdp / jdk directory, the JDK can be automatically loaded, detected, and installed during the installation of the big data management software.

[0112] Step 4: Execute the `ambari-server mysql` command for one-click configuration, completing the MySQL database check, installation, and configuration process. This includes the following steps:

[0113] Step 1: After adding the MySQL user group and MySQL user to the host, check if the installation directory / usr / local / mysql, the data directory / mysqldata / mysql_$PORT, and the log directory / mysqldata / binlog_3306 exist. If they do not exist, create them automatically.

[0114] Step 2: Delete the MariaDB database that comes with the Kylin operating system to prevent file overwriting with the MySQL database that will be installed, which could cause MySQL to malfunction.

[0115] Step 3: Install the libaio library, set permissions for the installation directory, data directory, and log directory, and extract the mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz file in the $ambari-server / resources / wdbdp / mysql directory of the Ambari source code to complete the installation of the MySQL database;

[0116] Step 4: Configure the / etc / my.cnf file, start the MySQL service, modify the default password and access permissions of the MySQL database, and add permission for remote machines to access the MySQL database.

[0117] Step 5: Call the Ambari-DDL-MySQL-CREATE.sql file located in the $ambari-server / resources / wdbdp / mysql / init_script directory in the Ambari source code to initialize the MySQL database, tables, and data information required during the installation and configuration of the big data management software;

[0118] Step 6: Use the Config class in the Ambari source code package to obtain the default configuration of the big data management software. Then, use the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the Ambari source code to overwrite the default configuration with the user-defined configuration and generate a new configuration file, which is output to the system path / etc / ambari-server / conf / ambari.properties. At the same time, based on the MySQL database driver package path in the user-defined configuration, copy the MySQL database driver package under that path to the system's / usr / share / java directory, and generate a symbolic link for the driver package based on the symbolic link path in the configuration, so that the big data management software can be correctly recognized during the initialization process.

[0119] Step 5: After the big data management software is running, place the prepared third-party plugins into the big data management software's loading directory "$ambari-server / resources / stacks / HDP / xy / services", restart the big data management service, and it will automatically recognize third-party application services and custom application services, and can install, monitor and manage the services.

[0120] In step five, third-party plugins typically consist of executable scripts [*.py or *.sh], configuration files, installation packages, and templates. These scripts automatically handle the distribution, installation, monitoring, and service dependencies of third-party big data application services. Using step five, various open-source third-party application services and custom application services can be added, allowing for the installation of big data application services beyond those included in the suite, based on actual project needs.

Claims

1. A method for big data management and deployment applied to a domestically developed operating system, characterized in that, Includes the following steps: Step 1: Download the Ambari source code on a host machine with a domestic operating system installed, and install the dependency environment required for compiling the Ambari source code at the same time; Step 2: Modify the Ambari source code, including the following: Modify the installation script and configuration files: Add description information about the domestic operating system to the configuration file os_family.json in the $ambari-server / lib / ambari_commons / resources directory; Modify the script file os_check.py in the $ambari-server / lib / ambari_commons directory to enable the script file os_check.py to recognize the domestic operating system and its version information, ensuring a smooth installation. Modify the script files params.py, yum_manager.py, script.py, os_check.py, and system.py for the big data management service and agent service, including: Modify the script file params.py to add logic code to obtain the base_url path of the repo file, so that the script for installing big data management software can recognize the Yum source path and URL information during the installation process, which is used for the installation of application service components; Modify the `yum_check_package_available` method in the script file `yum_manager.py` to obtain the `hdp-select` version by reading the global configuration of the big data management software, and use it to verify whether the big data management software installation package has been installed; Modify the STACK_VERSION_PLACEHOLDER parameter in the script file script.py to obtain the HDP version number information from the global configuration of the big data management software, so that the subsequent execution script can recognize the HDP version information; Modify the logic of the os_distribution method in the script file os_check.py to read the corresponding operating system and operating system version information from the global configuration of the big data management software, so that the big data management software can identify the Kylin operating system; Modify the `action_create` method in the script file `system.py` to determine whether the contents of the repo configuration file under the Yum repository are the same as the configuration file loaded when the big data management software starts. If they are different, the new Yum repository configuration file will overwrite the old configuration file. Modify the *.pom description file and *.py executable file in the Ambari source code package, manually replacing the keywords amd64 and noarch that are recognized by x86 systems with the keyword aarch64 that is recognized by Arm architecture systems; Modify the script files os_check.py and os_utils.py for the management service and proxy service: Modify the configuration file of the script file os_check.py, and modify the ambari_repo_file attribute in the get_ambari_repo_file_full_name method to specify that it should be obtained from the global configuration of the big data management software, so as to correctly identify the configuration path of the Yum source corresponding to the big data management software; Modify the operatingSystem attribute of the _get_os_type method in the two script files os_utils.py to set it to centos, and modify the type attribute of the _get_os_version method to set it to 7, so that it can be recognized by the big data management software installation script; Create a new directory $ambari-server / resources / wdbdp / jdk. In the newly created $ambari-server / resources / wdbdp / jdk directory, add a script file named install.sh. Write the following logic in the install.sh script: check if JDK is already installed on the host. If not, automatically call the command yum install oracle-j2sdk1.8 to install JDK. Modify the ambari-server file in the $ambari-server / sbin directory, and add a call to the install.sh script file in the $ambari-server / resources / wdbdp / jdk directory; Create a new directory $ambari-server / resources / wdbdp / mysql. In this new directory, add the script file install.sh and the file mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz. Add logic to the install.sh script to check and install MySQL. If MySQL is already installed on the host, skip the installation steps; otherwise, install MySQL. Create a new directory $ambari-server / resources / wdbdp / mysql / init_script, and add an Ambari-DDL-MySQL-CREATE.sql file to the newly created $ambari-server / resources / wdbdp / mysql / init_script directory; Create a new directory `$ambari-server / src / main / python / wdb_dm`. In this new directory, create a file named `ambari-config.py`. Write code in `ambari-config.py` to configure the database IP address, port, username, password, JDK installation path, database type, MySQL driver package path, and driver package symbolic link path required for the big data management software installation. Add a MySQL option to the ambari-server file in the $ambari-server / sbin directory, and call the script file install.sh in the $ambari-server / resources / wdbdp / mysql directory and the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the MySQL option to install and configure database information. Step 3: After modifying the Ambari source code, run the compilation command to compile the Ambari source code. After compilation, extract the executable binary files ambari-server-2.xy0-0.aarch64.rpm and ambari-agent-2.xy0-0.aarch64.rpm, create a Yum repository, and then configure the Yum repository for the big data management software for installation. Step 4: Execute the command `yum install ambari-server` to install the big data management software. Since the ambari-server file in the $ambari-server / sbin directory of the Ambari source code adds a call to the script file install.sh in the $ambari-server / resources / wdbdp / jdk directory, the JDK can be automatically loaded, detected, and installed during the installation of the big data management software. Step 5: Execute the `ambari-server mysql` command for one-click configuration, completing the MySQL database check, installation, and configuration process. This includes the following steps: Step 501: After adding the MySQL user group and MySQL user to the host, check if the installation directory / usr / local / mysql, the data directory / mysqldata / mysql_$PORT, and the log directory / mysqldata / binlog_3306 exist. If they do not exist, create them automatically. Step 502: Delete the MariaDB database that comes with the domestic operating system to prevent file overwriting with the MySQL database that will be installed, which would cause a conflict that would prevent MySQL from working properly. Step 503: Install the libaio library, set permissions for the installation directory, data directory, and log directory, and extract the mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz file under the $ambari-server / resources / wdbdp / mysql directory in the Ambari source code to complete the installation of the MySQL database; Step 504: Configure the / etc / my.cnf file, start the MySQL service, modify the default password and access permissions of the MySQL database, and add permission for remote machines to access the MySQL database. Step 505: Call the Ambari-DDL-MySQL-CREATE.sql file located in the $ambari-server / resources / wdbdp / mysql / init_script directory in the Ambari source code to initialize the MySQL database, tables, and data information required during the installation and configuration of the big data management software; Step 506: Use the Config class in the Ambari source code package to obtain the default configuration of the big data management software. Then, use the ambari-config.py file in the $ambari-server / src / main / python / wdb_dm directory in the Ambari source code to overwrite the default configuration with the user-defined configuration and generate a new configuration file, which is output to the system path / etc / ambari-server / conf / ambari.properties. At the same time, based on the MySQL database driver package path in the user-defined configuration, copy the MySQL database driver package under that path to the system's / usr / share / java directory, and generate a symbolic link for the driver package based on the symbolic link path in the configuration, so that the big data management software can be correctly recognized during the initialization process. Step 6: After the big data management software is running, place the prepared third-party plugins into the big data management software's loading directory $ambari-server / resources / stacks / HDP / xy / services, and restart the big data management service. It will then be able to automatically recognize third-party application services and custom application services, and perform service installation, monitoring, and management.

2. The big data management and deployment method applied to a domestically developed operating system as described in claim 1, characterized in that, The script files for the management service, params.py, yum_manager.py, script.py, os_check.py, and system.py, are located in the following directories: $ambari-server / resources / stack-hooks / before-INSTALL / scripts; $ambari-server / lib / ambari_commons / repo_manager; $ambari-server / lib / resource_management / libraries / script; $ambari-server / lib / ambari_commons; $ambari-server / lib / resource_management / core / providers; The script files for the proxy service, params.py, yum_manager.py, script.py, os_check.py, and system.py, are located in the following directories: $ambari-agent / resources / stack-hooks / before-INSTALL / scripts; $ambari-agent / lib / ambari_commons / repo_manager; $ambari-agent / lib / resource_management / libraries / script; $ambari-agent / lib / ambari_commons; $ambari-agent / lib / resource_management / core / providers.

3. The big data management and deployment method applied to a domestically developed operating system as described in claim 1, characterized in that, The script files os_check.py and os_utils.py for managing the services are located in the following directories: $ambari-server / lib / ambari_commons; $ambari-server / lib / ambari_commons; The script files os_check.py and os_utils.py for managing the proxy service are located in the following directories: $ambari-agent / lib / ambari_commons; $ambari-agent / lib / ambari_commons.

4. The big data management and deployment method applied to a domestically developed operating system as described in claim 1, characterized in that, In step 6, the third-party plugin is an executable script, configuration file, installation package, or template. The script code automatically handles the distribution, installation, monitoring, and service dependencies of third-party big data application services.

Images