Big data management tool implementation method and device

Abstract

The invention relates to the field of big data, and particularly provides a big data management tool implementation method and device, and the method comprises the following steps: S1, a Restserver provides a unified standardized interface; s2, further packaging the Restserver interface by the management end; s3, the distributed database adopts a clustering deployment mode; s4, the source management service is a minimum unit supporting a big data management tool, is developed according to a unified specification, runs in a micro-service form, and is deployed in a plug-in manner; s5, the package management tool is responsible for managing the installation package of the source management service; and S6, the authentication service is responsible for managing service authentication and data source account management of each component. Compared with the prior art, the method can solve the problem that management and access of multi-source and multi-version big data components by big data platform administrators and big data developers are tedious and non-uniform.

Description

Technical Field

[0001] The present invention relates to the field of big data, and specifically provides a method and device for implementing a big data management tool. Background Art

[0002] With the development of informatization, all industries and fields are actively building big data platforms. There is an urgent need to manage various data sources in their new and old information systems or connect them to big data platforms for data governance in order to tap the value of data.

[0003] There are many ways to store data in information systems, and the storage components are diverse, which can be divided into big data type, relational type, non-relational type, etc. The versions of the data components used are also different. Often, the new version of the driver no longer supports the old big data components, or each driver only supports a specific version of the big data components. This brings obstacles to the management of data sources of various information systems and access to the big data platform. It brings difficulties to big data managers and developers. The current big data management software has more or less the following problems.

[0004] 1. The big data management entrance is not unified, requiring a large workload for secondary development, and the interface is not unified;

[0005] 2. It only supports the management of data components of specific versions. Due to driver conflicts, it cannot support the simultaneous management of multiple versions of the same data source, making it impossible to uniformly manage data stored in old versions of big data components.

[0006] 3. It only supports the management of a few types of data components, and has no flexible deployment or plug-in deployment capabilities, making it impossible to effectively manage niche data components.

[0007] 4. No load balancing or high availability. Management tools are prone to failure and cannot effectively manage big data components. Summary of the invention

[0008] The present invention aims to overcome the deficiencies of the prior art and provides a method for implementing a big data management tool with strong practicality.

[0009] A further technical task of the present invention is to provide a big data management tool implementation device that is reasonably designed, safe and applicable.

[0010] The technical solution adopted by the present invention to solve its technical problem is:

[0011] A method for implementing a big data management tool comprises the following steps:

[0012] S1. Rest-server provides a unified and standardized interface;

[0013] S2. The management end further encapsulates the Rest-server interface;

[0014] S3,distributed database adopts clustered deployment mode;

[0015] S4, source management service is the smallest unit supporting big data management tools. It is developed according to unified specifications, runs in the form of microservices, and adopts plug-in deployment;

[0016] S5. The package management tool is responsible for managing the installation packages of the source management service;

[0017] S6. Authentication service is responsible for managing the service authentication of each component and data source account management.

[0018] Furthermore, in step S1, it includes:

[0019] The package management interface is responsible for uploading, downloading, deleting, enabling, and disabling source management service packages;

[0020] Service management interface, responsible for configuring, deploying, starting, stopping, migrating, and uninstalling source management services and log interfaces;

[0021] The data source interface provides services for adding, deleting, modifying, checking and managing data sources.

[0022] Furthermore, in step S2, the management end further encapsulates the Rest-server interface and provides it to customers in the form of UI interface, REST, CLI command line and SDK.

[0023] Furthermore, in step S3, the distributed database adopts a clustered deployment method to record the management service package list, service status, service location in the cluster, data source type, version, configuration, status and component authentication authority information.

[0024] Further, in step S4, the source management service uses the rest encapsulation interface and accepts unified management from the management end, including the rest-api, data source driver and connection pool;

[0025] The rest-api interface includes: configuration management, log interface, add, delete, modify and query interface, and interface list.

[0026] Furthermore, in step S5, the source management service is uploaded to the package management through the management end WEB interface, and the source management service is deployed, uninstalled, started, stopped and migrated on the management end.

[0027] Furthermore, in step S6, the deployment mode of the source management service is deployed in a clustered manner, and the provided rest-server interface includes: upload, download, delete, enable, and disable.

[0028] A big data management tool implementation device includes: at least one memory and at least one processor;

[0029] The at least one memory is used to store a machine-readable program;

[0030] The at least one processor is used to call the machine-readable program to execute a method for implementing a big data management tool.

[0031] Compared with the prior art, the big data management tool implementation method and device of the present invention have the following outstanding beneficial effects:

[0032] The present invention uses microservices, process isolation, high availability, plug-in, web management, unified interface, unified interactive portal and other means to integrate big data management tools with high reliability, high cohesion and weak coupling. It supports unified management and traceable configuration, and provides access capabilities based on the actual needs of information construction. It solves the problem that big data platform administrators and big data developers are cumbersome and inconsistent in managing and accessing multi-source and multi-version big data components. It pays more attention to interactive query and actual extraction of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Attached Figure 1 It is a schematic diagram of the framework of the implementation method of the big data management tool;

[0035] Attached Figure 2 It is a flowchart of a method for implementing a big data management tool. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] A best embodiment is given below:

[0038] like Figure 1 , 2 As shown, a method for implementing a big data management tool in this embodiment has the following steps:

[0039] S1. Rest-server provides a unified and standardized interface;

[0040] include:

[0041] The package management interface is responsible for uploading, downloading, deleting, enabling, and disabling source management service packages;

[0042] Service management interface, responsible for configuring, deploying, starting, stopping, migrating, and uninstalling source management services and log interfaces;

[0043] The data source interface provides services for adding, deleting, modifying, checking and managing data sources.

[0044] S2. The management end further encapsulates the Rest-server interface;

[0045] The management end further encapsulates the Rest-server interface and provides it to customers in the form of UI interface, REST, CLI (command line), SDK, etc.

[0046] S3,distributed database adopts clustered deployment mode;

[0047] The distributed database adopts a clustered deployment method to record and manage information such as the service package list, service status, service location in the cluster, data source type, version, configuration, status, component authentication permissions, etc.

[0048] S4, source management service is the smallest unit supporting big data management tools. It is developed according to unified specifications, runs in the form of microservices, and adopts plug-in deployment;

[0049] The source management service uses the REST encapsulation interface and is managed by the management end. It includes REST-API, data source driver, and connection pool.

[0050] The rest-api interface includes: configuration management (configuration query, configuration modification), log interface, add, delete, modify and query interface and interface list.

[0051] Rest-server example (Post):

[0052] http(s): / / {basePath} / {datasource_type} / {datasource_vers ion} / apiList.

[0053] S5. The package management tool is responsible for managing the installation packages of the source management service;

[0054] It can be uploaded to the package management through the management end WEB interface. On the management end, you can deploy, uninstall, start, stop and migrate the source management service. The deployment mode of the source management service adopts high availability and clustering deployment. The rest-server interfaces provided include: upload, download, delete, enable and disable.

[0055] Rest-server example (Post):

[0056] http(s): / / {basePath} / package_manage / {upload\download\re move\enable\disable\}.

[0057] S6. Authentication service is responsible for managing service authentication of each component and data source account management;

[0058] Rest-server example (Post):

[0059] http(s): / / {basePath} / authentication / getAccount.

[0060] Based on the above method, a big data management tool implementation device in this embodiment includes: at least one memory and at least one processor;

[0061] The at least one memory is used to store a machine-readable program;

[0062] The at least one processor is used to call the machine-readable program to execute a method for implementing a big data management tool.

[0063] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing a big data management tool, characterized in that: The steps are as follows: S1. Rest-server provides a unified and standardized interface; S2. The management end further encapsulates the Rest-server interface; S3,distributed database adopts clustered deployment mode; S4, source management service is the smallest unit supporting big data management tools. It is developed according to unified specifications, runs in the form of microservices, and adopts plug-in deployment; S5. The package management tool is responsible for managing the installation packages of the source management service; S6. Authentication service is responsible for managing the service authentication of each component and data source account management.

2. A method for implementing a big data management tool according to claim 1, characterized in that: In step S1, it includes: The package management interface is responsible for uploading, downloading, deleting, enabling, and disabling source management service packages; Service management interface, responsible for configuring, deploying, starting, stopping, migrating, and uninstalling source management services and log interfaces; The data source interface provides services for adding, deleting, modifying, checking and managing data sources.

3. A method for implementing a big data management tool according to claim 2, characterized in that: In step S2, the management end further encapsulates the Rest-server interface and provides it to customers in the form of UI interface, REST, CLI command line and SDK.

4. A method for implementing a big data management tool according to claim 3, characterized in that: In step S3, the distributed database adopts a clustered deployment method to record the management service package list, service status, service location in the cluster, data source type, version, configuration, status and component authentication authority information.

5. A method for implementing a big data management tool according to claim 4, characterized in that: In step S4, the source management service uses the rest encapsulation interface and is managed uniformly by the management end, including the rest-api, data source driver, and connection pool; The rest-api interface includes: configuration management, log interface, add, delete, modify and query interface, and interface list.

6. A method for implementing a big data management tool according to claim 5, characterized in that: In step S5, the source management service is deployed, uninstalled, started, stopped and migrated through the management end WEB interface.

7. A method for implementing a big data management tool according to claim 5, characterized in that: In step S6, the deployment mode of the source management service is clustered, and the provided rest-server interfaces include: upload, download, delete, enable, and disable.

8. A device for implementing a big data management tool, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images