A cross-platform big data resource management system and method

Abstract

The present invention provides a cross-platform big data resource management system and method, wherein the method comprises: an application interface module, which is used to receive a resource request sent by an application system, and when judging that the resource request is legal, sends the resource request to a resource management module; a resource management module, which is used to select a resource management sub-template for receiving a resource request according to the type of the resource request, and sends the resource request to a cluster interface module; a cluster interface module, which is used to forward the resource request to a corresponding Hadoop cluster according to the type of the resource request, and sends the resource request to a resource monitoring module. The present invention centrally and uniformly manages the resources of a Hadoop cluster or other open source software, improves the work efficiency of resource management, and reduces the operating cost of managing multiple clusters at the same time; realizes visual management of information such as users and tenants, simplifies the operation steps of resource management of a Hadoop cluster or other open source software, and reduces the risk of misoperation caused by manually executing commands.

Description

Technical Field

[0001] The present invention relates to the technical field of information management systems, and in particular to a cross-platform big data resource management system and method. Background Art

[0002] At present, the resources of most big data platform Hadoop (distributed system infrastructure) clusters are managed based on tenants of YARN (Hadoop resource management system). Open source Hadoop products require manual modification of configuration files to manage tenants, while commercial Hadoop products from companies such as Huawei and Cloudera can manage tenants through page-based configuration tools. Due to different purposes such as management methods or business scenarios, an organization usually chooses multiple Hadoop products and deploys multiple Hadoop clusters.

[0003] Since the resources of each cluster are managed independently, the operation steps of resource management of different Hadoop products are different, and the resource configuration content of different clusters of the same Hadoop product is different. In addition, in addition to using Hadoop clusters, most organizations also use software such as Elasticsearch, Redis, SBD, etc. These software are often not included in commercial big data products, and the resources of these software cannot be managed centrally. When the number of managed clusters reaches a certain scale, the resource management of multiple clusters will become very complex and cumbersome, and prone to errors, which greatly affects work efficiency. Summary of the invention

[0004] The present invention provides a cross-platform big data resource management system and method, which is compatible with different Hadoop products and other open source software, manages the resources of multiple clusters at the same time, and intuitively manages the resources of multiple clusters through page-based operations, thereby improving work efficiency and reducing cluster resource management costs.

[0005] An embodiment of the present invention provides a cross-platform big data resource management system, comprising: an application system interface module, a resource management module, a resource monitoring module, a cluster interface module, and a data storage module; wherein;

[0006] The application system interface module is used to receive a resource request sent by the application system, and when judging that the resource request is legal, send the resource request to the resource management module; wherein the resource request includes: a request to add a tenant, a request for authorization of a data directory or a database table, a request to add a cluster user, and a request to add a cluster;

[0007] The resource management module is used to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module; wherein the resource management sub-template includes: a tenant management sub-template, an authorization management sub-template, a user management sub-template, and a cluster management sub-template;

[0008] The cluster interface module is used to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module;

[0009] The resource monitoring module is used to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule, a tenant monitoring submodule, an application monitoring submodule, and a library table monitoring submodule;

[0010] The data storage module is used to store the usage data of the Hadoop cluster.

[0011] Furthermore, the cross-platform big data resource management system also includes: a rule engine module, which is used to provide resource request legitimacy judgment rules to the application interface module to judge the legitimacy of the resource request; and is also used to provide resource request type judgment rules to the resource management module and resource monitoring module to judge the type of the resource request.

[0012] Furthermore, the tenant management sub-template is used to receive the request for adding a new tenant and add the configuration information of the tenant in the tenant management;

[0013] The authorization management sub-template is used to receive the data directory or database table authorization request, query the authorized target in the tree structure in the authorization management, and allocate the read, write, execute and inherit permissions in a multiple-selection manner;

[0014] The user management sub-template is used to receive the request for adding a new cluster user and add the user's configuration information in the user management; wherein the configuration information includes: user name, user type, role, and user group information;

[0015] The cluster management sub-template is used to receive the new cluster request and add cluster configuration information in cluster management, wherein the configuration information includes: cluster name, cluster category, IP address, port, user name, and password information.

[0016] Furthermore, the cluster monitoring submodule is used to obtain and monitor the average CPU information, memory information, network IO information and storage space information of the entire cluster;

[0017] The tenant monitoring submodule is used to obtain and monitor the CPU information and storage space usage information of the tenant;

[0018] The application monitoring submodule is used to obtain and monitor the CPU and content usage information and running time information submitted to the application;

[0019] The library table monitoring submodule is used to obtain and monitor the access information of the database and data tables.

[0020] Furthermore, the cross-platform big data resource management system also includes: a platform management module, which is used to manage the management configuration of the platform; wherein the management configuration includes users, organizations, permissions, audits, parameter configurations and access management.

[0021] Furthermore, the cross-platform big data resource management system further includes: a scheduling engine module, which is used to manage the resource monitoring module to obtain monitoring information.

[0022] Furthermore, the cross-platform big data resource management system also includes: a WEB service module for user interaction.

[0023] An embodiment of the present invention further provides a cross-platform big data resource management method, comprising:

[0024] The control application system interface module receives the resource request sent by the application system, and when judging that the resource request is legal, sends the resource request to the resource management module; wherein the resource request includes: a request to add a tenant, a request for authorization of a data directory or a database table, a request to add a cluster user, and a request to add a cluster;

[0025] Control the resource management module to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module; wherein the resource management sub-template includes: a tenant management sub-template, an authorization management sub-template, a user management sub-template, and a cluster management sub-template;

[0026] Control the cluster interface module to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module;

[0027] The resource monitoring module is controlled to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule, a tenant monitoring submodule, an application monitoring submodule, and a library table monitoring submodule.

[0028] Furthermore, the cross-platform big data resource management method also includes: a control rule engine module provides resource request legitimacy judgment rules to the application interface module to judge the legitimacy of the resource request, and provides resource request type judgment rules to the resource management module and resource monitoring module to judge the type of the resource request.

[0029] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the cross-platform big data resource management method.

[0030] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0031] The present invention provides a cross-platform big data resource management system, comprising: an application system interface, a resource management module, a resource monitoring module, a cluster interface module, and a data storage module; wherein; the application interface module is used to receive a resource request sent by an application system, and when judging that the resource request is legal, the resource request is sent to the resource management module; wherein the resource request comprises: a request for adding a tenant, a request for authorization of a data directory or a database table, a request for adding a cluster user, and a request for adding a cluster; the resource management module is used to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module; wherein the resource management sub-template comprises: a tenant management sub-template, Authorization management sub-template, user management sub-template, cluster management sub-template; the cluster interface module is used to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module; the resource monitoring module is used to select the resource monitoring sub-module according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring sub-module includes: cluster monitoring sub-module, tenant monitoring sub-module, application monitoring sub-module, library table monitoring sub-module; the data storage module is used to store the usage data of the Hadoop cluster. The present invention centrally and uniformly manages the resources of the Hadoop cluster or other open source software, improves the work efficiency of resource management, and reduces the operating cost of managing multiple clusters at the same time; realizes the visual management of information such as users and tenants, simplifies the operation steps of Hadoop cluster or other open source software resource management, and reduces the risk of misoperation caused by manual execution of commands; covers the configuration and monitoring rules of common open source software, Huawei, Cloudera and other products, has high applicability, does not require secondary development, and saves manpower and time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is a structural diagram of a cross-platform big data resource management system provided by a certain embodiment of the present invention;

[0034] Figure 2 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0035] Figure 3 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0036] Figure 4 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0037] Figure 5 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0038] Figure 6 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0039] Figure 7 is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0040] Figure 8 It is a structural diagram of a cross-platform big data resource management system provided by another embodiment of the present invention;

[0041] Fig. 9 It is a flow chart of a cross-platform big data resource management method provided by a certain embodiment of the present invention;

[0042] Fig.10 is a flow chart of a cross-platform big data resource management method provided by another embodiment of the present invention;

[0043] Fig.11 This is a flowchart of a cross-platform big data resource management method provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0046] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0048] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0049] The first aspect.

[0050] At present, the resources of most Hadoop clusters on big data platforms are managed based on YARN tenants. Open source Hadoop products require manual modification of configuration files to manage tenants, while commercial Hadoop products from companies such as Huawei and Cloudera can manage tenants through page-based configuration tools. Due to different purposes such as management methods or business scenarios, an organization usually chooses multiple Hadoop products and deploys multiple Hadoop clusters. Since the resources of each cluster are managed independently, the operation steps of resource management of different Hadoop products are different, and the resource configuration content of different clusters of the same Hadoop product is different. In addition, in addition to using Hadoop clusters, most organizations also use software such as Elasticsearch, Redis, and SBD. These software are often not included in commercial big data products, and the resources of these software cannot be centrally managed. When the number of managed clusters reaches a certain scale, the resource management of multiple clusters will become very complex and cumbersome, and prone to errors, greatly affecting work efficiency.

[0051] See also Figure 1-8 ,An embodiment of the present invention provides a cross-platform big data resource management system, including: an application system interface 10, a resource management module 20, a resource monitoring module 30, a cluster interface module 40, and a data storage module 50; wherein;

[0052] The application interface module 10 is used to receive a resource request sent by the application system, and when it is determined that the resource request is legal, the resource request is sent to the resource management module 20; wherein the resource request includes: a request to add a tenant, a request for authorization of a data directory or a database table, a request to add a cluster user, and a request to add a cluster.

[0053] In one embodiment, if Figure 8 As shown in the figure, the application system interface is mainly responsible for the message processing between the resource management platform and each application system, which mainly includes two important interfaces: data authorization and key distribution. The data authorization interface is when the upstream application system needs to authorize certain data, the application system sends a message in a specific format to the resource management platform. After receiving the message, the resource management platform first checks whether the authorization is legal in the local configuration library, and then checks whether the authorization is legal in the Hadoop cluster or other open source software. If it is legal, it is sent to the corresponding Hadoop cluster or other open source software to execute the authorization command, and the execution result is returned to the upstream application system after the execution is completed. The key distribution interface is when the upstream application system needs to obtain the key information of a certain user, the upstream application system first sends a message requesting the key, and the resource management platform will package the key information of the user and store it locally, and return the storage location to the upstream application system. After the message times out or fails to be processed, the resource management platform will return the corresponding flag and specific reason to the upstream system for the administrator to handle. The message timeout is set in the parameter configuration in the platform management. The message header mainly determines which Hadoop cluster or other open source software it is, and the message body is the specific command and parameters to be executed.

[0054] The resource management module 20 is used to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module 40; wherein the resource management sub-template includes: a tenant management sub-template 21, an authorization management sub-template 22, a user management sub-template 23, and a cluster management sub-template 24.

[0055] In one embodiment, if Figure 8 As shown in the figure, the resource management module is the core module of the resource management platform, which is mainly responsible for cluster management, tenant management, data access authorization, and cluster user management of Hadoop and other open source software. When adding a new tenant to the cluster, it is necessary to add the tenant's configuration information in the tenant management, such as the tenant name, the maximum and minimum number of CPU cores, the maximum and minimum amount of memory, the allocated data storage directory and size, etc. When authorizing a data directory or database table, first query the authorized target in the tree structure in the authorization management, and assign read, write, execute, and inherit permissions through multiple selection. When adding a new cluster user, it is necessary to add the user's configuration information in the user management, such as user name, user type, role, user group, etc. The user type is divided into human-machine user and program user. Human-machine user requires to set a password, while program user does not require to set a password, and a secret key file is automatically generated.

[0056] In a specific implementation, the tenant management sub-template 21 is used to receive the request for adding a new tenant and add the configuration information of the tenant in the tenant management.

[0057] The authorization management sub-template 22 is used to receive the data directory or database table authorization request, query the authorized target in the tree structure in the authorization management, and allocate the read, write, execute and inherit permissions in a multiple-selection manner.

[0058] The user management sub-template 23 is used to receive the request for adding a cluster user and add the user's configuration information in the user management; wherein the configuration information includes: user name, user type, role, and user group information.

[0059] The cluster management sub-template 24 is used to receive the new cluster request and add cluster configuration information in cluster management, wherein the configuration information includes: cluster name, cluster type, IP address, port, user name, and password information.

[0060] The cluster interface module 40 is used to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module 30 .

[0061] In one embodiment, if Figure 8 As shown, the cluster interface is mainly responsible for message processing between the resource management platform and each Hadoop cluster or other open source software. It mainly includes three interfaces: tenant management, user management, and data authorization, which correspond to the three functions of the resource management module. After the resource management platform sends a message to the specified Hadoop cluster or other open source software, it waits for the message returned by the cluster. After receiving the message returned by the Hadoop cluster or other open source software, it is displayed on the platform page or forwarded to the upstream application system. When the message times out or the processing fails, the resource management platform will display the corresponding flag and specific reason on the page or forward it to the upstream system for the administrator to handle. The message timeout is set in the parameter configuration in the platform management. The message header mainly determines which Hadoop cluster or other open source software it is, and the message body is the specific command and parameters to be executed.

[0062] The resource monitoring module 30 is used to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module 50 for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule 31, a tenant monitoring submodule 32, an application monitoring submodule 33, and a library table monitoring submodule 34.

[0063] In one embodiment, if Figure 8As shown in the figure, the resource monitoring module is mainly responsible for monitoring the resource usage of the Hadoop cluster and other open source software, including cluster monitoring, tenant monitoring, application monitoring, and library table monitoring. The monitoring displays statistical information within one hour by default, and a longer time period can be selected. The resource monitoring module collects the corresponding monitoring information through the cluster interface, formats it, and saves it in the data storage. Cluster monitoring monitors the average CPU, memory, network IO, and storage space of the entire cluster. Tenant monitoring monitors the CPU and storage space usage of tenants. Application monitoring monitors the CPU and content usage, running time, and other information of the Application submitted to YARN. Library table monitoring monitors the access status, size, and other information of databases and data tables such as HIVE, HBASE, and Redis.

[0064] In a specific implementation, the cluster monitoring submodule 31 is used to obtain and monitor the average CPU information, memory information, network IO information and storage space information of the entire cluster.

[0065] The tenant monitoring submodule 32 is used to obtain and monitor the CPU information and storage space usage information of the tenant.

[0066] The application monitoring submodule 33 is used to obtain and monitor the CPU and content usage information and running time information submitted to the application.

[0067] The database table monitoring submodule 34 is used to obtain and monitor the access information of the database and data tables.

[0068] The data storage module 50 is used to store the usage data of the Hadoop cluster.

[0069] In one embodiment, if Figure 8 As shown in the figure, data storage: mainly responsible for the storage of various monitoring information of the resource management platform, which can be displayed and queried on the platform. The stored data is divided into hot data and cold data, and different types of data are stored in different components. Hot data (set the cycle according to management needs, such as one week) is stored in components with higher query performance, and cold data is stored in components with lower query performance. Expired hot data is automatically transferred to the component storing cold data every day.

[0070] In a specific embodiment, a cross-platform big data resource management system provided by the present invention also includes: a rule engine module 60, which is used to provide resource request legitimacy judgment rules to the application interface module 10 to judge the legitimacy of the resource request; and is also used to provide resource request type judgment rules to the resource management module 20 and the resource monitoring module 30 to judge the type of the resource request.

[0071] In one embodiment, if Figure 8 As shown in the figure, the rule engine is mainly responsible for maintaining resource rules of the resource management platform. Because the configuration parameters of different clusters are different and need to be managed separately, it is necessary to maintain resource configuration rules. Resource rules mainly include cluster information, tenant information, user information, and authorization information. The platform formulates different command rules according to the product type of Hadoop cluster or other open source software. For example, the tenant configuration commands of Huawei's Hadoop cluster and Cloudera's Hadoop cluster are somewhat different, so two configuration templates must be formulated. The resource management platform must be able to select different configuration templates according to different types of products, send different instructions, receive different return messages, store different monitoring information formats, and display them uniformly on the page.

[0072] In a specific embodiment, a cross-platform big data resource management system provided by the present invention also includes: a platform management module 70, which is used to manage the management configuration of the management platform; wherein the management configuration includes users, organizations, permissions, audits, parameter configurations and access management.

[0073] In one embodiment, if Figure 8 As shown in the figure, the platform management module is mainly responsible for the management and configuration of the resource management platform, including users, organizations, permissions, audits, parameter configurations and access management. User management is the management of user information on the resource management platform. After adding a new user, you can log in to the resource management platform and view or modify the relevant configurations according to the organization and permission dimensions. Organization management is the management of organizational structure information on the resource management platform. The management scope of users of different organizations is different. Accordingly, there is also corresponding organizational information in the configuration of cluster tenants, authorizations and users. Permission management is the management of platform users' permissions on the resource management platform. Users of different roles have different permissions, such as configuration administrators, operators, auditors, etc. Audit management is to query audit information on the resource management platform, including user name, operation time, and specific content includes login and logout status, user-performed operation details and other information. Parameter configuration is the resource management platform's own parameter settings, such as interface corresponding duration, data collection cycle, etc. Access management is to configure the IP addresses of upstream application systems, downstream Hadoop clusters or other open source software that are allowed to access. Interface information sent by information sources that have not been configured will be rejected.

[0074] In a specific implementation, the cross-platform big data resource management system provided by the present invention further includes: a scheduling engine module 80, which is used to manage the resource monitoring module to obtain monitoring information.

[0075] In one embodiment, if Figure 8As shown in the figure, the scheduling engine is mainly responsible for the task management of the resource management platform, including various monitoring information collection tasks of different Hadoop clusters or other open source software. The resource management platform will collect monitoring information from the Hadoop cluster or other open source software at regular intervals, and the collected information will be saved in the data storage for display and query on the platform. The information collection cycle is set in the parameter configuration of the resource management platform.

[0076] In a specific implementation, the cross-platform big data resource management system provided by the present invention further includes: a WEB service module 90 for user interaction.

[0077] In one embodiment, if Figure 8 As shown, WEB service: is the front-end page of the resource management platform, which is mainly responsible for user interaction.

[0078] The second aspect.

[0079] See also Figure 9-10 The present invention provides a cross-platform big data resource management method, comprising:

[0080] S10. Control the application interface module to receive the resource request sent by the application system, and when judging that the resource request is legal, send the resource request to the resource management module; wherein the resource request includes: a request for adding a new tenant, a request for authorization of a data directory or a database table, a request for adding a new cluster user, and a request for adding a new cluster.

[0081] S20. Control the resource management module to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module; wherein the resource management sub-template includes: a tenant management sub-template, an authorization management sub-template, a user management sub-template, and a cluster management sub-template.

[0082] S30: Control the cluster interface module to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module.

[0083] S40 controls the resource monitoring module to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and sends the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule, a tenant monitoring submodule, an application monitoring submodule, and a library table monitoring submodule.

[0084] In a specific implementation, the cross-platform big data resource management method further includes:

[0085] S50. The control rule engine module provides resource request legitimacy judgment rules to the application interface module to judge the legitimacy of the resource request, and provides resource request type judgment rules to the resource management module and resource monitoring module to judge the type of the resource request.

[0086] In one embodiment, if Fig.11 As shown, a cross-platform big data resource management method includes:

[0087] The application system sends a resource request to the application system interface of the resource management platform. The application interface determines whether the request is legal in the rule engine, and immediately terminates if an illegal request is returned. After receiving a legal request, the application interface forwards the request to the resource management module. After receiving the request, the resource management module queries the corresponding configuration template in the rule engine and converts the received message information into the message format of the corresponding cluster. The resource management module sends the request message to the cluster interface. The cluster interface sends instructions to the corresponding cluster, and the cluster returns the processing result. The cluster interface forwards the returned result to the resource management module. The resource management module writes the execution result to the data storage. The resource management module forwards the execution result to the application interface. The application interface returns the execution result to the application system. The third aspect.

[0088] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any one of the above-described cross-platform big data resource management methods when executing the computer program.

[0089] The fourth aspect.

[0090] An embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned cross-platform big data resource management methods.

Claims

1. A cross-platform big data resource management system, characterized in that: include: Application system interface module, resource management module, resource monitoring module, cluster interface module, data storage module; in; The application system interface module is used to receive a resource request sent by the application system, and when the resource request is judged to be legal, the resource request is sent to the resource management module; wherein the resource request includes: a request to add a tenant, a request for authorization of a data directory or a database table, a request to add a cluster user, and a request to add a cluster; The resource management module is used to select a resource management sub-template for receiving the resource request according to the type of the resource request, and send the resource request to the cluster interface module; wherein the resource management sub-template includes: a tenant management sub-template, an authorization management sub-template, a user management sub-template, and a cluster management sub-template; The cluster interface module is used to forward the resource request to the corresponding Hadoop cluster according to the type of the resource request, and send the resource request to the resource monitoring module; The resource monitoring module is used to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule, a tenant monitoring submodule, an application monitoring submodule, and a library table monitoring submodule; The data storage module is used to store the usage data of the Hadoop cluster; The tenant management sub-template is used to receive the request for adding a new tenant and add the configuration information of the tenant in the tenant management; The authorization management sub-template is used to receive the data directory or database table authorization request, query the authorized target in the tree structure in the authorization management, and allocate the read, write, execute and inherit permissions in a multiple-selection manner; The user management sub-template is used to receive the request for adding a new cluster user and add the user's configuration information in the user management; wherein the configuration information includes: user name, user type, role, and user group information; The cluster management sub-template is used to receive the new cluster request and add cluster configuration information in cluster management, wherein the configuration information includes: cluster name, cluster category, IP address, port, user name, and password information.

2. A cross-platform big data resource management system as claimed in claim 1, characterized in that: Also includes: A rule engine module, used for providing a resource request legitimacy judgment rule to the application system interface module to judge the legitimacy of the resource request; It is also used to provide resource request type judgment rules to the resource management module and the resource monitoring module to judge the type of the resource request.

3. A cross-platform big data resource management system as claimed in claim 1, characterized in that: The cluster monitoring submodule is used to obtain and monitor the average CPU information, memory information, network IO information and storage space information of the entire cluster; The tenant monitoring submodule is used to obtain and monitor the CPU information and storage space usage information of the tenant; The application monitoring submodule is used to obtain and monitor the CPU and content usage information and running time information submitted to the application; The library table monitoring submodule is used to obtain and monitor the access information of the database and data tables.

4. A cross-platform big data resource management system as claimed in claim 1, characterized in that: Also includes: The platform management module is used to manage the management configuration of the platform; wherein the management configuration includes users, organizations, permissions, audits, parameter configuration and access management.

5. A cross-platform big data resource management system as claimed in claim 1, characterized in that: Also includes: The scheduling engine module is used to manage the resource monitoring module to obtain monitoring information.

6. A cross-platform big data resource management system as claimed in claim 1, characterized in that: Also includes: WEB service module, used for user interaction.

7. A cross-platform big data resource management method, characterized in that: include: The control application system interface module receives the resource request sent by the application system, and when judging that the resource request is legal, sends the resource request to the resource management module; wherein the resource request includes: a request for adding a tenant, a request for data directory or database table authorization, a request for adding a cluster user, and a request for adding a cluster; The control resource management module selects a resource management sub-template for receiving the resource request according to the type of the resource request, and sends the resource request to the cluster interface module; wherein the resource management sub-template includes: a tenant management sub-template, an authorization management sub-template, a user management sub-template, and a cluster management sub-template; The control cluster interface module forwards the resource request to the corresponding Hadoop cluster according to the type of the resource request, and sends the resource request to the resource monitoring module; Control the resource monitoring module to select a resource monitoring submodule according to the type of the resource request within a preset time to obtain the usage data of the Hadoop cluster, and send the usage data of the Hadoop cluster to the data storage module for storage; wherein the resource monitoring submodule includes: a cluster monitoring submodule, a tenant monitoring submodule, an application monitoring submodule, and a library table monitoring submodule; The tenant management sub-template is used to receive the request for adding a new tenant and add the configuration information of the tenant in the tenant management; The authorization management sub-template is used to receive the data directory or database table authorization request, query the authorized target in the tree structure in the authorization management, and allocate the read, write, execute and inherit permissions in a multiple-selection manner; The user management sub-template is used to receive the request for adding a new cluster user and add the user's configuration information in the user management; wherein the configuration information includes: user name, user type, role, and user group information; The cluster management sub-template is used to receive the new cluster request and add cluster configuration information in cluster management, wherein the configuration information includes: cluster name, cluster category, IP address, port, user name, and password information.

8. A cross-platform big data resource management method as claimed in claim 7, characterized in that: Also includes: The control rule engine module provides resource request legitimacy judgment rules to the application system interface module to judge the legitimacy of the resource request, and provides resource request type judgment rules to the resource management module and resource monitoring module to judge the type of the resource request.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a cross-platform big data resource management method as described in any one of claims 7 to 8.

Images