Dynamic registration method and device of data processing engine, computer equipment, medium and product

Abstract

The invention relates to a dynamic registration method and device of a data processing engine, computer equipment, a medium and a product. The method comprises the steps of displaying an operation page when it is determined that a big data task scheduling system meets an updating condition, obtaining big data base information of a target tenant input in a specified area in the operation page in response to an input operation for the operation page, and updating the target tenant according to the big data base information of the target tenant. A computer cluster is configured for the target tenant, dynamic registration operation of a data processing engine corresponding to the target tenant is executed according to big data base information of the target tenant and a resource group corresponding to the computer cluster, and the big data task scheduling system comprises a tenant sharing area and a tenant isolation area; the data processing engine is deployed in a tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in a tenant sharing area; the management unit is deployed in a container environment. By adopting the method, multi-tenant resource isolation can be effectively realized.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, computer device, medium, and product for dynamically registering a data processing engine. Background Art

[0002] With the deep penetration of big data technology in various fields, more and more enterprises choose to migrate their data processing and analysis services to public cloud platforms to achieve efficient data-driven decision-making. However, due to the extremely high requirements of different tenants for data security and privacy, and the data processed by each tenant often contains sensitive content such as business secrets and customer information, if the resources between tenants are not effectively isolated, it will lead to problems such as data leakage and cross-contamination.

[0003] Traditional big data scheduling platforms usually adopt overly complex or inefficient software layer isolation mechanisms. In terms of resource allocation, they lack the ability to dynamically scale in and out, and the process of scaling in and out is complex and the response is slow, which cannot meet the real-time changing business needs of tenants. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method, device, computer device, medium, and product for dynamically registering a data processing engine that can effectively isolate resources of multiple tenants.

[0005] In a first aspect, the present application provides a method for dynamically registering a data processing engine, including:[[]]

[0006] When it is determined that the big data task scheduling system meets the update condition, an operation page is displayed, and in response to an input operation on the operation page, the big data base information of the target tenant input in the specified area on the operation page is obtained;

[0007] For each target tenant, according to the big data base information of the target tenant, a computer cluster is configured for the target tenant;

[0008] According to the big data base information of the target tenant and the resource group corresponding to the computer cluster, a dynamic registration operation of the data processing engine corresponding to the target tenant is performed;

[0009] Among them, the big data task scheduling system includes a tenant sharing area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant sharing area; the management unit is deployed in a container environment.

[0010] In one embodiment, the step of configuring a computer cluster for the target tenant according to the big data base information of the target tenant includes:[[]]

[0011] Obtain the target task corresponding to the target tenant according to the big data base information of the target tenant;

[0012] Configure the corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0013] In one embodiment, the steps of performing the dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster include:

[0014] Obtain the cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtain the engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster;

[0015] Start the process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code;

[0016] When the process starts successfully, send the engine information of the data processing engine to the management unit for dynamic registration of the data processing engine.

[0017] In one embodiment, the method further includes:

[0018] Obtain the storage interface corresponding to the target tenant, connect to the storage interface through the operation page, and obtain the big data base information of the target tenant through the storage interface.

[0019] In one embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in the tenant shared area; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement the big data task scheduling of all target tenants.

[0020] In one embodiment, each target tenant corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0021] In a second aspect, the present application also provides a dynamic registration device for a data processing engine, including:

[0022] An information acquisition module, configured to display an operation page when it is determined that the big data task scheduling system meets the update condition, and acquire the big data base information of the target tenant input in the specified area of the operation page in response to an input operation on the operation page;

[0023] A cluster configuration module, configured to configure a computer cluster for each target tenant according to the big data base information of the target tenant;

[0024] A dynamic registration module is used to perform the dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster. Among them, the big data task scheduling system includes a tenant shared area and a tenant isolation area. The data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through the management unit deployed in the tenant shared area. The management unit is deployed in a container environment.

[0025] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method steps of any one of the first aspects are implemented.

[0026] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps of any one of the first aspects are implemented.

[0027] In a fifth aspect, the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method steps of any one of the first aspects are implemented.

[0028] For the above-mentioned dynamic registration method, device, computer device, medium and product of the data processing engine, by dividing the tenant shared area and the tenant isolation area through the big data task scheduling system, and deploying the data processing engine in the tenant isolation area, the data processing engines and related data of each tenant are in their respective isolation areas, and the data between different tenants is isolated from each other, improving the security and privacy of the data. By configuring the computer cluster for the target tenant according to the big data base information of the target tenant and combining the resource group to perform the dynamic registration of the data processing engine, the computing resources can be flexibly allocated and adjusted according to the actual needs of different tenants, realizing the on-demand configuration of resources, improving the resource utilization rate, and avoiding resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0030] Figure 1 It is an application environment diagram of the dynamic registration method of the data processing engine in an embodiment;

[0031] Figure 2 It is a flowchart of the dynamic registration method of the data processing engine in an embodiment;

[0032] Figure 3 It is a structural block diagram of a big data task scheduling system in an embodiment;

[0033] Figure 4 It is a schematic flowchart of a dynamic registration method for a data processing engine in another embodiment;

[0034] Figure 5 It is a structural block diagram of a dynamic registration device for a data processing engine in an embodiment;

[0035] Figure 6 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] The dynamic registration method for a data processing engine provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or can be placed in the cloud or other network servers. Among them, the terminal 102 is used to display an operation page when it is determined that the big data task scheduling system meets the update condition, respond to an input operation on the operation page, obtain the big data base information of the target tenant entered in the specified area on the operation page, and for each target tenant, configure a computer cluster according to the big data base information of the target tenant, and perform a dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, a smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0038] In an exemplary embodiment, as Figure 2As shown, a dynamic registration method for a data processing engine is provided. Taking the method applied to Figure 1 the terminal 102 in

[0039] S202: When it is determined that the big data task scheduling system meets the update condition, display an operation page. In response to an input operation on the operation page, obtain the big data base information of the target tenant entered in the specified area on the operation page.

[0040] Optionally, the big data task scheduling system allows users to define various big data tasks in a visual or programming manner, can clearly define the dependencies between tasks, and ensure that tasks are executed in a specific order. Taking the DolphinScheduler system as an example, DolphinScheduler is a distributed big data workflow scheduling system that provides a visual DAG orchestration interface, supports multi-tenancy, high availability, and resource isolation. Users define task flows by dragging and dropping components in the visual interface, including adding task nodes, setting dependencies between tasks, and configuring task parameters, etc. The scheduler obtains task information from the metadata database according to the scheduled configuration and dependencies of the tasks, and assigns the tasks to the worker nodes for execution. The worker nodes are responsible for the actual task execution and will call the corresponding execution engine according to the task type to run the tasks, such as calling Hadoop to execute MapReduce tasks, calling Hive to execute SQL queries, etc. During the task execution process, the system will monitor the execution status of the tasks in real time, including whether the tasks are running, whether they are executed successfully, whether there are exceptions, etc.

[0041] Optionally, the system continuously monitors its own status to determine whether it meets the update condition. When it is necessary to update the big data base information, prompt the user through the operation page and obtain the big data base information of the target tenant entered in the specified area on the operation page. Among them, the big data base information includes various key information related to the target tenant's big data environment. The target tenant refers to a specific user who uses big data services. Each target tenant may have different business requirements, data processing requirements, and resource usage patterns, etc.

[0042] S204: For each target tenant, configure a computer cluster according to the big data base information of the target tenant.

[0043] Optionally, after obtaining the big data base information of the target tenant, determine the data processing requirements and resource requirements of the target tenant, and determine the required computer cluster according to the data processing scale, complexity, and expected concurrency of the target tenant, etc., to meet the user's task requirements.

[0044] S206: Perform the dynamic registration operation of the data processing engine corresponding to the target tenant based on the big data base information of the target tenant and the resource group corresponding to the computer cluster. Among them, the big data task scheduling system includes a tenant shared area and a tenant isolation area. The data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through the management unit deployed in the tenant shared area. The management unit is deployed in a container environment.

[0045] Optionally, the big data task scheduling system has an architecture design with a tenant shared area and a tenant isolation area. The underlying big data bases such as Hadoop, Hive, and Spark are generally tenant-isolated. The scheduling execution engine that submits Yarn tasks to the big data base also needs to have tenant isolation capabilities. Therefore, the tenant isolation area is used to deploy the data processing engine exclusive to each tenant to ensure that the data and computing environments between different tenants are isolated from each other, avoiding interference and data leakage. The tenant shared area deploys the management unit. The management unit runs in a container environment and has good portability, elastic scalability, and resource isolation. It is responsible for uniformly managing and coordinating operations such as the registration of the data processing engines of each tenant.

[0046] Optionally, after determining the big data base information of the target tenant and the resource group corresponding to the computer cluster, perform the dynamic registration operation of the data processing engine corresponding to the target tenant. Among them, the data processing engine is a software system or tool for processing, analyzing, and transforming data. It is the core component in the big data processing architecture and is responsible for executing various data processing tasks to meet the data needs in different business scenarios. In the DolphinScheduler system, the WorkerServer engine is used to execute data processing tasks. Through dynamic registration, the Worker Server engine can be deployed into the DolphinScheduler context environment.

[0047] In the above method for dynamically registering the data processing engine, by dividing the tenant shared area and the tenant isolation area through the big data task scheduling system and deploying the data processing engine in the tenant isolation area, the data processing engines and related data of each tenant are in their respective isolation areas, and the data between different tenants is isolated from each other, improving the security and privacy of the data. By configuring the computer cluster for the target tenant according to the big data base information of the target tenant and dynamically registering the data processing engine in combination with the resource group, it is possible to flexibly allocate and adjust computing resources according to the actual needs of different tenants, achieve on-demand configuration of resources, improve resource utilization, and avoid resource waste.

[0048] In an exemplary embodiment, the steps of configuring a computer cluster for a target tenant according to the big data base information of the target tenant include: obtaining a target task corresponding to the target tenant according to the big data base information of the target tenant; and configuring a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0049] Optionally, the big data base information includes various relevant information about the data processing of the target tenant. By analyzing this information, the data processing tasks that the tenant needs to complete, that is, the target tasks, can be clarified. After clarifying the target tasks, further analyze the specific information of the tasks, such as the complexity of the tasks, the estimated execution time, the data processing volume, etc., and the required resources. According to these resource requirements, a corresponding number of computer devices are configured for the target tenant. If the task needs to process a large amount of data and has a high requirement for computing speed, more computer devices with powerful CPUs and sufficient memory need to be configured to ensure that the task can be completed efficiently and quickly.

[0050] In this embodiment, by obtaining the target task corresponding to the target tenant according to the big data base information of the target tenant, and configuring a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task, the computer device resources can be accurately configured according to the specific task requirements of the tenant, avoiding over - configuration or under - configuration of resources, thereby improving the utilization rate of computer cluster resources.

[0051] In an exemplary embodiment, the steps of performing a dynamic registration operation of a data processing engine corresponding to a target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster include: obtaining a cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtaining the engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; starting a process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; and when the process starts successfully, sending the engine information of the data processing engine to the management unit for dynamic registration of the data processing engine.

[0052] Optionally, the big data base information includes various relevant data and configuration details of the target tenant. By parsing this information, the cluster configuration file applicable to the tenant can be determined. According to the resource group corresponding to the computer cluster, the engine code corresponding to the target tenant is obtained. Among them, the resource group defines a set of resources in the computer cluster used to provide services for a specific tenant, including computing resources, storage resources, etc. According to the corresponding relationship between the resource group and the tenant, the engine code corresponding to the target tenant stored in the system can be found. The engine code is the core program of the data processing engine, which contains various algorithms, logics, and operation instructions for implementing data processing functions.

[0053] Further, after obtaining the cluster configuration file and the engine code, the system will initialize and start the process of the data processing engine according to the parameter settings in the configuration file. The information in the configuration file will guide how the engine interacts with each node in the computer cluster, how to allocate computing tasks, and how to manage data storage and transmission, etc. The engine code will be loaded into memory during the process startup and start executing the relevant logic of data processing, enabling the data processing engine to run normally in the specified computer cluster environment and be ready to receive and process the data tasks of the target tenant. When the process starts successfully, the engine information of the data processing engine will be sent to the management unit for dynamic registration of the data processing engine, so that subsequent appropriate data processing tasks can be assigned to the data processing engine according to the tenant's task requests, realizing the dynamic management and scheduling of the data processing engine.

[0054] In this embodiment, by obtaining specific cluster configuration files and engine codes according to the big data base information and resource groups of the tenant, and realizing the dynamic registration of the data processing engine through the management unit, the dynamic registration mechanism can dynamically add or delete data processing engines according to the actual needs of the tenant, thereby improving the utilization efficiency of resources and avoiding waste of resources.

[0055] In an exemplary embodiment, the method further includes: obtaining the storage interface corresponding to the target tenant and connecting to the storage interface through the operation page to obtain the big data base information of the target tenant through the storage interface.

[0056] Optionally, in the big data system, the data of each tenant is usually stored in a specific location. When there is a storage interface for the data system corresponding to the target tenant, the big data base information of the target tenant can be directly obtained through the storage interface.

[0057] In this embodiment, by obtaining the storage interface corresponding to the target tenant and connecting to the storage interface through the operation page to obtain the big data base information of the target tenant, the data acquisition efficiency can be improved, thereby improving the operation efficiency and performance of the entire big data processing system.

[0058] In an exemplary embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in the tenant shared area; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement the big data task scheduling of all target tenants.

[0059] Optionally, in the big data task scheduling system, the tenant sharing area is a common area where some general functional units are deployed, such as API interfaces, alarm units, and task scheduling units. These units provide services for all target tenants, aiming to achieve the sharing of resources and functions among different tenants, avoiding the separate deployment of the same functions by each tenant, thereby improving the overall efficiency and resource utilization rate of the system. Among them, the API interface serves as a bridge for interaction between the tenant and the system, providing a standardized way for the target tenant to access various functions and services of the system. The alarm unit is responsible for monitoring the running state of the system. When abnormal situations occur (such as task failures, resource shortages, system failures, etc.), it sends alarm information to relevant personnel in a timely manner. The task scheduling unit is one of the core components of the big data task scheduling system. It is responsible for reasonably allocating and scheduling computing resources according to the task requests of each target tenant and the system resource status to ensure that tasks can be executed efficiently and orderly.

[0060] In this embodiment, by deploying API interfaces, alarm units, and task scheduling units in the tenant sharing area, the sharing of these functions is achieved. Multiple target tenants can jointly use these resources, avoiding the resource waste and cost increase caused by each tenant separately deploying and maintaining the same functions.

[0061] In an exemplary embodiment, each target tenant corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0062] Optionally, in the big data task scheduling system, an independent data processing engine group is set for each target tenant. Each data processing engine group is composed of at least one data processing engine, and these data processing engines are responsible for processing the big data tasks of the corresponding target tenant. Since the data processing engine groups of different target tenants are isolated from each other, it means that the data processing processes of each user are independent both logically and physically. The data processing engine group of one user will not directly access or interfere with the data processing engine groups of other users.

[0063] In this embodiment, by isolating the data processing engine groups of different target tenants from each other, data leakage can be effectively prevented. When a failure or abnormality occurs in the data processing engine group of one target tenant, due to the isolation, it will not affect the normal operation of the data processing engine groups of other users, improving the reliability and availability of the big data processing system.

[0064] In an exemplary embodiment, as Figure 3 shown, Figure 3It is a structural block diagram of a big data task scheduling system deployed in a multi-tenant environment. Among them, the big data task scheduling system includes a tenant shared area and a tenant isolation area. The tenant shared area is configured with an operation page (UI), an API interface (API Server), a management unit (Zookeeper), a task scheduling unit (Master Server), and a configuration unit (DB); the tenant isolation area is configured with multiple data processing engine groups (WorkerGroup), different target tenants correspond to different Worker Groups, each Worker Group contains multiple data processing engines (Worker Server), and the Worker Groups are isolated from each other. Each target tenant has an independent big data base, including components such as Hadoop, Hive, and Spark, which stores the original data of the tenant and provides the data required for task processing for the Worker Server, realizing the isolation of tenant data and task processing.

[0065] Among them, the UI is the user operation interface for users to interact with the system. The API Server is used to process user requests and manage workflow metadata, and it is the interface between users and the backend functions of the system. The main functions of Zookeeper are service registration and heartbeat monitoring. Each service (such as Master Server, Worker Server, etc.) registers its own information on Zookeeper. Zookeeper monitors the service status through the heartbeat mechanism to ensure the normal operation of the service and timely detect faulty nodes. The MasterServer is used to receive tasks and dispatch them, allocate tasks to the corresponding Worker Group. At the same time, it monitors the database (DB) through Command to obtain relevant instructions and data to ensure the smooth progress of the task processing flow. The DB stores system operation-related data, such as task instructions, configuration information, etc., and provides data support for components such as the Master Server. It is the core of system data storage and management. The Worker Group receives the dispatched tasks from the Master Server and processes the tasks internally. The Worker Server is used to actually execute the tasks, extract data from the big data base corresponding to the tenant (such as big data components such as Hadoop, Hive, and Spark) and process it, and return the results after processing.

[0066] In an exemplary embodiment, as Figure 4 shown, a dynamic registration method for a data processing engine is provided. The method includes the following steps:

[0067] When it is determined that the big data task scheduling system meets the update conditions, an operation page is displayed, and in response to an input operation on the operation page, the big data base information of the target tenant input in the specified area on the operation page is obtained. Alternatively, the storage interface corresponding to the target tenant is obtained, and the operation page is connected to the storage interface to obtain the big data base information of the target tenant through the storage interface.

[0068] For each target tenant, according to the big data base information of the target tenant, the target task corresponding to the target tenant is obtained; according to the task information and resource requirements of the target task, the corresponding number of computer devices is configured for the target tenant.

[0069] According to the big data base information of the target tenant, the cluster configuration file corresponding to the target tenant is obtained, and according to the resource group corresponding to the computer cluster, the engine code corresponding to the target tenant is obtained; according to the cluster configuration file and the engine code, the process of the data processing engine corresponding to the target tenant is started; when the process is successfully started, the engine information of the data processing engine is sent to the management unit to perform dynamic registration on the data processing engine.

[0070] Among them, the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through the management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

[0071] Among them, the tenant shared area is deployed with an API interface, an alarm unit, and a task scheduling unit; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement the big data task scheduling of all target tenants.

[0072] Among them, each target tenant corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0073] In this embodiment, the tenant shared area and the tenant isolation area are divided through the big data task scheduling system, and the data processing engine is deployed in the tenant isolation area, so that the data processing engines and related data of each tenant are in their respective isolation areas, and the data between different tenants is isolated from each other, improving the security and privacy of the data. By configuring a computer cluster for the target tenant according to the big data base information of the target tenant and dynamically registering the data processing engine in combination with the resource group, the computing resources can be flexibly allocated and adjusted according to the actual needs of different tenants, realizing on-demand configuration of resources, improving resource utilization rate, and avoiding resource waste.

[0074] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0075] Based on the same inventive concept, an embodiment of the present application further provides a dynamic registration device for a data processing engine for implementing the dynamic registration method of the data processing engine involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the dynamic registration device for the data processing engine provided below can refer to the limitations on the dynamic registration method of the data processing engine in the above text, and will not be repeated here.

[0076] In an exemplary embodiment, as Figure 5 shown, a dynamic registration device for a data processing engine is provided, including: an information acquisition module 10, a cluster configuration module 20, and a dynamic registration module 30, where:

[0077] The information acquisition module 10 is configured to display an operation page when it is determined that the big data task scheduling system meets the update condition, and in response to an input operation on the operation page, acquire the big data base information of the target tenant input in a specified area on the operation page.

[0078] The cluster configuration module 20 is configured to configure a computer cluster for each target tenant according to the big data base information of the target tenant.

[0079] The dynamic registration module 30 is configured to perform a dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster; wherein, the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

[0080] In an exemplary embodiment, the cluster configuration module 20 is further configured to obtain a target task corresponding to the target tenant according to the big data base information of the target tenant; and configure a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0081] In an exemplary embodiment, the dynamic registration module 30 is further configured to obtain a cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, obtain an engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; start a process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; and send the engine information of the data processing engine to the management unit in the case that the process is successfully started, so as to perform dynamic registration on the data processing engine.

[0082] In an exemplary embodiment, the information acquisition module 10 is further configured to obtain a storage interface corresponding to the target tenant, connect to the storage interface through an operation page, so as to obtain the big data base information of the target tenant through the storage interface.

[0083] In an exemplary embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in a tenant shared area involved in the dynamic registration module 30; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement big data task scheduling for all target tenants.

[0084] In an exemplary embodiment, each target tenant involved in the dynamic registration module 30 corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0085] Each module in the above dynamic registration device of the data processing engine can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0086] In an exemplary embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for dynamically registering a data processing engine. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0087] Those skilled in the art can understand that Figure 6 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0088] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented: when it is determined that the big data task scheduling system meets the update condition, display an operation page, and in response to an input operation on the operation page, obtain the big data base information of the target tenant input in a specified area on the operation page; for each target tenant, configure a computer cluster for the target tenant according to the big data base information of the target tenant; perform a dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster; where the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

[0089] In one embodiment, when the processor executes a computer program, configuring a computer cluster for a target tenant according to the big data base information of the target tenant includes: obtaining a target task corresponding to the target tenant according to the big data base information of the target tenant; and configuring a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0090] In one embodiment, when the processor executes a computer program, performing a dynamic registration operation of a data processing engine corresponding to a target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster includes: obtaining a cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtaining an engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; starting a process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; and sending the engine information of the data processing engine to a management unit to perform dynamic registration on the data processing engine when the process is successfully started.

[0091] In one embodiment, when the processor executes a computer program, the following steps are further implemented: obtaining a storage interface corresponding to the target tenant, and connecting to the storage interface through an operation page to obtain the big data base information of the target tenant through the storage interface.

[0092] In one embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in a tenant shared area; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement big data task scheduling for all target tenants.

[0093] In one embodiment, each target tenant corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0094] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: when it is determined that the big data task scheduling system meets the update condition, an operation page is displayed, and in response to an input operation on the operation page, the big data base information of the target tenant input in the specified area of the operation page is obtained; for each target tenant, according to the big data base information of the target tenant, a computer cluster is configured for the target tenant; according to the big data base information of the target tenant and the resource group corresponding to the computer cluster, a dynamic registration operation of the data processing engine corresponding to the target tenant is performed; wherein, the big data task scheduling system includes a tenant sharing area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant sharing area; the management unit is deployed in a container environment.

[0095] In one embodiment, configuring a computer cluster for the target tenant according to the big data base information of the target tenant when the computer program is executed by the processor includes: obtaining the target task corresponding to the target tenant according to the big data base information of the target tenant; and configuring a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0096] In one embodiment, performing a dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster when the computer program is executed by the processor includes: obtaining the cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtaining the engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; starting the process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; and when the process is successfully started, sending the engine information of the data processing engine to the management unit for dynamic registration of the data processing engine.

[0097] In one embodiment, the following steps are further implemented when the computer program is executed by the processor: obtaining the storage interface corresponding to the target tenant, and connecting to the storage interface through the operation page to obtain the big data base information of the target tenant through the storage interface.

[0098] In one embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in the tenant sharing area; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement the big data task scheduling of all target tenants.

[0099] In one embodiment, each target tenant involved when the computer program is executed by a processor corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0100] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the following steps: when it is determined that the big data task scheduling system meets the update condition, display an operation page, and in response to an input operation on the operation page, obtain the big data base information of the target tenant input in the specified area of the operation page; for each target tenant, configure a computer cluster for the target tenant according to the big data base information of the target tenant; perform a dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster; wherein, the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

[0101] In one embodiment, configuring a computer cluster for a target tenant according to the big data base information of the target tenant when the computer program is executed by a processor includes: obtaining the target task corresponding to the target tenant according to the big data base information of the target tenant; configuring a corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

[0102] In one embodiment, performing a dynamic registration operation of the data processing engine corresponding to a target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster when the computer program is executed by a processor includes: obtaining the cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtaining the engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; starting the process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; when the process is successfully started, sending the engine information of the data processing engine to the management unit for dynamic registration of the data processing engine.

[0103] In one embodiment, the computer program also implements the following steps when executed by a processor: obtaining the storage interface corresponding to the target tenant, and connecting to the storage interface through the operation page to obtain the big data base information of the target tenant through the storage interface.

[0104] In one embodiment, an API interface, an alarm unit, and a task scheduling unit are deployed in a tenant shared area involved when a computer program is executed by a processor; the API interface, the alarm unit, and the task scheduling unit are shared by multiple target tenants and are used to implement big data task scheduling for all target tenants.

[0105] In one embodiment, each target tenant involved when a computer program is executed by a processor corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

[0106] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, a database, or other media used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., and are not limited thereto.

[0107] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0108] The above-described embodiments merely represent several implementation manners of this application, and their descriptions are relatively specific and detailed. However, it should not be construed as a limitation to the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A dynamic registration method for a data processing engine, characterized in that, Applied to a big data task scheduling system deployed in a multi-tenant environment; the method includes: When it is determined that the big data task scheduling system meets the update condition, display an operation page, and in response to an input operation on the operation page, obtain the big data base information of the target tenant entered in the specified area on the operation page; For each target tenant, configure a computer cluster for the target tenant according to the big data base information of the target tenant; Execute the dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster; Wherein, the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

2. The method according to claim 1, wherein The configuring a computer cluster for the target tenant according to the big data base information of the target tenant includes: Obtain the target task corresponding to the target tenant according to the big data base information of the target tenant; Configure the corresponding number of computer devices for the target tenant according to the task information and resource requirements of the target task.

3. The method according to claim 1, characterized in that, The executing the dynamic registration operation of the data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster includes: Obtain the cluster configuration file corresponding to the target tenant according to the big data base information of the target tenant, and obtain the engine code corresponding to the target tenant according to the resource group corresponding to the computer cluster; Start the process of the data processing engine corresponding to the target tenant according to the cluster configuration file and the engine code; When the process is successfully started, send the engine information of the data processing engine to the management unit to perform dynamic registration of the data processing engine.

4. The method according to claim 1, wherein The method further includes: Obtain the storage interface corresponding to the target tenant, connect to the storage interface through the operation page, so as to obtain the big data base information of the target tenant through the storage interface.

5. The method according to claim 1, characterized in that The tenant shared area is deployed with an API interface, an alarm unit and a task scheduling unit; the API interface, the alarm unit and the task scheduling unit are shared by multiple target tenants for realizing the big data task scheduling of all target tenants.

6. The method according to claim 1, characterized in that, Each target tenant corresponds to a data processing engine group, and each data processing engine group includes at least one data processing engine; the data processing engine groups corresponding to different target tenants are isolated from each other.

7. A dynamic registration device for a data processing engine, characterized in that Applied to a big data task scheduling system deployed in a multi-tenant environment; the device includes: An information acquisition module, configured to display an operation page when it is determined that the big data task scheduling system meets the update condition, and in response to an input operation on the operation page, obtain the big data base information of the target tenant entered in the specified area on the operation page; A cluster configuration module, which is used to configure a computer cluster for each target tenant according to the big data base information of the target tenant; A dynamic registration module, which is used to perform a dynamic registration operation of a data processing engine corresponding to the target tenant according to the big data base information of the target tenant and the resource group corresponding to the computer cluster; wherein, the big data task scheduling system includes a tenant shared area and a tenant isolation area; the data processing engine is deployed in the tenant isolation area, and the data processing engine is dynamically registered through a management unit deployed in the tenant shared area; the management unit is deployed in a container environment.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images