Big data platform operation and maintenance method and device, computer device and readable storage medium

Abstract

The application discloses a big data platform operation and maintenance method and device, computer equipment and a readable storage medium, comprising: firstly judging whether the current node is a master node. If yes, then executing a preset maintenance process based on the master node; if not, then sending a verification request to the master node at a preset interval. When the feedback result obtained from the verification request indicates that the master node has failed, the system will promote the current node to a new master node. This method can ensure that the big data platform can quickly perform failover when the master node has a problem, thereby ensuring the high availability of the system.

Description

Technical Field

[0001] This invention relates to the field of database operation and maintenance technology, and more specifically, to a big data platform operation and maintenance method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] In traditional big data platforms, a master node is typically responsible for managing and coordinating the operation of the entire system, such as task allocation and resource management. However, if the master node encounters problems or fails, the entire system may malfunction, impacting normal business operations. While some systems employ a master-slave architecture to improve availability, manual intervention is required to switch to a standby node after the master node fails, which is time-consuming and increases operational complexity. Summary of the Invention

[0003] The purpose of this invention is to provide a big data platform operation and maintenance method, apparatus, computer equipment, and readable storage medium.

[0004] In a first aspect, embodiments of the present invention provide a big data platform operation and maintenance method, including:

[0005] Get the current node and determine if the current node is the master node;

[0006] If so, then a preset maintenance process is executed based on the master node;

[0007] If not, a master node verification request is sent to the master node at a preset interval;

[0008] If the feedback result of the master node verification request indicates that the master node has failed, the current node shall be designated as the master node.

[0009] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0010] Based on the master node, the agent node is registered as a server, and if the registration is successful, the agent node is added.

[0011] Store the node data of the proxy node in the database;

[0012] Send the configured plugins to the agent node using the query plugin list;

[0013] Send the configured scheme to the agent node;

[0014] Request server resource utilization from the proxy node and execute the task balancing process. 。

[0015] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0016] Get new plugins;

[0017] After successfully uploading the new plugin, record the basic information of the new plugin in the database;

[0018] Broadcast the newly added plugin to the proxy node;

[0019] In response to a request for a new plugin feature, check if the new plugin exists in the plugin directory of the proxy node.

[0020] If so, then the feedback already exists;

[0021] If not, the newly added plugin will be stored in the plugin directory.

[0022] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0023] Determine if a target task with the same name exists, and if not, select a task solution and the required information for the task solution.

[0024] Request server resource utilization from the proxy node;

[0025] Idle proxy nodes are identified based on the server resource utilization rate, and a new task request is sent to the idle proxy nodes.

[0026] In response to a request for a new task interface, the system checks whether the target task exists based on idle proxy nodes.

[0027] If so, then the feedback indicates that the task already exists;

[0028] If not, complete the required information for the task and start the target task.

[0029] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0030] Request a list of tasks currently running on the proxy node;

[0031] Determine whether the task list stored in the database is consistent with the task execution list;

[0032] If the task list stored in the database is inconsistent with the task execution list, determine the missing tasks of the agent node;

[0033] Based on the missing task, determine the task information required for the missing task, and send a new task request to the agent node;

[0034] In response to the receipt of the new task request, a solution for a new task corresponding to the new task request is found based on the idle agent node;

[0035] If so, then the feedback indicates that the task already exists;

[0036] If not, complete the information required for the new task and start the new task.

[0037] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0038] At preset intervals, send liveness verification requests to the proxy nodes;

[0039] If the liveness verification request indicates that the proxy node has failed, an alarm message is sent.

[0040] Query the task list of the proxy node and request the health status detection interface for the task list;

[0041] If a task in the task list is unhealthy, then request to determine the server resource utilization of the surviving proxy nodes;

[0042] Send a new task request to an idle live agent node so that the idle live agent node can re-execute the tasks in the task list.

[0043] In one possible implementation, the execution of the preset maintenance process based on the master node includes:

[0044] When a new agent node is added, a determination is triggered to determine whether the new agent node is configured with a liveness task.

[0045] Without configuring a live task, request server resource utilization from each proxy node and filter out idle proxy nodes;

[0046] Send a new task request to the idle agent node;

[0047] In response to the receipt of the new task request, a solution for a new task corresponding to the new task request is found based on the idle agent node;

[0048] If so, then the feedback indicates that the task already exists;

[0049] If not, then complete the information required for the new task, and the new task will be started by the idle agent node.

[0050] Secondly, embodiments of the present invention provide a big data platform operation and maintenance device, comprising:

[0051] The acquisition module is used to acquire the current node and determine whether the current node is the master node.

[0052] The operation and maintenance module is used to execute a preset maintenance process based on the master node if the condition is met; otherwise, it sends a master node verification request to the master node at a preset interval; and if the feedback result of the master node verification request indicates that the master node has failed, it designates the current node as the master node.

[0053] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the big data platform operation and maintenance method described in at least one possible implementation of the first aspect.

[0054] Fourthly, embodiments of the present invention provide a readable storage medium, the readable storage medium including a computer program, wherein when the computer program is executed, it controls the computer device where the readable storage medium is located to execute the big data platform operation and maintenance method described in at least one possible implementation of the first aspect.

[0055] Compared to existing technologies, the beneficial effects provided by this invention include: By employing the big data platform operation and maintenance method, apparatus, computer equipment, and readable storage medium disclosed in this invention, the system determines whether the current node is the master node. If it is, a preset maintenance process is executed based on that master node; otherwise, verification requests are sent to the master node at preset intervals. When the feedback result from the verification request indicates that the master node has failed, the system promotes the current node to the new master node. This method ensures that the big data platform can quickly perform failover when the master node encounters a problem, thereby guaranteeing the high availability of the system. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the steps of the big data platform operation and maintenance method provided in this embodiment of the invention.

[0058] Figure 2 A schematic block diagram of the structure of a big data platform operation and maintenance device provided in an embodiment of the present invention;

[0059] Figure 3A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the big data platform operation and maintenance method provided in this embodiment. The following is a detailed description of the big data platform operation and maintenance method.

[0063] Step S201: Obtain the current node and determine whether the current node is the master node;

[0064] Step S202: If yes, then execute the preset maintenance process based on the master node;

[0065] Step S203: If not, send a master node verification request to the master node at a preset interval;

[0066] Step S204: If the feedback result of the master node verification request indicates that the master node has failed, the current node shall be designated as the master node.

[0067] In this embodiment of the invention, for example, suppose you are managing a big data cluster consisting of multiple nodes (servers). When you begin the operation and maintenance process, you first need to determine whether the node you are operating on is the master node. For example, you are checking the status of node A. If node A is the master node, then you can perform preset maintenance procedures on this node. For example, you may need to update software, check the system's operating status, or perform other maintenance tasks. If node A is not the master node, then it needs to periodically send verification requests to the master node to confirm whether the master node is operating normally. For example, node A sends a "Are you okay?" message to the master node every 5 minutes. If the master node does not respond to node A's verification request, or if the information returned by the master node indicates that the master node has failed, then node A will take over and become the new master node. For example, if the master node does not respond to node A's verification request within a specified time, then the system will initiate a failover procedure to set node A as the new master node to ensure the normal operation of the system.

[0068] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0069] (1) Register the proxy node with the main node as a server, and add the proxy node if the registration is successful;

[0070] (2) Store the node data of the proxy node in the database;

[0071] (3) Send the configured plugins to the agent node using the query plugin list;

[0072] (4) Send the configured scheme to the agent node;

[0073] (5) Request the server resource utilization rate from the proxy node and execute the task balancing process.

[0074] In this embodiment of the invention, for example, the master node may be managing a series of agent nodes. When a new agent needs to be added, the master node is responsible for registering the new agent. For example, if you want to add node B as a new agent, you first need to register node B on the master node. After successful registration, the master node stores the agent's relevant information (such as IP address, running status, etc.) in the database. For example, the master node saves information such as node B's IP address, operating system version, and CPU usage in the database. The master node checks the list of currently available plugins and sends the configured plugins to the newly registered agent. For example, the master node sends the json_to_sql data conversion plugin and the mongo_to_greenplum full extraction plugin to node B. In addition to plugins, the master node also sends configured plans to the agents. For example, the master node may send node B a plan containing detailed steps on how to synchronize data. To ensure the smooth operation of the entire system, the master node also queries the agents about their server resource usage. For example, the master node may ask node B about its current CPU and memory usage. Based on this information, the master node can schedule tasks to maintain load balancing. For example, if node B's resource utilization is low, the master node may allocate more tasks to node B for execution.

[0075] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0076] (1) Obtain the new plugin;

[0077] (2) Based on the successful upload of the new plugin, record the basic information of the new plugin in the database;

[0078] (3) Broadcast the newly added plugin to the agent node;

[0079] (4) In response to the request for the new plugin function, check whether the new plugin exists in the plugin directory of the proxy node;

[0080] (5) If so, then the feedback already exists;

[0081] (6) If not, the newly added plugin is stored in the plugin directory.

[0082] In this embodiment of the invention, for example, suppose a new plugin needs to be added to the system, such as a new data cleaning plugin. First, you need to obtain this new plugin on the master node. After obtaining the plugin, upload it to the master node, and after successful upload, record the basic information of the new plugin (such as plugin name, version, function description, etc.) in the database. Then, the master node broadcasts this new plugin to all agents. For example, the master node will notify all agents: "We now have a new data cleaning plugin." When a task needs to use this new plugin, the master node will check the plugin directory of each agent to see if the new plugin already exists. For example, if a task needs to use the new data cleaning plugin, the master node will check if node B already has this plugin. If the check shows that node B already has this plugin, the master node will report "exists". If node B does not yet have this plugin, then the master node will send the new plugin to node B and save it to its plugin directory. It is worth noting that the process of obtaining a new solution is basically the same as the process of obtaining a new plugin described above.

[0083] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0084] (1) Determine whether there is a target task with the same name, and if not, select a task solution and the task information required for the corresponding task solution;

[0085] (2) Request server resource utilization from the proxy node;

[0086] (3) Determine the idle proxy nodes based on the server resource utilization rate, and send a new task request to the idle proxy nodes;

[0087] (4) In response to a request for a new task interface, check whether the target task exists based on idle proxy nodes;

[0088] (5) If so, then the feedback is that the task already exists;

[0089] (6) If not, complete the information required for the task and start the target task.

[0090] In this embodiment of the invention, for example, suppose you need to add a new data synchronization task to the system. First, you need to check on the master node whether a task with the same name already exists. If not, you need to choose a suitable task solution and prepare the relevant task information. For example, you might choose a synchronization solution using the json_to_sql plugin and prepare the connection information for the source and target databases. After determining the task solution, the master node needs to query all agents about their server resource usage. For example, the master node will ask node B about its current CPU and memory usage. Based on the resource usage returned by the agents, the master node can determine which agents are currently relatively idle. Then, the master node will send new task requests to these idle agents. For example, if node B has the lowest resource usage, the master node will send a request to node B to create a new data synchronization task. When an agent receives a request to create a new task, it will check locally whether the target task already exists. For example, node B will check if a data synchronization task with the same name already exists. If node B already has this task, it will report "task already exists" to the master node. If not, then node B will complete the task configuration based on the task information provided by the master node and start this new data synchronization task.

[0091] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0092] (1) Request a list of execution tasks currently running on the proxy node;

[0093] (2) Determine whether the task list stored in the database is consistent with the task execution list;

[0094] (3) If the task list stored in the database is inconsistent with the task execution list, determine the missing tasks of the agent node;

[0095] (4) Based on the missing task, determine the task information required for the missing task, and send a new task request to the agent node;

[0096] (5) In response to the receipt of the new task request, search for a solution for the new task corresponding to the new task request based on the idle agent node;

[0097] (6) If so, then the feedback is that the task already exists;

[0098] (7) If not, complete the information required for the new task and start the new task.

[0099] In this embodiment of the invention, for example, the master node needs to know which tasks each agent is currently running. For instance, the master node might request a list of tasks currently being executed by node B. The master node compares the task list obtained from the agent with the task list stored in the database. For example, if the task list recorded in the database is "A,B,C", while the task list returned by node B is "A,B", then the master node knows that the two lists are inconsistent. If the two task lists are found to be inconsistent, the master node can determine which tasks the agent is missing. In the example above, the master node will find that node B is missing task "C". Once the missing task is determined, the master node will prepare the relevant task information and send a request to the agent to create a new task. For example, the master node will prepare the information for task "C" and send a request to node B to create a new task "C". When the agent receives the request to create a new task, it will search locally to see if a solution for this task already exists. For example, node B will check if a solution for task "C" already exists. If node B already has a solution for this task, then it will report back to the master node that "the task already exists". If not, then node B will complete the task configuration based on the task information provided by the master node and start the new task "C".

[0100] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0101] (1) Send liveness verification requests to the proxy node at preset intervals;

[0102] (2) If the liveness verification request reports that the proxy node has failed, then an alarm message is sent;

[0103] (3) Query the task list of the proxy node and request the health status detection interface for the task list;

[0104] (4) If the tasks in the task list are unhealthy, then request to determine the server resource utilization of the surviving proxy nodes;

[0105] (5) Send a new task request to an idle live agent node so that the idle live agent node can re-execute the tasks in the task list.

[0106] In this embodiment of the invention, for example, the master node needs to periodically check whether each Agent is still running normally. For instance, the master node might send a "Are you okay?" message to Node B every 5 minutes. If Node B does not respond to the liveness verification request, or if the returned information indicates that Node B has failed, the master node will send an alarm message. For example, the master node might notify the administrator via email or SMS: "Node B has failed." Besides checking whether the Agent itself is running normally, the master node also needs to check whether the tasks being executed by the Agent are healthy. For example, the master node will request a list of tasks currently being executed by Node B and perform health checks on these tasks. If a task of Node B is found to be unhealthy (e.g., not progressing, or repeatedly failing), the master node needs to find another Agent to take over the task. First, the master node will request the server resource usage of all surviving Agents. Based on the returned resource usage, the master node will find the currently idle Agent and send it a request to create a new task. For example, if Node C is found to be currently idle, the master node will send a request to Node C: "Please take over the failed task of Node B."

[0107] In this embodiment of the invention, the aforementioned step S204 can be implemented through the following examples.

[0108] (1) When a new agent node is added, a determination is triggered to determine whether the new agent node is configured with a liveness task;

[0109] (2) Without configuring live tasks, request server resource utilization from each proxy node and filter out idle proxy nodes;

[0110] (3) Send a new task request to the idle agent node;

[0111] (4) In response to the receipt of the new task request, search for a solution for the new task corresponding to the new task request based on the idle agent node;

[0112] (5) If so, then the feedback is that the task already exists;

[0113] (6) If not, complete the information required for the new task and start the new task by the idle agent node.

[0114] In this embodiment of the invention, for example, when a new Agent (e.g., node D) is added to the system, the master node needs to determine whether this new Agent has been configured with a task to be executed. If node D has not yet been configured with any task, the master node will first request the server resource usage of all Agents and then find the currently idle Agent. For example, if the returned result shows that node D is currently idle, the master node will consider node D to be an idle Agent. After determining the idle Agent, the master node can send a request to it to create a new task. For example, the master node may send a request to node D: "Please start executing the data synchronization task." When the Agent receives the request to create a new task, it will search locally to see if there is already a solution for this task. For example, node D will check if there is already a solution for the data synchronization task. If node D already has a solution for this task, it will report back to the master node that "the task already exists." If not, node D will complete the task configuration according to the task information provided by the master node and start this new data synchronization task.

[0115] In summary, this embodiment of the invention provides a highly available server platform, mainly composed of a server and an agent, capable of performing functions such as task management, plugin management, scheme management, task monitoring, and server health monitoring. The server is primarily responsible for overall function management, while the agent is responsible for executing specific tasks and managing server resources.

[0116] The following is a summary of this system:

[0117] High availability: The system adopts a master-standby scheme, that is, one master node and one backup node. When the master node fails, the backup node can take over the work of the master node, thus ensuring the high availability of the system.

[0118] Task Management: The server is responsible for assigning tasks to various agents for execution. Each task consists of one or more shell scripts, which are called a synchronization scheme. The execution status of tasks is monitored in real time, and the system will immediately issue an alert if a task fails or an agent node malfunctions.

[0119] Plugin Management: The system uses various plugins to perform data synchronization. For example, there are plugins specifically for handling dynamic changes in database data, plugins for converting JSON data to SQL, plugins for extracting all table data, and plugins for extracting all data from MongoDB to Greenplum, etc.

[0120] Synchronization Scheme Management: Each synchronization task consists of one or more synchronization schemes. A synchronization scheme is a set of plugins that execute in a specific order and logic. The server is responsible for managing these synchronization schemes, including creating, modifying, and deleting schemes, as well as assigning corresponding tasks.

[0121] Server monitoring: The system will periodically check the health status and resource usage of each Agent node, and schedule tasks based on this information to achieve a balanced distribution of tasks.

[0122] In summary, this system achieves automated management and efficient execution of data synchronization functions through collaboration between the server and agent sides.

[0123] Please refer to the following: Figure 2 , Figure 2 A big data platform operation and maintenance device 110 provided for an embodiment of the present invention includes:

[0124] The acquisition module 1101 is used to acquire the current node and determine whether the current node is the master node;

[0125] The operation and maintenance module 1102 is used to execute a preset maintenance process based on the master node if the condition is met; otherwise, it sends a master node verification request to the master node at a preset interval; and if the feedback result of the master node verification request indicates that the master node has failed, it uses the current node as the master node.

[0126] It should be noted that the implementation principle of the aforementioned big data platform operation and maintenance device 110 can be referred to the implementation principle of the aforementioned big data platform operation and maintenance device 110, and will not be repeated here. It should be understood that the division of the various modules of the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented by processing element calls to software, and some modules can be implemented in hardware. For example, the big data platform operation and maintenance device 110 can be a separately established processing element, or it can be integrated into a chip of the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and called and executed by a processing element of the aforementioned device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuit in the hardware of the processor element or through software instructions.

[0127] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0128] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned big data platform operation and maintenance device 110. Figure 3 As shown, Figure 3 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a big data platform operation and maintenance device 110, a memory 111, a processor 112, and a communication unit 113.

[0129] To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The big data platform operation and maintenance device 110 includes at least one software function module that can be stored in the memory 111 or embedded in the operating system (OS) of the computer device 100 in the form of software or firmware. The processor 112 is used to execute the big data platform operation and maintenance device 110 stored in the memory 111, such as the software function modules and computer programs included in the big data platform operation and maintenance device 110.

[0130] This invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the aforementioned big data platform operation and maintenance method.

[0131] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A big data platform operation and maintenance method, characterized in that, include: Get the current node and determine if the current node is the master node; If so, then a preset maintenance process is executed based on the master node; If not, a master node verification request is sent to the master node at a preset interval; If the feedback result of the master node verification request indicates that the master node has failed, the current node shall be designated as the master node. The execution of the preset maintenance process based on the master node includes: At preset intervals, send liveness verification requests to the proxy nodes; If the liveness verification request indicates that the proxy node has failed, an alarm message is sent. Query the task list of the proxy node and request the health status detection interface for the task list; If a task in the task list is unhealthy, then request to determine the server resource utilization of the surviving proxy nodes; Send a new task request to an idle surviving agent node so that the idle surviving agent node can re-execute the tasks in the task list; The execution of the preset maintenance process based on the master node includes: Based on the master node, the agent node is registered as a server, and if the registration is successful, the agent node is added. Store the node data of the proxy node in the database; Send the configured plugins to the agent node using the query plugin list; Send the configured scheme to the agent node; Request server resource utilization from the proxy node and execute the task balancing process.

2. The method of claim 1, wherein, The execution of the preset maintenance process based on the master node includes: Get new plugins; After successfully uploading the new plugin, record the basic information of the new plugin in the database; Broadcast the newly added plugin to the proxy node; In response to a request for a new plugin feature, check if the new plugin exists in the plugin directory of the proxy node. If so, then the feedback already exists; If not, the newly added plugin will be stored in the plugin directory.

3. The method of claim 1, wherein, The execution of the preset maintenance process based on the master node includes: Determine if a target task with the same name exists, and if not, select a task solution and the required information for the task solution. Request server resource utilization from the proxy node; Idle proxy nodes are identified based on the server resource utilization rate, and a new task request is sent to the idle proxy nodes. In response to a request for a new task interface, the system checks whether the target task exists based on idle proxy nodes. If so, then the feedback indicates that the task already exists; If not, complete the required information for the task and start the target task.

4. The method of claim 1, wherein, The execution of the preset maintenance process based on the master node includes: Request a list of tasks currently running on the proxy node; Determine whether the task list stored in the database is consistent with the task execution list; If the task list stored in the database is inconsistent with the task execution list, determine the missing tasks of the agent node; Based on the missing task, determine the task information required for the missing task, and send a new task request to the agent node; In response to the receipt of the new task request, a solution for a new task corresponding to the new task request is found based on the idle agent node; If so, then the feedback is that the task already exists; If not, complete the information required for the new task and start the new task.

5. The method of claim 1, wherein, The execution of the preset maintenance process based on the master node includes: When a new agent node is added, a determination is triggered to determine whether the new agent node is configured with a liveness task; Without configuring a live task, request server resource utilization from each proxy node and filter out idle proxy nodes; Send a new task request to the idle agent node; In response to the receipt of the new task request, a solution for a new task corresponding to the new task request is found based on the idle agent node; If so, then the feedback is that the task already exists; If not, then complete the information required for the new task, and the new task will be started by the idle agent node.

6. A big data platform operation and maintenance device, characterized in that, include: The acquisition module is used to acquire the current node and determine whether the current node is the master node. The operation and maintenance module is used to execute a preset maintenance process based on the master node if the condition is met. If not, a master node verification request is sent to the master node at a preset interval; if the feedback result of the master node verification request indicates that the master node is invalid, the current node is designated as the master node. The operation and maintenance module is specifically used for: At preset intervals, a liveness verification request is sent to the proxy node; if the liveness verification request indicates that the proxy node has failed, an alarm message is sent; the task list of the proxy node is queried, and a health status detection interface for the task list is requested; if the tasks in the task list are unhealthy, the server resource utilization rate of the surviving proxy node is determined. Send a new task request to an idle surviving agent node so that the idle surviving agent node can re-execute the tasks in the task list; Based on the master node, the proxy node is registered as a server. If the registration is successful, the proxy node is added; the node data of the proxy node is stored in the database. The system sends configured plugins to the proxy node using the plugin list query; sends the configured scheme to the proxy node; requests server resource utilization from the proxy node and executes the task balancing process.

7. A computer device, comprising: The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device performs the big data platform operation and maintenance method according to any one of claims 1-5.

8. A readable storage medium, characterized by, The readable storage medium includes a computer program, which, when executed, controls the computer device containing the readable storage medium to perform the big data platform operation and maintenance method according to any one of claims 1-5.

Images