Data processing system, method and device and computer equipment

Abstract

The invention relates to a data processing system, method and device and computer equipment. The system comprises at least one issuing node and at least one consumption node, and the issuing node is used for generating and issuing an operation and maintenance task to a registration node and monitoring a task result queue in the registration node; and the consumption node is used for monitoring each release node through the registration node, reading consumption node information in the operation and maintenance task under the condition that the operation and maintenance task is monitored, and executing the operation and maintenance task according to the task information under the condition that the consumption node information is matched with the consumption node. Issuing an execution result of the operation and maintenance task to a task result queue in the registration node; and the publishing node is also used for pulling the execution result when monitoring that the execution result matched with the operation and maintenance task exists, and generating result information for the operation and maintenance task according to each execution result when all the execution results corresponding to the operation and maintenance task are pulled. By adopting the method, the release efficiency of the operation and maintenance task can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing system, method, apparatus and computer equipment. Background Art

[0002] With the rapid development of information technology, concepts such as distributed systems and microservice architectures have become increasingly popular. Under these system architectures, a large number of services are deployed on different servers. How to achieve automated operation and maintenance capabilities for these servers in a complex network environment is a problem that needs to be solved.

[0003] In the related art, a control node is usually set up in the system, which collects the tasks that each server needs other servers to execute, and dispatches these tasks to the corresponding servers for execution. When the server cluster becomes larger, the computing power of the control node will become the bottleneck of the entire cluster, resulting in low efficiency of task release. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing system, method, device and computer equipment to address the above technical problems.

[0005] In a first aspect, the present application provides a data processing system. The system includes at least one publishing node and at least one consuming node, wherein:

[0006] The publishing node is used to generate an operation and maintenance task, publish the operation and maintenance task to the registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information;

[0007] The consumption node is used to monitor each of the publishing nodes through the registration node, and when monitoring any of the publishing nodes to publish the operation and maintenance task, read the consumption node information in the operation and maintenance task, and when the consumption node information matches the consumption node, execute the operation and maintenance task according to the task information, and publish the execution result of the operation and maintenance task to the task result queue in the registration node;

[0008] The publishing node is also used to pull the execution result when it monitors the task result queue and finds that there is an execution result matching the operation and maintenance task, and to generate result information for the operation and maintenance task based on each execution result when all the execution results corresponding to the operation and maintenance task are pulled.

[0009] In one of the embodiments, the consumer node is further configured to write the task information into a task instantiation queue if the consumer node information matches the consumer node;

[0010] The consumer node is further used to sequentially read task information from the task instantiation queue, generate tasks to be executed according to the task information, and write the tasks to be executed into the task running queue;

[0011] The consumption node is further used to sequentially read the tasks to be executed from the task running queue, execute the tasks to be executed, and publish the execution results of the tasks to be executed to the task result queue in the registration node.

[0012] In one of the embodiments, the publishing node is further used to create an execution result summary queue after publishing the operation and maintenance task to the registration node, and when monitoring the task result queue and finding that there is an execution result matching the operation and maintenance task, pull the execution result and write the execution result to the execution result summary queue;

[0013] The publishing node is further configured to generate result information for the operation and maintenance task according to each of the execution results when all the execution results corresponding to the operation and maintenance task exist in the execution result summary queue.

[0014] In one of the embodiments, the publishing node is also used to cancel the monitoring of the task result queue when the creation time of the execution result summary queue reaches a preset timeout time, and use the task failure information as the result information for the operation and maintenance task.

[0015] In one of the embodiments, the consumption node information includes a target consumption node identifier and / or consumption node function information;

[0016] The consumption node is further used to determine that the consumption node information matches the consumption node when the consumption node information includes the target consumption node identifier and the identifier of the consumption node matches the target consumption node identifier; or to determine that the consumption node information matches the consumption node when the consumption node information does not include the target consumption node identifier and the function supported by the consumption node matches the consumption node function information.

[0017] In a second aspect, the present application also provides a data processing method. The method is applied to a publishing node, comprising:

[0018] Generate an operation and maintenance task, publish the operation and maintenance task to a registered node, and monitor the task result queue in the registered node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to perform the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that performs the operation and maintenance task;

[0019] When monitoring that there is an execution result matching the operation and maintenance task in the task result queue, pull the execution result;

[0020] When all the execution results corresponding to the operation and maintenance task are pulled, result information for the operation and maintenance task is generated according to each of the execution results.

[0021] In a third aspect, the present application also provides a data processing method. The method is applied to a consumption node, comprising:

[0022] Monitor each publishing node through the registration node;

[0023] When monitoring any of the publishing nodes publishing an operation and maintenance task, reading the consumer node information in the operation and maintenance task;

[0024] When the consumption node information matches the consumption node, the operation and maintenance task is executed according to the task information in the operation and maintenance task, and the execution result of the operation and maintenance task is published to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

[0025] In a fourth aspect, the present application further provides a data processing device. The device is applied to a publishing node, comprising:

[0026] A generation module, used to generate an operation and maintenance task, publish the operation and maintenance task to a registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to execute the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that executes the operation and maintenance task;

[0027] The reading module is used to pull the execution result when it is monitored that there is an execution result matching the operation and maintenance task in the task result queue, and generate result information for the operation and maintenance task based on each execution result when all the execution results corresponding to the operation and maintenance task are pulled.

[0028] In a fifth aspect, the present application further provides a data processing device. The device is applied to a consumption node, comprising:

[0029] The monitoring module is used to monitor each publishing node through the registration node;

[0030] A reading module, used for reading the consumer node information in the operation and maintenance task when monitoring any of the publishing nodes publishing the operation and maintenance task;

[0031] An execution module is used to execute the operation and maintenance task according to the task information in the operation and maintenance task when the consumption node information matches the consumption node, and publish the execution result of the operation and maintenance task to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

[0032] In a sixth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0033] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0034] In an eighth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0035] The above-mentioned data processing system, method, device and computer equipment, after the publishing node publishes the operation and maintenance task to the registration node, each consumer node can query whether it can perform the operation and maintenance task through the consumer node information carried in the operation and maintenance task. In the case where a certain consumer node matches the consumer node information, the consumer node will publish the execution result to the task result queue of the registration platform after executing the operation and maintenance task, so that the publishing node can read the execution result from the task result queue and generate the result information of the operation and maintenance task according to the execution result. The embodiment of the present application adopts a decentralized strategy, and does not set a control node for dispatching tasks, but allows each consumer node to decide whether it needs to perform the operation and maintenance task after reading the operation and maintenance task. Therefore, the step of the control node selecting a node to perform the task from each consumer node can be omitted, thereby improving the distribution efficiency of the operation and maintenance task. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of signaling interaction in a data processing system in one embodiment;

[0037] Figure 2 is a flow chart of a data processing method in one embodiment;

[0038] Figure 3 is a flow chart of a data processing method in one embodiment;

[0039] Figure 4 is a schematic diagram of a data processing system in one embodiment;

[0040] Figure 5It is a schematic diagram of the internal logic of an Agent node in one embodiment;

[0041] Figure 6 This is a schematic diagram of the internal logic of a Client node in one embodiment;

[0042] Figure 7 This is a schematic diagram of viewing registered Client services on the Nacos platform in one embodiment;

[0043] Figure 8 is a structural block diagram of a data processing device in one embodiment;

[0044] Fig. 9 is a structural block diagram of a data processing device in one embodiment;

[0045] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In one embodiment, Figure 1 As shown, a data processing system is provided, the system includes at least one publishing node and at least one consuming node, wherein the publishing node is used to generate an operation and maintenance task, publish the operation and maintenance task to a registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information; the consumer node is used to monitor each publishing node through the registration node, and when it is detected that any publishing node publishes an operation and maintenance task, read the consumer node information in the operation and maintenance task, and when the consumer node information matches the consumer node, execute the operation and maintenance task according to the task information, and publish the execution result of the operation and maintenance task to the task result queue in the registration node; the publishing node is also used to pull the execution result when it is monitored that there is an execution result matching the operation and maintenance task in the task result queue, and when all the execution results corresponding to the operation and maintenance task are pulled, generate result information for the operation and maintenance task according to each execution result.

[0048] In the embodiment of the present application, a node can be a server or a cluster composed of multiple servers. The publishing node is used to generate operation and maintenance tasks and summarize the execution results of the operation and maintenance tasks. The operation and maintenance tasks are tasks related to the operation and maintenance of the entire data processing system, for example: monitoring the memory and disk usage of each server in the data processing system, and triggering an alarm when an abnormal value occurs; regularly backing up the files in the specified directory of each server to the database; regularly upgrading the services deployed in each server, etc.

[0049] The consumer node is used to perform operation and maintenance tasks. Services for performing operation and maintenance tasks are deployed on the consumer node. An operation and maintenance task can be performed by a consumer node alone or by multiple consumer nodes together.

[0050] The registration node is used to record the information uploaded by the consumer node and the publishing node to the registration node, so that the consumer node and the publishing node can access each other's information by accessing the registration node. The function of the registration node can be implemented by any existing platform that can perform service management, such as Nacos, Kubernetes, etc.

[0051] The release node is pre-configured with preset conditions for generating operation and maintenance tasks. When the preset conditions are met, the release node generates an operation and maintenance task corresponding to the preset conditions and pushes the operation and maintenance task to the registration node. For example, if the technician sets the requirement to back up important files in the data processing system every week, the preset condition is that the current time is 7 days from the last time the file backup task was released, and the operation and maintenance task corresponding to the preset condition is the file backup task.

[0052] The operation and maintenance task includes at least task information and consumption node information. Task information includes information related to the execution of the operation and maintenance task, such as the execution process of the operation and maintenance task, task parameters of the operation and maintenance task, and information about subtasks contained in the operation and maintenance task that can be split into different consumption nodes for execution.

[0053] The consumption node information includes information related to the consumption node that performs the operation and maintenance task. The consumption node information may include at least one of the target consumption node identifier and the consumption node function information. The target consumption node identifier is the identifier of the consumption node designated by the publishing node to perform the operation and maintenance task. The consumption node function information is used to indicate the functions that the node that performs the operation and maintenance task should have, such as the services that should be deployed in the consumption node, the plug-ins that the consumption node needs to install, the network status that the consumption node should have, etc.

[0054] After the publishing node publishes the operation and maintenance task, it monitors the task result queue in the registered node to obtain the execution result of the operation and maintenance task.

[0055] The consumer node can monitor whether each publishing node has published new operation and maintenance tasks through the registration node. The monitoring method can be to deploy a message subscription service in the registration node, and the consumer node subscribes to the message subscription service. When any publishing node pushes an operation and maintenance task to the registration node, the registration node pushes a message to all consumer nodes that subscribe to the message subscription service, so that the consumer node can access the registration node to obtain the operation and maintenance task after receiving the push message. Alternatively, the consumer node can also periodically poll the registration node for information about each publishing node. When the consumer node finds that a publishing node has published a new operation and maintenance task, the consumer node obtains the operation and maintenance task.

[0056] After detecting a new operation and maintenance task, the consumer node compares the consumer node information in the operation and maintenance task with its own information. If the consumer node information includes the target consumer node identifier and the consumer node identifier matches the target consumer node identifier, it is determined that the consumer node information matches the consumer node. Alternatively, if the consumer node information does not include the target consumer node identifier and the functions supported by the consumer node match the consumer node function information, it is determined that the consumer node information matches the consumer node.

[0057] If the consumer node's own information matches the consumer node information, the consumer node executes the operation and maintenance task (or a subtask under the operation and maintenance task). At the same time, the consumer node also needs to send a notification message to the registration node to instruct the registration node to mark the operation and maintenance task (or a subtask under the operation and maintenance task) as an executed task to prevent other consumer nodes from repeating the task.

[0058] If the consumer node's own information does not match the consumer node information, the consumer node does not perform the operation and maintenance task and continues to monitor each publishing node.

[0059] When a consumer node performs an operation and maintenance task and sends a notification message to it, the registration node can also push the information of the consumer node to the publishing node, so that the publishing node can determine which consumer node is performing the operation and maintenance task, so that the publishing node can later determine whether it has received all the execution results of the operation and maintenance task. Alternatively, if the registration node does not notify the publishing node which consumer node is performing the operation and maintenance task, the publishing node can also determine whether it has received all the execution results of the operation and maintenance task by other means, as described below, which will not be repeated here.

[0060] After the consumer node completes the operation and maintenance task, it publishes the execution result of the operation and maintenance task to the task result queue of the registered node. If the operation and maintenance task requires the consumer node to report specific data (for example, the task of obtaining the memory usage of each server requires the consumer node to report the collected memory usage), the execution result is the data that needs to be reported as specified by the operation and maintenance task. If the operation and maintenance task does not require the consumer node to report data (such as a file backup task), the execution result can be a notification message indicating that the consumer node has completed the task or that the task execution failed.

[0061] The execution result may carry the identifier of the operation and maintenance task or the identifier of the publishing node. When the publishing node monitors the task result queue and finds that there is an execution result that matches the operation and maintenance task it has published (for example, the identifier of the operation and maintenance task carried in the execution result matches the identifier of the operation and maintenance task published by the publishing node, or the identifier of the publishing node carried in the execution result matches the identifier of the publishing node itself), it pulls the execution result to the local computer and temporarily stores the execution result. The publishing node will continue to monitor the task result queue until all the execution results corresponding to the operation and maintenance task are pulled.

[0062] If the previously registered node pushes the information of the consumer node that performs the operation and maintenance task to the publishing node, the consumer node can also carry the identifier of the consumer node in the execution result when publishing the execution result. In this way, the publishing node can read the identifier of the consumer node in the execution result to know which consumer nodes have completed the operation and maintenance task and which consumer nodes have not completed the operation and maintenance task, so as to determine whether the publishing node has read all the execution results of the operation and maintenance task.

[0063] The publishing node can also start the timer after publishing the operation and maintenance task to the registered node. When the timer records a preset timeout, the publishing node can push a notification message to the registered node, carrying the identifier of the consumer node that has not yet written the execution result to the task result queue. After the consumer node listens to the publishing node publishing the notification message, if the identifier in the notification message is the same as the identifier of the consumer node itself, the consumer node can push the execution status of the operation and maintenance task (task execution success, task execution failure, or task being executed) to the registered node, so that the listening node can pull the execution status of the operation and maintenance task, and determine whether to continue waiting for all the execution results corresponding to the operation and maintenance task according to the execution status, or give up waiting and determine that the operation and maintenance task execution has failed.

[0064] If the publishing node cannot obtain the information of the consumer node that specifically executes the operation and maintenance task, since the number of subtasks in the operation and maintenance task and the number of execution results that the publishing node needs to obtain should be the same, the publishing node can determine whether all execution results have been obtained by counting the number of execution results that have been obtained and whether the number of subtasks in the operation and maintenance task is the same.

[0065] After the publishing node obtains all the execution results, it generates result information for the operation and maintenance task based on each execution result. The result information may include information indicating whether the operation and maintenance task is completed. If the execution result is data, the result information may also include analysis information of each execution result. For example, when the execution result is the memory usage rate of each server, the analysis information may include the average memory usage rate, the maximum memory usage rate, the server load status of the entire data processing cluster analyzed based on the above information, etc.

[0066] The publishing node can then push the result information to the device that needs to obtain the result information, such as the terminal device of the operation and maintenance personnel, the database, etc.

[0067] The data processing system provided by the embodiment of the present application, after the publishing node publishes the operation and maintenance task to the registration node, each consumption node can query whether it can perform the operation and maintenance task through the consumption node information carried in the operation and maintenance task. In the case where a certain consumption node matches the consumption node information, the consumption node will publish the execution result to the task result queue of the registration platform after executing the operation and maintenance task, so that the publishing node can read the execution result from the task result queue and generate the result information of the operation and maintenance task according to the execution result. The embodiment of the present application adopts a decentralized strategy, and does not set a control node for dispatching tasks, but allows each consumption node to decide whether it needs to perform the operation and maintenance task after reading the operation and maintenance task. Therefore, the step of the control node selecting a node to perform the task from each consumption node can be omitted, thereby improving the distribution efficiency of the operation and maintenance task.

[0068] In one embodiment, the consumption node is also used to write the task information to the task instantiation queue when the consumption node information matches the consumption node; sequentially read the task information from the task instantiation queue, generate tasks to be executed according to the task information, and write the tasks to be executed to the task running queue; sequentially read the tasks to be executed from the task running queue, execute the tasks to be executed, and publish the execution results of the tasks to be executed to the task result queue in the registration node.

[0069] In the embodiment of the present application, the consumption node uses multiple queues and multiple threads to process the operation and maintenance tasks, further improving the ability of the consumption node to concurrently process multiple operation and maintenance tasks.

[0070] At least three threads can be set in the consumer node. Thread 1 processes the task of writing task information into the task instantiation queue, thread 2 processes the task of reading task information in the task instantiation queue and generating tasks to be executed, and thread 3 processes the task of executing tasks to be executed.

[0071] When the consumer node information matches the consumer node, thread 1 reads the task information in the operation and maintenance task and writes the task information to the task instantiation queue in the consumer node.

[0072] Thread 2 sequentially pulls task information from the task instantiation queue, parses the description of the operation and maintenance task in the task information, and generates pending tasks that can be executed by the consumer node based on the task information.

[0073] Thread 3 actually executes the task to be executed, obtains the execution result of the task to be executed, and publishes the execution result to the task result queue of the registered node.

[0074] In one embodiment, the publishing node is also used to create an execution result summary queue after publishing the operation and maintenance task to the registration node, and when it is monitored that there is an execution result matching the operation and maintenance task in the task result queue, pull the execution result and write the execution result to the execution result summary queue; when there are all execution results corresponding to the operation and maintenance task in the execution result summary queue, result information for the operation and maintenance task is generated based on each execution result.

[0075] In the embodiment of the present application, the publishing node also uses multiple queues and multiple threads to process the execution results. After publishing the operation and maintenance task to the registration node, the publishing node can create an execution result summary queue for temporarily storing the execution results. At least two threads can be set in the publishing node, thread 1 processes the task of obtaining the execution result, and thread 2 processes the task of generating result information.

[0076] Thread 1 monitors the task result queue, and when there is an execution result matching the operation and maintenance task in the task result queue, pulls the execution result and writes the execution result to the execution result summary queue of the publishing node.

[0077] Thread 2 monitors each execution result in the execution result summary queue. When all execution results corresponding to the operation and maintenance task exist in the execution result summary queue, result information for the operation and maintenance task is generated according to each execution result.

[0078] In one embodiment, the publishing node is further used to cancel monitoring of the task result queue when the creation time of the execution result summary queue reaches a preset timeout time, and use the task failure information as the result information for the operation and maintenance task.

[0079] In an embodiment of the present application, after creating the execution result summary queue, the publishing node starts a timer to monitor the creation time of the execution result summary queue. When the creation time reaches the preset timeout time, the publishing node can determine that the operation and maintenance task execution has timed out. The publishing node can cancel the monitoring of the task result queue and use the task failure information as the result information for the operation and maintenance task. The task failure information is used to indicate that the operation and maintenance task execution failed.

[0080] In addition, the publishing node can also delete all execution results pulled by the publishing node before. The publishing node can also push a notification message to the registered node, which carries the identifier of the operation and maintenance task to indicate that the registered node can discard the execution result when it receives the execution result corresponding to the operation and maintenance task later.

[0081] In one embodiment, Figure 2 As shown, a data processing method is provided. The embodiment of the present application takes the method applied to a publishing node as an example, and includes the following steps:

[0082] Step 202: Generate an operation and maintenance task, publish the operation and maintenance task to the registered node, and monitor the task result queue in the registered node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to perform the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that performs the operation and maintenance task;

[0083] Step 204, when the task result queue is monitored and there is an execution result matching the operation and maintenance task, read the execution result;

[0084] Step 206: When all execution results corresponding to the operation and maintenance task are read, result information for the operation and maintenance task is generated according to each execution result.

[0085] In the embodiment of the present application, the publishing node is used to publish the operation and maintenance task. After generating the operation and maintenance task, the publishing node publishes the operation and maintenance task to the registration node and monitors the task result queue in the registration node.

[0086] If the publishing node listens to the execution result matching the operation and maintenance task published by this node in the task result queue, the publishing node obtains the execution result. After the publishing node collects all the execution results corresponding to the operation and maintenance task, it generates result information based on each execution result and pushes the result information to the device that needs to obtain the result information.

[0087] Among them, the specific information included in the operation and maintenance tasks, the specific method of generating result information, etc. can all refer to the description of the previous embodiments, and the embodiments of this application will not be repeated here.

[0088] In one embodiment, Figure 3 As shown, a data processing method is provided. The embodiment of the present application takes the method applied to a consumption node as an example, and includes the following steps:

[0089] Step 302, monitoring each publishing node through the registration node;

[0090] Step 304, when monitoring any publishing node to publish an operation and maintenance task, read the consumer node information in the operation and maintenance task;

[0091] Step 306, when the consumption node information matches the consumption node, the operation and maintenance task is executed according to the task information in the operation and maintenance task, and the execution result of the operation and maintenance task is published to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

[0092] In the embodiment of the present application, the consumption node is used to perform operation and maintenance tasks. The consumption node monitors the information of each publishing node through the registration node. In the case of monitoring that the consumption node publishes a new operation and maintenance task, the consumption node determines whether it needs to perform the operation and maintenance task by comparing its own information with the consumption node information in the operation and maintenance task.

[0093] If the consuming node needs to execute the operation and maintenance task, the consuming node executes the operation and maintenance task based on the task information in the operation and maintenance task, and publishes the execution result of the operation and maintenance task to the task result queue in the registration node, so that the publishing node can read the execution result through the task result queue.

[0094] Among them, the specific information included in the operation and maintenance tasks, the specific monitoring methods, the methods of executing the operation and maintenance tasks, etc. can all refer to the description of the previous embodiments, and the embodiments of this application will not be repeated here.

[0095] In one embodiment, Figure 4 As shown, the registration node is implemented through the Nacos platform, and the publishing node is Figure 4 The client node in the client node is Figure 4 The Agent node in , where:

[0096] The Agent node is implemented based on the Client mechanism in the Nacos platform and is obtained by installing the Agent module on the server in the data processing system. The Agent module is implemented in golang language and can be set on any type of server that needs to be managed. It supports http / https proxy forwarding capabilities and implements multi-threaded and multi-port proxy forwarding through hpProxy, which can be applied to any complex network topology.

[0097] In the Agent module, the standard jobAction interface is defined through the commonAction component to implement operation and maintenance tasks, and os, exec.Command, etc. are encapsulated internally to execute common cmd commands. The commonAction component also encapsulates runtime to dynamically obtain system information; in addition, the Agent module can also set docker-related interfaces to perform container-related operation and maintenance operations, etc.

[0098] After the Agent module is started on the server, the configClient component and namingClient component in the Agent module are initialized. The metadata information of the Agent node is registered on the Nacos platform through the namingClient component, and a heartbeat connection is maintained with the Nacos platform.

[0099] The Client node is also implemented based on the Client mechanism in the Nacos platform, and is obtained by installing the Client module on the server in the data processing system. The Client module is implemented in golang language and can run on any server that needs to request Nacos services. It also supports http / https proxy forwarding capabilities, and is suitable for any client in any region to submit operation and maintenance tasks. The Client node can also install the Nacos authentication system clientAuth module, which can control the authority of the Client node to publish operation and maintenance tasks through the account permission mechanism in the Nacos platform.

[0100] After the Client module is started on the server, it initializes the configClient component and the namingClient component, registers its own Client service instance with the Nacos platform through the namingClient component, and maintains a heartbeat connection with the Nacos platform.

[0101] See also Figure 5 , Figure 6 As shown in the figure, the specific interaction process between Agent node, Nacos platform and Client node is as follows:

[0102] S1, the Client node obtains the orchestrated operation and maintenance process and generates operation and maintenance tasks according to the operation and maintenance process.

[0103] The orchestrated operation and maintenance process can be input into the Client node by the technician through the command line of the Client node, and the Client node can convert the information input through the command line into a file in yaml format. Alternatively, the technician can also directly submit a file in yaml format to the Client node through the command line.

[0104] S2, the Client node registers the operation and maintenance task as a Client service instance in Nacos, and writes the task information of the operation and maintenance task into the metadata of the Client service instance. The Client node can also persist the Client service in Nacos so that the Client service instance can be submitted repeatedly later or by other nodes. In this way, there is no need to repeatedly publish the same operation and maintenance task to Nacos. After the service is registered, it can be viewed on the Nacos platform. For a schematic diagram, please refer to Figure 7 shown.

[0105] S3, the Client node monitors the task result queue in Nacos through the configClient module.

[0106] S4, the Agent node initiates a poll of the Client service instances in all Client nodes to the Nacos platform through the Nacos platform's SelectAllInstances interface through the namingClient component, and listens to the Client service instances in the yaml format registered by the Client nodes.

[0107] S5, after the Agent node monitors the new Client service instance, it determines whether the Agent node matches the Client node through the metadata of the Client service instance. If they match, the Agent node generates task information and sends it to the task instantiation queue.

[0108] S6, the Agent node instantiates each task information in the task instantiation queue sequentially through the task instantiation thread, and writes the instantiated tasks to be executed into the task running queue.

[0109] In S7, the Agent node obtains each task to be executed in the task running queue in sequence through the task running thread, and then executes each task to be executed. The obtained results are parsed into the execution results in yaml format and published to the Nacos platform through the configClient component.

[0110] S8, the Client node monitors the task result queue through the configClient component. If the task result queue is monitored to have an execution result matching the Client service published by the Client node, the execution result is sent to the execution result summary queue of the Client node.

[0111] S9, the Client node summarizes the execution results in the execution result summary queue through the execution result summary thread and outputs them.

[0112] S10, if you need to query the previously persisted Client service instance, you can enter the client –history command through any Client node to query and find the corresponding Client service instance. After querying the Client service instance, you can re-run the Client service instance through the client –recover command.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0115] In one embodiment, Figure 8 As shown, a data processing device 800 is provided, including: a generating module 802, a reading module 804, wherein:

[0116] A generation module 802 is used to generate an operation and maintenance task, publish the operation and maintenance task to a registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to perform the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that performs the operation and maintenance task;

[0117] The reading module 804 is used to pull the execution result when it is monitored that there is an execution result matching the operation and maintenance task in the task result queue, and generate result information for the operation and maintenance task based on each execution result when all the execution results corresponding to the operation and maintenance task are pulled.

[0118] In one embodiment, Fig. 9As shown, a data processing device 900 is provided, including: a monitoring module 902, a reading module 904, and an execution module 906, wherein:

[0119] A monitoring module 902 is used to monitor each publishing node through a registration node;

[0120] The reading module 904 is used to read the consumer node information in the operation and maintenance task when monitoring any of the publishing nodes publishing the operation and maintenance task;

[0121] Execution module 906 is used to execute the operation and maintenance task according to the task information in the operation and maintenance task when the consumption node information matches the consumption node, and publish the execution result of the operation and maintenance task to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

[0122] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0124] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing system, characterized in that: The system includes at least one publishing node and at least one consuming node, wherein: The publishing node is used to generate an operation and maintenance task, publish the operation and maintenance task to the registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information; The consumption node is used to monitor each of the publishing nodes through the registration node, and when monitoring any of the publishing nodes to publish the operation and maintenance task, read the consumption node information in the operation and maintenance task, and when the consumption node information matches the consumption node, execute the operation and maintenance task according to the task information, and publish the execution result of the operation and maintenance task to the task result queue in the registration node; The publishing node is also used to pull the execution result when it monitors the task result queue and finds that there is an execution result matching the operation and maintenance task, and to generate result information for the operation and maintenance task based on each execution result when all the execution results corresponding to the operation and maintenance task are pulled.

2. The system according to claim 1, characterized in that The consumer node is further configured to write the task information into a task instantiation queue when the consumer node information matches the consumer node; The consumer node is further used to sequentially read task information from the task instantiation queue, generate tasks to be executed according to the task information, and write the tasks to be executed into the task running queue; The consumption node is further used to sequentially read the tasks to be executed from the task running queue, execute the tasks to be executed, and publish the execution results of the tasks to be executed to the task result queue in the registration node.

3. The system according to claim 1, characterized in that The publishing node is further used to create an execution result summary queue after publishing the operation and maintenance task to the registration node, and when monitoring the task result queue and finding that there is an execution result matching the operation and maintenance task, pull the execution result and write the execution result to the execution result summary queue; The publishing node is further configured to generate result information for the operation and maintenance task according to each of the execution results when all the execution results corresponding to the operation and maintenance task exist in the execution result summary queue.

4. The system according to claim 3, characterized in that The publishing node is also used to cancel the monitoring of the task result queue when the creation time of the execution result summary queue reaches a preset timeout time, and use the task failure information as the result information for the operation and maintenance task.

5. The system according to claim 1, characterized in that The consumption node information includes a target consumption node identifier and / or consumption node function information; The consumption node is further used to determine that the consumption node information matches the consumption node when the consumption node information includes the target consumption node identifier and the identifier of the consumption node matches the target consumption node identifier; or to determine that the consumption node information matches the consumption node when the consumption node information does not include the target consumption node identifier and the function supported by the consumption node matches the consumption node function information.

6. A data processing method, characterized in that: The method is applied to a publishing node, comprising: Generate an operation and maintenance task, publish the operation and maintenance task to a registered node, and monitor the task result queue in the registered node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to perform the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that performs the operation and maintenance task; When monitoring that there is an execution result matching the operation and maintenance task in the task result queue, pull the execution result; When all the execution results corresponding to the operation and maintenance task are pulled, result information for the operation and maintenance task is generated according to each of the execution results.

7. A data processing method, characterized in that: The method is applied to a consumption node, comprising: Monitor each publishing node through the registration node; When monitoring any of the publishing nodes publishing an operation and maintenance task, reading the consumer node information in the operation and maintenance task; When the consumption node information matches the consumption node, the operation and maintenance task is executed according to the task information in the operation and maintenance task, and the execution result of the operation and maintenance task is published to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

8. A data processing device, characterized in that: The device is applied to a publishing node, and includes: A generation module, used to generate an operation and maintenance task, publish the operation and maintenance task to a registration node, and monitor the task result queue in the registration node; the operation and maintenance task includes task information and consumer node information, the task information is used to instruct the consumer node to execute the operation and maintenance task based on the task information, and the consumer node information is used to indicate the consumer node that executes the operation and maintenance task; The reading module is used to pull the execution result when it is monitored that there is an execution result matching the operation and maintenance task in the task result queue, and generate result information for the operation and maintenance task based on each execution result when all the execution results corresponding to the operation and maintenance task are pulled.

9. A data processing device, characterized in that: The device is applied to a consumption node, and includes: The monitoring module is used to monitor each publishing node through the registration node; A reading module, used for reading the consumer node information in the operation and maintenance task when monitoring any of the publishing nodes publishing the operation and maintenance task; An execution module is used to execute the operation and maintenance task according to the task information in the operation and maintenance task when the consumption node information matches the consumption node, and publish the execution result of the operation and maintenance task to the task result queue in the registration node; the task result queue is used to push the execution result to the publishing node.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to claim 5 or 6 are implemented.

Images