Implementation method and device of distributed actuator and distributed cluster system

Abstract

The invention discloses a distributed executor implementation method and device and a distributed cluster system.The executors achieve distributed deployment in an NIO mode, each executor is installed on a client side, the executors have the functions of independently calculating and executing scripts, and the method comprises the steps that the executors receive data operation instructions sent by a server side; and the executor executes corresponding data operation based on the deployed client according to the data operation instruction. According to the scheme, the actuator adopts an NIO mode to realize distributed deployment, tedious operation of conventional distributed deployment is reduced, and a low-dependence operation environment can be realized. Therefore, when a plurality of clients / actuators are configured, service does not need to be registered to other third-party registration centers to form a distributed system, and the problem that the system needs to depend on other software to form a distributed system is solved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and more specifically, to a method and device for implementing a distributed actuator and a distributed cluster system. Background Art

[0002] Currently, distributed systems are mainly implemented through Zookeeper. However, Zookeeper is too dependent on software installation and operation configuration. Every host machine needs to download and install Zookeeper. Therefore, the current distributed system implementation process has problems such as low configuration efficiency, cumbersome installation process, and high environmental dependence. Summary of the invention

[0003] In view of this, this application provides the following technical solutions:

[0004] In a first aspect, the present application provides a method for implementing a distributed executor, wherein the executor is deployed in a distributed manner using NIO, each executor is installed on a client, and the executor has the function of independently calculating and executing scripts, and the method includes:

[0005] The executor receives the data operation instructions sent by the server;

[0006] The executor performs corresponding data operations based on the client deployed thereon according to the data operation instructions.

[0007] In a possible implementation, before the executor receives the data operation instruction sent by the server, the method further includes:

[0008] After the executor successfully registers with the server and establishes a communication connection with the server, the executor sends a heartbeat packet to the server;

[0009] After receiving a response message from the server to the heartbeat packet, a long connection is established with the server.

[0010] In a possible implementation, it also includes:

[0011] If the executor does not receive a response message returned by the server after sending a heartbeat packet to the server, it resends a connection request to the server.

[0012] In a possible implementation, the data operation instruction is a data collection instruction, and the executor performs corresponding data operations based on the deployed client according to the operation instruction, including:

[0013] The executor analyzes the data collection instruction and determines the collection target;

[0014] Collecting target data, and processing the target data based on the target type of the collected target, wherein the processing includes at least one of the following: data cleaning and data conversion;

[0015] Back up or store the processed data.

[0016] In a possible implementation, the data operation instruction is a data processing instruction, and the executor performs corresponding data operations based on the client deployed according to the operation instruction, including:

[0017] Parsing the data processing instruction to determine the processing type and processing object;

[0018] Obtaining a script file or an execution file corresponding to the client based on the processing type;

[0019] Run the script file or the execution file to perform corresponding data processing on the processing object measured by the client.

[0020] In a possible implementation, it also includes:

[0021] The processing result of the data collection instruction or the data processing instruction is returned to the server.

[0022] The second aspect of the present application provides a distributed executor implementation device, wherein the executor is deployed in a distributed manner using NIO, each executor is installed on a client, and the executor has the function of independently calculating and executing scripts, and the device includes:

[0023] The instruction receiving module is used to receive the data operation instructions sent by the server;

[0024] The instruction execution module is used to execute corresponding data operations based on the client deployed according to the data operation instructions.

[0025] A third aspect of the present application provides a distributed cluster system, including a distributed client and a server, wherein:

[0026] The distributed client includes multiple clients, and each client is equipped with an executor, and the multiple executors are distributedly deployed in NIO mode, and the executors have the functions of independent calculation and script execution;

[0027] The server includes a main server and at least one slave server, wherein a communication channel is established between the main server and the at least one slave server, wherein the main server is used to provide services for each of the clients.

[0028] In a possible implementation, if the master server fails, the service is switched to any slave server to provide services for each client.

[0029] In a possible implementation, the master server and the slave server each include a monitor for monitoring the connection status between the server and the client;

[0030] If the monitor detects that the connection between the client and the server is disconnected, the registration information of the client is deleted.

[0031] Through the above technical solutions, it can be known that the embodiments of the present application disclose a method, device and distributed cluster system for implementing a distributed executor, wherein the executor adopts the NIO method to implement distributed deployment, each executor is installed on a client, and the executor has the function of independently calculating and executing scripts, and the method includes: the executor receives the data operation instruction sent by the server; the executor performs the corresponding data operation based on the client deployed according to the data operation instruction. The executor of the above scheme adopts the NIO method to implement distributed deployment, which reduces the cumbersome operations of conventional distributed deployment and can realize a low-dependency operating environment. Therefore, when configuring multiple clients / executors, there is no need to register services with other third-party registration centers to form a distributed system. In the application scenario, users can arbitrarily expand the client / executor based on actual needs. Based on the multiplexing model of the Netty network framework, a highly available distributed cluster environment can be realized, which solves the problem that the system needs to rely on other software to form a distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 A flow chart of a method for implementing a distributed actuator disclosed in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of the execution process of the executor disclosed in the embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a distributed actuator implementation device disclosed in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of the structure of the distributed cluster system disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0037] For the purpose of reference and clarity, the descriptions, abbreviations or acronyms of the technical terms used below are summarized as follows:

[0038] High availability: High availability is one of the factors that must be considered in the design of distributed system architecture. It usually means reducing the time that the system cannot provide services through design.

[0039] Distributed: Distributed computing is a research direction in computer science. It studies how to divide a problem that requires huge computing power to solve into many small parts, and then distribute these parts to multiple computers for processing, and finally combine these calculation results to get the final result. Distributed network storage technology is to store data in a decentralized manner on multiple independent machines and devices. The distributed network storage system adopts an extensible system structure, uses multiple storage servers to share the storage load, and uses location servers to locate storage information. It not only solves the bottleneck problem of a single storage server in the traditional centralized storage system, but also improves the reliability, availability and scalability of the system.

[0040] Netty: Netty is a Java open source framework that provides asynchronous, event-driven network application frameworks and tools for quickly developing high-performance, high-reliability network servers and client programs. Netty is a client and server programming framework based on NIO. Using Netty can ensure that you can quickly and easily develop a network application, such as a client and server application that implements a certain protocol.

[0041] NIO: NIO (Non-blocking I / O) framework is a new I / O API introduced by Java since version 1.4, which aims to provide a more efficient and flexible I / O operation mode than traditional blocking I / O.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] Figure 1 This is a flow chart of a distributed executor implementation method disclosed in the embodiment of the present application. The executor described in the embodiment of the present application uses NIO to implement distributed deployment. Each executor is installed on a client. The executor has the function of independent calculation and script execution. Figure 1 As shown, the implementation method of the distributed executor may include:

[0044] Step 101: The executor receives a data operation instruction sent by the server.

[0045] In this application, the client may refer to a computer, a local PC, a PAD, etc., and the server refers to a device that provides a functional interface, such as a service area, a terminal, etc.

[0046] Since the actuator has the ability to independently calculate and execute tasks, it can operate without relying on other components and drivers. After the actuator establishes a communication connection with the server, it can receive data operation instructions sent by the server and process the data operation instructions without relying on other third-party components. The communication connection between the actuator and the server can be a traditional TCP (Transmission Control Protocol) connection.

[0047] Step 102: The executor performs corresponding data operations based on the deployed client according to the data operation instruction.

[0048] It is understandable that some data collection work needs to be obtained based on the data collection components of the client, and some data are also stored on the client. The executor has the functions of data calculation and script execution, but the operational basis of the data operations it needs to perform may require client support. Therefore, when processing data operation instructions, it is necessary to execute corresponding data operations based on the client deployed by the executor.

[0049] The executor described in this application adopts the NIO method to achieve distributed deployment, which reduces the cumbersome operations of conventional distributed deployment and can achieve a low-dependency operating environment. Therefore, when configuring multiple clients / executors, there is no need to register services with other third-party registration centers to form a distributed system. You only need to connect to the IP and port of the server to achieve data transmission and real-time communication; in the application scenario, users can arbitrarily expand the client / executor based on actual needs. Based on the multiplexing model of the Netty network framework, a highly available distributed cluster environment can be achieved, solving the problem that the system needs to rely on other software to form a distributed system.

[0050] Figure 2 This is a schematic diagram of the structure of the execution process of the executor disclosed in the embodiment of this application, combined with Figure 2 As shown, in one implementation, before the executor receives the data operation instruction sent by the server, it can also include: after the executor successfully registers on the server and establishes a communication connection with the server, the executor sends a heartbeat packet to the server; after receiving the response message from the server to the heartbeat packet, a long connection is established with the server.

[0051] After startup, the executor (client) will first send a connection request message to the server, and add the client's information to the server. The client will periodically initiate a heartbeat packet to establish a long connection. When the server does not receive the heartbeat packet within a certain period of time, the corresponding client will be removed. After receiving the heartbeat, the heartbeat confirmation message will be sent back to the client. When the client does not receive the heartbeat confirmation message after sending the heartbeat, it will try to re-initiate the connection, so that the client can always maintain communication with the server to ensure that it can receive the task assignment from the server at any time; when receiving the task assigned by the server, the client can perform task collection, data writing and log feedback, etc.

[0052] Based on this, the implementation method of the distributed executor may further include: if the executor does not receive a response message returned by the server after sending a heartbeat packet to the server, resending a connection request to the server.

[0053] In one implementation, the data operation instruction is a data collection instruction, and the executor performs corresponding data operations based on the client deployed according to the operation instruction, which may include: the executor parses the data collection instruction to determine the collection target; collects target data, and processes the target data based on the target type of the collection target, and the processing includes at least one of the following: data cleaning, data conversion; and backing up or storing the processed data.

[0054] As a separate application, the executor needs to initiate commands from the server to control the executor to perform corresponding operations. The executor can also be run through a specified interface. This application encapsulates conventional data operations into the executor, giving the executor rich functions. Different commands can be used to allow the executor to perform different operations.

[0055] After receiving the data collection instruction, the executor will parse the specific collection target in the command, perform corresponding operations according to the target type of the collection target, such as data cleaning or conversion, and back up or save the data after collection. The collected data can include but is not limited to files or data in Hadoop and FTP.

[0056] In another implementation, the data operation instruction is a data processing instruction, and the executor performs corresponding data operations based on the client on which it is deployed in accordance with the operation instruction, which may include: parsing the data processing instruction to determine the processing type and processing object; obtaining the script file or execution file corresponding to the client based on the processing type; running the script file or execution file to perform corresponding data processing on the processing object measured by the client.

[0057] After receiving the data processing instruction, the executor will find the script file or execution file for data processing and execute the running instruction to perform the corresponding data processing. The processing process log can be recorded synchronously during the processing. The data processing can be, but is not limited to, executing Shell scripts, SQL statements, Python scripts, and PowerShell scripts.

[0058] Combination Figure 2 As shown, after the executor executes the task instruction assigned by the server, the execution result can be returned to the server, that is, the processing result of the data acquisition instruction or the data processing instruction is returned to the server.

[0059] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0060] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.

[0061] Figure 3 A schematic diagram of the structure of an implementation device of a distributed actuator disclosed in an embodiment of the present application.

[0062] The executor is deployed in a distributed manner using NIO. Each executor is installed on a client. The executor has the functions of independent calculation and script execution. Figure 3 As shown, the distributed actuator implementation device 30 may include:

[0063] The instruction receiving module 301 is used to receive the data operation instruction sent by the server.

[0064] The instruction execution module 302 is used to execute corresponding data operations based on the client deployed according to the data operation instruction.

[0065] When configuring multiple clients / executors, this application solution does not need to register services with other third-party registration centers to form a distributed system. It only needs to connect to the server to achieve data transmission and real-time communication. The client / executor can be expanded based on actual needs, solving the problem of relying on other software to form a distributed system.

[0066] The specific implementation of the implementation device of the above-mentioned distributed executor and each module it contains can be found in the content introduction of the corresponding part of the method embodiment, which will not be repeated here.

[0067] Furthermore, the present application also discloses a distributed cluster system. Figure 4 This is a schematic diagram of the structure of the distributed cluster system disclosed in the embodiment of the present application, wherein the connection between the client and the server represents a virtual communication channel. Figure 4 As shown, the distributed cluster system includes a distributed client 401 and a server 402. Among them:

[0068] The distributed client includes multiple clients, and each client is equipped with an executor. Multiple executors are deployed in a distributed manner using NIO. The executor has the functions of independent calculation and script execution, and can execute shell scripts, SQL statements, read FTP files, etc.

[0069] The server includes a main server and at least one slave server, wherein a communication channel is established between the main server and the at least one slave server, wherein the main server is used to provide services for each of the clients, and can dispatch execution tasks to the executor / client, receive execution feedback, etc.

[0070] The distributed cluster system described in this embodiment is a low-dependency, high-availability distributed system. The client and the server do not need to install other software to form a highly available distributed system, and at the same time have the ability to collect data and execute scripts. The server and the client can communicate data through traditional TCP. Compared with traditional solutions that need to rely on third-party components, the deployment, configuration and installation process of the distributed cluster system is greatly simplified, making it easier and more efficient to use.

[0071] In the implementation, if the master server fails, it switches to any slave server to provide services for each client. That is, the servers establish a communication relationship with each other, and when the master server fails, other slave servers take over the related work, thereby achieving high availability of the distributed cluster system.

[0072] In addition, a monitor may be deployed in each of the master server and the slave server to monitor the connection status between the server and the client; if the monitor detects that the connection between the client and the server is disconnected, the registration information of the client is deleted.

[0073] The monitor installed in the server is triggered when the client goes offline. It can be triggered within a very short time when the client goes offline. The monitor is used to control whether the client is disconnected from the server. When the monitor is triggered, it means that the client loses connection with the server, and the client needs to actively re-initiate a connection to the server to ensure the high availability of the system. At the same time, the server also removes the offline client, that is, deletes the registration information of the client.

[0074] The distributed cluster system described in this embodiment is easy to deploy and has low dependence on the environment. The executor of the client can perform independent data collection and data processing operations, making it more convenient and efficient for users to use.

[0075] The implementation device of any one of the distributed executors described in the above embodiments includes a processor and a memory. The instruction receiving module, instruction execution module, etc. in the above embodiments are stored in the memory as program modules, and the processor executes the above program modules stored in the memory to implement corresponding functions.

[0076] The processor includes a kernel, which retrieves the corresponding program module from the memory. One or more kernels can be set, and the processing of the access data can be realized by adjusting the kernel parameters.

[0077] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0078] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned distributed executor implementation method.

[0079] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer, wherein the computer program contains software code, and after being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the implementation method of the distributed executor described above.

[0080] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0081] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0082] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for implementing a distributed actuator, characterized in that: The executor is deployed in a distributed manner using NIO. Each executor is installed on a client. The executor has the functions of independently calculating and executing scripts. The method includes: The executor receives the data operation instructions sent by the server; The executor performs corresponding data operations based on the client deployed thereon according to the data operation instructions.

2. The method for implementing a distributed actuator according to claim 1, characterized in that: Before the executor receives the data operation instruction sent by the server, it also includes: After the executor successfully registers with the server and establishes a communication connection with the server, the executor sends a heartbeat packet to the server; After receiving a response message from the server to the heartbeat packet, a long connection is established with the server.

3. The method for implementing a distributed actuator according to claim 2, characterized in that: Also includes: If the executor does not receive a response message returned by the server after sending a heartbeat packet to the server, it resends a connection request to the server.

4. The method for implementing a distributed actuator according to claim 1, characterized in that: The data operation instruction is a data collection instruction. The executor performs corresponding data operations based on the deployed client according to the operation instruction, including: The executor analyzes the data collection instruction and determines the collection target; Collecting target data, and processing the target data based on the target type of the collected target, wherein the processing includes at least one of the following: data cleaning and data conversion; Back up or store the processed data.

5. The distributed executor method according to claim 1, characterized in that: The data operation instruction is a data processing instruction. The executor performs corresponding data operations based on the deployed client according to the operation instruction, including: Parsing the data processing instruction to determine the processing type and processing object; Obtaining a script file or an execution file corresponding to the client based on the processing type; Run the script file or the execution file to perform corresponding data processing on the processing object measured by the client.

6. The method for implementing a distributed actuator according to claim 4 or 5, characterized in that: Also includes: The processing result of the data collection instruction or the data processing instruction is returned to the server.

7. A distributed actuator implementation device, characterized in that: The executor is deployed in a distributed manner using NIO. Each executor is installed on a client. The executor has the function of independently calculating and executing scripts. The device includes: The instruction receiving module is used to receive the data operation instructions sent by the server; The instruction execution module is used to execute corresponding data operations based on the client deployed according to the data operation instructions.

8. A distributed cluster system, characterized in that: It includes distributed clients and servers, including: The distributed client includes multiple clients, and each client is equipped with an executor, and the multiple executors are distributedly deployed in NIO mode, and the executors have the functions of independent calculation and script execution; The server includes a main server and at least one slave server, wherein a communication channel is established between the main server and the at least one slave server, wherein the main server is used to provide services for each of the clients.

9. The distributed cluster system according to claim 8, characterized in that: If the master server fails, switch to any slave server to provide services for each client.

10. The distributed cluster system according to claim 8, characterized in that: The master server and the slave server each include a monitor for monitoring the connection status between the server and the client; If the monitor detects that the connection between the client and the server is disconnected, the registration information of the client is deleted.

Images