Data processing method and data processing system for distributed cluster network

Abstract

The embodiment of the invention discloses a data processing method and a data processing system for a distributed cluster network. A specific embodiment of the method comprises the following steps: in response to determining that a network client receives a data processing task, selecting a target cluster node from locally stored cluster node information, and sending a task request to the target cluster node in a service cluster; sending a task request to the leader node in response to the received information of the leader node sent by the target cluster node; and in response to the received information of the processing node sent by the leader node, establishing a network connection with the processing node to execute the data processing task. The implementation mode is related to an underlying service technology, the DNS service and the LB service in the request link can be removed, the link is shortened, the data processing efficiency is improved, and meanwhile the influence of DNS or LB service faults on the request can be avoided.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of underlying service technology, and in particular to a data processing method and a data processing system for a distributed cluster network. Background Art

[0002] The high availability of the backend services of existing systems is usually based on the high availability function of the network LB (Load Balancer). When a backend service instance fails, the LB detects the backend and switches the traffic to the normal instance. Figure 1 As shown, the request from the client to the backend generally needs to be resolved by DNS (Domain Name System) to obtain the LB address. After obtaining the LB address, the traffic will be forwarded to the backend through the LB.

[0003] However, the inventors found that the existing service system needs to call the DNS service first and then forward it through the LB cluster. This not only makes the request link from the client to the backend service instance longer, affecting the request processing efficiency. Moreover, if the DNS or LB service appears, it will affect the request from the client to the backend instance, causing the service to be unavailable.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a data processing method, a data processing system, an electronic device, a computer-readable medium, and a computer program product for a distributed cluster network to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a data processing method for a distributed cluster network, comprising: in response to determining that a network client has received a data processing task, selecting a target cluster node from locally stored cluster node information, and sending a task request to the target cluster node in a service cluster; in response to receiving leader node information sent by the target cluster node, sending a task request to the leader node; in response to receiving processing node information sent by the leader node, establishing a network connection with the processing node to perform the data processing task.

[0008] In some embodiments, the method also includes: sending a cluster node information query request to the leader node; in response to receiving the cluster node information sent by the leader node, comparing and analyzing the received cluster node information with the stored cluster node information; in response to determining that there is a difference between the two information, storing the received cluster node information as the latest information.

[0009] In some embodiments, the method further includes: in response to receiving notification information sent by the leader node indicating a change in cluster node information, storing the changed cluster node information indicated by the notification information as the latest information.

[0010] In some embodiments, the method further includes: in response to detecting that the network client is restarted, obtaining the latest cluster node information stored locally, and connecting to the corresponding service cluster through the cluster node information.

[0011] In some embodiments, the method further includes: in response to determining that the network client is started for the first time, sending registration information to the service cluster to obtain cluster node information for local storage through the monitoring agent of the service cluster; wherein the registration information is used to enable the monitoring component of the service cluster to interact with the monitoring agent to start service scheduling, and the monitoring agent stores the cluster node information contained in the interaction information in a configuration file.

[0012] In a second aspect, some embodiments of the present disclosure provide a data processing device for a distributed cluster network, including: a node selection unit, configured to select a target cluster node from locally stored cluster node information in response to determining that a network client has received a data processing task, and send a task request to the target cluster node in a service cluster; a request sending unit, configured to send a task request to a leader node in response to receiving leader node information sent by the target cluster node; and a node connection unit, configured to establish a network connection with the processing node in response to receiving processing node information sent by the leader node to perform the data processing task.

[0013] In some embodiments, the data processing device also includes a node information query unit, which is configured to send a cluster node information query request to the leading node; in response to receiving the cluster node information sent by the leading node, compare and analyze the received cluster node information with the stored cluster node information; in response to determining that there is a difference between the two information, store the received cluster node information as the latest information.

[0014] In some embodiments, the data processing device also includes a node information updating unit configured to store the changed cluster node information indicated by the notification information as the latest information in response to receiving notification information sent by the leader node indicating the change of cluster node information.

[0015] In some embodiments, the data processing device further includes a connection unit configured to obtain the latest locally stored cluster node information in response to detecting that the network client is restarted, and connect to the corresponding service cluster through the cluster node information.

[0016] In some embodiments, the data processing device also includes a node information storage unit, which is configured to send registration information to the service cluster in response to determining that the network client is started for the first time, so as to obtain cluster node information for local storage through the monitoring agent of the service cluster; wherein the registration information is used to enable the monitoring component of the service cluster to interact with the monitoring agent to start service scheduling, and the monitoring agent stores the cluster node information contained in the interaction information in a configuration file.

[0017] In a third aspect, some embodiments of the present disclosure provide a data processing method for a distributed cluster network, including: in response to receiving a task request sent by a network client to a target cluster node in a service cluster, determining whether the target cluster node is a leader node, wherein the target cluster node is selected by the network client from locally stored cluster node information; in response to determining that the target cluster node is not a leader node, sending information about the leader node in the service cluster to the network client via the target cluster node; in response to receiving a task request sent by the network client to the leader node, sending information about the processing node in the service cluster to the network client via the leader node; in response to receiving a connection request sent by the network client to the processing node, establishing a network connection between the processing node and the network client to execute the data processing task of the network client.

[0018] In some embodiments, the method further includes: in response to detecting that cluster node information of the service cluster has changed, sending notification information representing the change in cluster node information to a network client, so that the network client updates the locally stored cluster node information.

[0019] In a fourth aspect, some embodiments of the present disclosure provide a data processing device for a distributed cluster network, including: a node determination unit, configured to determine whether a target cluster node is a leader node in response to receiving a task request sent by a network client to a target cluster node in a service cluster, wherein the target cluster node is selected by the network client from locally stored cluster node information; a leader node information sending unit, configured to send information about the leader node in the service cluster to the network client through the target cluster node in response to determining that the target cluster node is not a leader node; a processing node information sending unit, configured to send information about the processing node in the service cluster to the network client through the leader node in response to receiving a task request sent by the network client to the leader node; a network connection unit, configured to establish a network connection between the processing node and the network client in response to receiving a connection request sent by the network client to the processing node, so as to execute the data processing task of the network client.

[0020] In some embodiments, the data processing device further comprises a change notification unit configured to send notification information representing the change of cluster node information to the network client in response to detecting a change in cluster node information of the service cluster, so that the network client updates the locally stored cluster node information.

[0021] In a fifth aspect, some embodiments of the present disclosure provide a data processing system, comprising: a network client, configured to, in response to receiving a data processing task, send a task request to a service cluster by using the data processing method for a distributed cluster network described in any one of the implementation methods in the first aspect above; and a service cluster, configured to, in response to receiving a task request sent by the network client, process the task request by using the data processing method for a distributed cluster network described in any one of the implementation methods in the third aspect above.

[0022] In a sixth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the data processing method for a distributed cluster network described in any implementation method in the first aspect or the third aspect above.

[0023] In a seventh aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data processing method for a distributed cluster network described in any implementation method in the first aspect or the third aspect above is implemented.

[0024] In an eighth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the data processing method for a distributed cluster network described in any implementation manner in the first aspect or the third aspect above.

[0025] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the data processing method for a distributed cluster network in some embodiments of the present disclosure can shorten the request link from the client to the backend service, thereby improving the data processing efficiency. Figure 1 As shown, in the existing service system, when a network client (i.e., client) has a need to access data, it needs to first send a request to the DNS to obtain the LB address. After that, the network client will send a request to the LB to obtain the storage node (storage) address. The LB will forward the request to the monitor instance, that is, forward it to the management node (Master) in the cluster (monitorcluster). Finally, the network client will establish a network connection with the storage node to perform IO (Input / Output) reading and writing of data.

[0026] However, this service system needs to call the DNS service first and then forward it through the LB cluster. Therefore, the request link from the client to the backend service instance is long, which affects the data processing efficiency. In addition, if the DNS or LB service fails, the upper-layer services that depend on it will become unavailable, affecting the client's request to the backend.

[0027] Based on this, in some embodiments of the present disclosure, the data processing method for a distributed cluster network, when there is a data processing task, the network client can select the target cluster node from the locally stored cluster node information, thereby directly sending a task request to the target cluster node in the service cluster. In this way, the DNS service in the existing service system can be eliminated, thereby shortening the request link, which helps to improve data processing efficiency. In addition, if the target cluster node is a follower node rather than a leader node, the target cluster node will send the information of the leader node to the network client. In this way, the network client can send a task request to the leader node of the service cluster to obtain the information of the processing node. Then establish a network connection with the processing node to perform the data processing task. At this time, the LB service can be further eliminated, and there is no need for a corresponding forwarding process, thereby further shortening the request link. In addition, since the DNS and LB services are eliminated, the network client directly communicates with the service cluster, so the impact of DNS or LB service failures on the request can be avoided, reducing request link abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0029] Figure 1 It is a schematic diagram of the backend services of the existing system;

[0030] Figure 2 is a flow chart of some embodiments of the data processing method for a distributed cluster network disclosed in the present invention on a network client;

[0031] Figure 3 It is a schematic diagram of the structure of some embodiments of the data processing device for distributed cluster network disclosed in the present invention;

[0032] Figure 4 It is a flowchart of some embodiments of the data processing method for a distributed cluster network disclosed in the present invention at the service cluster end;

[0033] Figure 5 It is a schematic diagram of the structure of some other embodiments of the data processing device for a distributed cluster network disclosed in the present invention;

[0034] Figure 6 is a schematic diagram of a data processing system of the present disclosure;

[0035] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] In addition, the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0042] Figure 2 The process 200 of some embodiments of the data processing method for a distributed cluster network disclosed in the present invention is shown. The method is generally applied to a network client and may include the following steps:

[0043] Step 201 , in response to determining that a network client receives a data processing task, selecting a target cluster node from locally stored cluster node information, and sending a task request to the target cluster node in a service cluster.

[0044] In some embodiments, the execution body of the data processing method for a distributed cluster network disclosed in the present invention is as follows: Figure 6 The network client client shown in the figure can be connected to other electronic devices (such as Figure 6 It can be understood that for the back-end service cluster, the execution subject in this embodiment is equivalent to the network client. For the user terminal or other electronic devices, the execution subject in this embodiment may also be the server.

[0045] In addition, the service cluster in this embodiment generally adopts a network topology structure, which can be a distributed cluster based on a consensus algorithm (Raft). The service cluster can include multiple cluster nodes, such as Figure 6 As shown in the figure, there are monitor-1, monitor-2, monitor-3, etc. In the service cluster, only the leader node provides business services, and the leader node information (such as address or domain name, etc.) can be queried through the follower node.

[0046] Here, when the execution subject determines that a data processing task has been received, the target cluster node can be selected from the locally stored cluster node information. As an example, when the execution subject detects that a user performs a related data processing operation, it can be determined that the data processing task has been received. Or for another example, when the execution subject receives relevant information characterizing the data processing task sent by other electronic devices, it can be determined that the data processing task has been received. In this case, the execution subject can randomly select a cluster node information from the locally stored cluster node information, and use the cluster node indicated by the selected information as the target cluster node. Among them, the cluster node information can be information used to access each node in the cluster, such as a domain name or an address of a backend instance (such as an IP), etc. Alternatively, in order to improve the selection efficiency, the execution subject can use the first queried cluster node, or the first cluster node stored in the information, as the target cluster node.

[0047] Furthermore, after obtaining the information of the target cluster node, the execution subject can send a task request to the target cluster node in the service cluster. The task request is usually a task request representing the above-mentioned data processing task, and the specific content of the task request is not limited. The data processing task here can be a task that requires any data processing, such as data storage and reading (i.e. access).

[0048] It should be noted that there is usually only one leader node in a cluster, and multiple follower nodes. Therefore, when the execution subject selects the target cluster node, it is likely to be a follower node. So as an example, Figure 6 As shown, the execution subject can send a task request for the monitor master (ie, Leader) address to the target cluster node.

[0049] Optionally, in order to improve data processing efficiency, the execution subject may also send a task request representing the content of the data processing task to the target cluster node. In this way, if the target cluster node is a follower node, the information of the leader node will be returned to the execution subject. If the target cluster node is a leader node, the information of the processing node responsible for the data processing task will be directly returned to the execution subject. This can reduce the number of interactions between the network client and the service cluster, thereby helping to further improve data processing efficiency.

[0050] Step 202: In response to receiving the information of the leader node sent by the target cluster node, send a task request to the leader node.

[0051] In some embodiments, if the execution subject receives the information of the leader node sent by the target cluster node, it can send a task request to the leader node again. The content of the task request sent this time can be different from the content of the task request sent previously, or it can be the same. As an example, Figure 6 As shown, the execution subject can send a task request to the leader node to request the storage address. In other words, the client can randomly request the monitor node. If the request is to the Follower, the Leader's address will be returned, and the client will redirect the request to the Leader's node.

[0052] It is understandable that after receiving the cluster node information sent by the target cluster node, the execution subject may find that the information is different from the target cluster node information, which means that the target cluster node is not the leader node, and the returned cluster node information is the leader node information.

[0053] In some embodiments, after receiving the information of the leader node, the execution subject may also send a cluster node information query request to the leader node, such as Figure 6 The watch cluster topology information shown in . When the cluster node information sent by the leader node is received, the received cluster node information can be compared and analyzed with the stored cluster node information. If it is determined that there is a difference between the two information, the received cluster node information can be stored as the latest information, that is, replacing the previously locally stored cluster node information. In this way, the cluster node information can be updated in a timely manner.

[0054] Step 203, in response to receiving the information of the processing node sent by the leader node, establish a network connection with the processing node to execute the data processing task.

[0055] In some embodiments, after receiving the information of the processing node sent by the leader node, the execution subject may send a connection request to the processing node, thereby establishing a network connection with the processing node to perform the above data processing task. Figure 6 As shown in the figure, after the client obtains the address of the storage node to be accessed from the Leader node, it can establish a long network connection with the storage node to perform IO read and write operations.

[0056] It can be seen from the above description that in some embodiments of the present disclosure, the data processing method for a distributed cluster network can select a target cluster node from the locally stored cluster node information when there is a data processing task, thereby directly sending a task request to the target cluster node in the service cluster. In this way, the DNS service in the existing service system can be eliminated, thereby shortening the request link and helping to improve data processing efficiency. In addition, if the target cluster node is a follower node rather than a leader node, the target cluster node will send the information of the leader node to the network client. In this way, the network client can send a task request to the leader node of the service cluster to obtain the information of the processing node. Then establish a network connection with the processing node to perform the data processing task. At this time, the LB service can be further eliminated, and there is no need for a corresponding forwarding process, thereby further shortening the request link. In addition, since the DNS and LB services are eliminated and the network client communicates directly with the service cluster, the impact of DNS or LB service failures on the request can be avoided, reducing request link abnormalities.

[0057] It should be noted that the data processing method disclosed in the present invention removes the storage's dependence on DNS and LB. In this way, after the cluster information changes, the client needs to connect to the new cluster after restarting. For example, when the client is started for the first time, the cluster node it connects to is (ip1, ip2, ip3). Later, because the machine went offline, the machine node became (ip4, ip5, ip6). To address this issue, when the cluster nodes are changed, the reverse notification mechanism of the watch can be used to notify the client of the latest node information of the cluster for persistence.

[0058] In some embodiments, the service cluster in this embodiment implements the logic of the leader renewing the lease to the follower based on the Raft consensus algorithm. When a node becomes a leader, it can renew the lease to the follower through heartbeat. The leader can process the request only when the majority of responses consider the renewal successful. In addition, when the cluster node changes (addition or deletion), the leader can also notify the client of the new cluster node through reverse notification.

[0059] In this case, if the execution subject receives notification information sent by the leader node indicating a change in cluster node information, the changed cluster node information indicated by the notification information may be stored as the latest information.

[0060] Furthermore, in some application scenarios, if a network client restart is detected, the latest cluster node information stored locally can be obtained, and the corresponding service cluster can be connected through the cluster node information. In other words, when the client restarts abnormally locally, the cluster nodes may have been completely replaced, resulting in an inability to connect to the new service cluster after the restart. Therefore, after the client obtains the Leader node of the service cluster, it will query the cluster node information. When there is a change in the cluster node, the client will persist the latest cluster information notified by the service cluster. When restarting, it will connect to the new service cluster through the latest cluster information persisted locally.

[0061] It is understandable that, in the case of removing the dependence on the DNS service, the network client in this embodiment needs to obtain the backend "entrance" when accessing the backend service cluster. Whether the domain name or the addresses of multiple backend instances are used as the "entrance", obtaining the "entrance" information also requires the "entrance of the entrance". For this problem of recursively configuring the "entrance", the first startup of the client can be started by scheduling the service through the monitoring component (monitor). The information of starting the service includes the information of the service cluster itself.

[0062] Specifically, the execution subject can send registration information to the service cluster when it is first started. In this way, the cluster node information can be obtained through the monitoring agent (monitor-agent) of the service cluster for local storage. Among them, the registration information can be used to enable the monitoring component of the service cluster to interact with the monitoring agent to start service scheduling. And the monitoring agent stores the cluster node information contained in the interaction information in the configuration file. In other words, the first startup of the client can be pulled up by the monitor-agent from the service scheduling event watched on the monitor, and the event carries the current cluster node information of the service cluster.

[0063] It should be noted that all services in the cluster are scheduled by the monitor. When the monitor schedules services, it interacts with the monitor-agent running on the machine node. The monitor-agent starts the service to be run, and this service will connect to other services in the cluster as a client. In addition, when the monitor notifies the monitor-agent to start the service, it will send the node information (such as the IP list) in the current service cluster to the monitor-agent. The monitor-agent itself is also a network client. To connect to the monitor, it also needs to know the IP list of the service cluster. This information can be stored in the monitor-agent configuration file.

[0064] The data processing method in the disclosed embodiment, through the privatization protocol of the back-end instance and the network client, realizes that the client can actively perceive the latest cluster node information after the back-end instance node is changed. In addition, in the existing service system, if the back-end instance fails, it is necessary to go through the LB's live detection cycle before the request can be transferred to the normal node, and the switching time is relatively long. In this embodiment, all requests will be sent to the leader node in the service cluster for processing. The leader node can allocate available nodes for processing, thereby eliminating the live detection process, which is conducive to further improving data processing efficiency.

[0065] Further references Figure 3 , as a response to the above Figure 2 The present disclosure provides some embodiments of a data processing device for a distributed cluster network. Figure 2 The device can be specifically applied to various electronic devices.

[0066] like Figure 3 As shown, the data processing device 300 of some embodiments may include: a node selection unit 301, configured to select a target cluster node from locally stored cluster node information in response to determining that a network client has received a data processing task, and send a task request to the target cluster node in the service cluster; a request sending unit 302, configured to send a task request to the leader node in response to receiving leader node information sent by the target cluster node; and a node connection unit 303, configured to establish a network connection with the processing node in response to receiving processing node information sent by the leader node to perform the data processing task.

[0067] In some embodiments, the data processing device 300 may also include a node information query unit (not shown in the figure), which is configured to send a cluster node information query request to the leader node; in response to receiving the cluster node information sent by the leader node, compare and analyze the received cluster node information with the stored cluster node information; in response to determining that there is a difference between the two information, store the received cluster node information as the latest information.

[0068] In some embodiments, the data processing device 300 may also include a node information update unit (not shown in the figure), which is configured to store the changed cluster node information indicated by the notification information as the latest information in response to receiving notification information sent by the leader node indicating a change in cluster node information.

[0069] In some embodiments, the data processing device 300 may also include a connection unit (not shown in the figure), which is configured to obtain the latest cluster node information stored locally in response to detecting that the network client is restarted, and connect to the corresponding service cluster through the cluster node information.

[0070] In some embodiments, the data processing device 300 may also include a node information storage unit (not shown in the figure), which is configured to send registration information to the service cluster in response to determining that the network client is started for the first time, so as to obtain cluster node information through the monitoring agent of the service cluster for local storage; wherein the registration information is used to enable the monitoring component of the service cluster to interact with the monitoring agent to start service scheduling, and the monitoring agent stores the cluster node information contained in the interaction information in a configuration file.

[0071] It is understood that the units described in the data processing device 300 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the data processing device 300 and the units included therein, and will not be described in detail here.

[0072] Reference below Figure 4 , which shows a process 400 of another embodiment of the data processing method for a distributed cluster network disclosed in the present invention. The method is generally applied to a service cluster end and may include the following steps:

[0073] Step 401 : in response to receiving a task request sent by a network client to a target cluster node in a service cluster, determining whether the target cluster node is a leader node.

[0074] In some embodiments, the execution body of the data processing method for a distributed cluster network disclosed in the present invention is as follows: Figure 6 The monitor cluster shown in FIG. 1 can be connected to other electronic devices (such as Figure 6 Here, when the execution subject receives the task request sent by the network client, it can first determine whether the target cluster node receiving the task request is a leader node. For example, it can be determined by node identification or node information. If not, the following step 402 can be continued. If yes, step 403 can be executed. Among them, the target cluster node is selected by the network client from the cluster node information stored locally.

[0075] Step 402 , in response to determining that the target cluster node is not a leader node, sending information of a leader node in the service cluster to a network client via the target cluster node.

[0076] Step 403 , in response to receiving the task request sent by the network client to the leader node, the information of the processing nodes in the service cluster is sent to the network client through the leader node.

[0077] Step 404, in response to receiving a connection request sent by the network client to the processing node, establishing a network connection between the processing node and the network client to execute the data processing task of the network client.

[0078] In some embodiments, if the execution subject detects that the cluster node information of the service cluster has changed, it may send notification information indicating the cluster node information change to the network client through the leader node, so that the network client updates the locally stored cluster node information.

[0079] It should be noted that the specific process and beneficial effects of the above steps can be found in Figure 2 The relevant description in the embodiments will not be repeated here.

[0080] Continue to refer Figure 5 , as a response to the above Figure 4 The present disclosure provides some embodiments of a data processing device for a distributed cluster network. Figure 4 The device can be specifically applied to various electronic devices.

[0081] like Figure 5 As shown, the data processing device 500 of some embodiments may include: a node determination unit 501, configured to determine whether the target cluster node is a leader node in response to receiving a task request sent by a network client to a target cluster node in a service cluster, wherein the target cluster node is selected by the network client from the cluster node information stored locally; a leader node information sending unit 502, configured to send the information of the leader node in the service cluster to the network client through the target cluster node in response to determining that the target cluster node is not a leader node; a processing node information sending unit 503, configured to send the information of the processing node in the service cluster to the network client through the leader node in response to receiving a task request sent by the network client to the leader node; a network connection unit 504, configured to establish a network connection between the processing node and the network client in response to receiving a connection request sent by the network client to the processing node, so as to execute the data processing task of the network client.

[0082] In some embodiments, the data processing device 500 may also include a change notification unit (not shown in the figure), which is configured to send notification information representing the change of cluster node information to the network client in response to detecting a change in the cluster node information of the service cluster, so that the network client updates the locally stored cluster node information.

[0083] It is understood that the units described in the data processing device 500 are similar to those described in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the data processing device 500 and the units included therein, and will not be described in detail here.

[0084] Some embodiments of the present disclosure also provide a data processing system. Figure 6 As shown, the data processing system may include a network client and a service cluster. The network client may be configured to respond to receiving a data processing task by using the above Figure 2 The data processing method for a distributed cluster network described in any implementation of the embodiment sends a task request to the service cluster. The service cluster can be configured to respond to the task request sent by the network client by using the above Figure 4 The data processing method for a distributed cluster network described in any implementation manner in the embodiments processes the task request.

[0085] Reference below Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0086] like Figure 7 As shown, the electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0087] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a speaker, a vibrator, etc.; storage devices 708 including, for example, a disk, a hard disk, a memory card, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0088] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0089] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0090] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0091] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to determining that the network client receives a data processing task, selects a target cluster node from the locally stored cluster node information, and sends a task request to the target cluster node in the service cluster; in response to receiving the information of the leader node sent by the target cluster node, sends a task request to the leader node; in response to receiving the information of the processing node sent by the leader node, establishes a network connection with the processing node to perform the data processing task.

[0092] Alternatively, in response to receiving a task request sent by a network client to a target cluster node in a service cluster, determining whether the target cluster node is a leader node, wherein the target cluster node is selected by the network client from locally stored cluster node information; in response to determining that the target cluster node is not a leader node, sending information about the leader node in the service cluster to the network client via the target cluster node; in response to receiving a task request sent by the network client to the leader node, sending information about the processing node in the service cluster to the network client via the leader node; in response to receiving a connection request sent by the network client to the processing node, establishing a network connection between the processing node and the network client to execute a data processing task of the network client.

[0093] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a node selection unit, a request sending unit, and a node connection unit; or including a node determination unit, a leader node information sending unit, a processing node information sending unit, and a network connection unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the node selection unit may also be described as "a unit for selecting a target cluster node from locally stored cluster node information".

[0096] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0097] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned data processing methods for a distributed cluster network.

[0098] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A data processing method for a distributed cluster network, comprising: In response to determining that the network client receives the data processing task, selecting a target cluster node from the locally stored cluster node information, and sending a task request to the target cluster node in the service cluster; In response to receiving the information of the leader node sent by the target cluster node, sending the task request to the leader node; In response to receiving the information of the processing node sent by the leader node, a network connection is established with the processing node to execute the data processing task.

2. The data processing method for a distributed cluster network according to claim 1, wherein: The method further comprises: Sending a cluster node information query request to the leader node; In response to receiving the cluster node information sent by the leader node, comparing and analyzing the received cluster node information with the stored cluster node information; In response to determining that there is a difference between the two information, the received cluster node information is stored as the latest information.

3. The data processing method for a distributed cluster network according to claim 1, wherein: The method further comprises: In response to receiving the notification information sent by the leader node and indicating the change of the cluster node information, the changed cluster node information indicated by the notification information is stored as the latest information.

4. The data processing method for a distributed cluster network according to claim 1, wherein: The method further comprises: In response to detecting that the network client is restarted, the latest cluster node information stored locally is obtained, and the corresponding service cluster is connected through the cluster node information.

5. The data processing method for a distributed cluster network according to any one of claims 1 to 4, wherein: The method further comprises: In response to determining that the network client is started for the first time, sending registration information to the service cluster to obtain cluster node information through a monitoring agent of the service cluster for local storage; The registration information is used to enable the monitoring component of the service cluster to interact with the monitoring agent to start service scheduling, and the monitoring agent stores the cluster node information included in the interaction information in a configuration file.

6. A data processing method for a distributed cluster network, comprising: In response to receiving a task request sent by a network client to a target cluster node in a service cluster, determining whether the target cluster node is a leader node, wherein the target cluster node is selected by the network client from locally stored cluster node information; In response to determining that the target cluster node is not a leader node, sending information of a leader node in the service cluster to the network client via the target cluster node; In response to receiving a task request sent by the network client to the leader node, sending information of processing nodes in the service cluster to the network client through the leader node; In response to receiving a connection request sent by the network client to the processing node, a network connection is established between the processing node and the network client to execute a data processing task of the network client.

7. The data processing method for a distributed cluster network according to claim 6, wherein: The method further comprises: In response to detecting that the cluster node information of the service cluster has changed, notification information indicating the cluster node information change is sent to the network client, so that the network client updates the locally stored cluster node information.

8. A data processing system comprising: A network client, configured to send a task request to a service cluster using the data processing method according to any one of claims 1 to 5 in response to receiving a data processing task; The service cluster is configured to, in response to receiving a task request sent by a network client, process the task request using the data processing method according to any one of claims 6-7.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method for a distributed cluster network as described in any one of claims 1-5 or 6-7.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the data processing method for a distributed cluster network as described in any one of claims 1-5 or 6-7 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data processing method for a distributed cluster network as claimed in any one of claims 1 to 5 or 6 to 7.

Images