Distributed computing system, method and equipment based on hybrid cluster and storage medium

Abstract

The invention provides a distributed computing system, method and device based on a hybrid cluster and a storage medium, the distributed computing system based on the hybrid cluster comprises a cluster manager used for distributing a stream computing topology configuration file to a task manager; the cluster coordinator is used for collecting the states of the task process and the task manager, sending the state of the task process to the task manager and sending the state of the task manager to the cluster manager; the task manager is used for creating a task process based on the flow calculation topology configuration file, establishing a flow calculation topology based on the task process and managing the task process based on the state of the task process; wherein the task process serves as a node of the stream computing topology, and the task process is used for executing distributed stream computing and runs in an independent process mode. According to the method, the task process is limited to run in an independent process mode, resource scrambling and logic pollution caused by multi-service task sharing processes are avoided, and the stability of the whole computing cluster is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, and in particular to a distributed computing system, method, device and storage medium based on a hybrid cluster. Background Technology

[0002] With the rapid development and widespread application of computer technology, more and more applications have brought convenience to life and work, while also increasing reliance on computer technology. Although the performance and reliability of single computers are improving, there are still many tasks that are difficult for a single server to complete. Therefore, cluster technology has emerged, connecting servers located in different geographical locations to work collaboratively to complete corresponding tasks. Based on this, distributed real-time stream computing systems are widely used, especially in scenarios involving real-time processing of large amounts of data. However, in existing distributed real-time stream processing systems, the sharing of processes by multiple business tasks leads to resource contention and logic pollution, creating risks of task interference and affecting the processing of distributed real-time streams. Summary of the Invention

[0003] This invention provides a distributed computing system, method, device, and storage medium based on a hybrid cluster to address the deficiencies in the prior art.

[0004] This invention provides a distributed computing system based on a hybrid cluster, comprising: The cluster manager is used to distribute streaming computing topology configuration files to the task manager; The cluster coordinator is used to collect the status of task processes and task managers, and send the status of the task managers to the cluster manager and the status of the task processes to the task managers. The task manager is used to create task processes based on the stream computing topology configuration file, establish stream computing topologies based on the task processes, and manage the task processes based on their status. The task processes are established as nodes in the stream computing topology and are used to perform distributed stream computing, running as independent processes.

[0005] According to the distributed computing system based on a hybrid cluster provided by the present invention, the task process loads only one business module and runs one business module logic when performing distributed stream computing.

[0006] According to a distributed computing system based on a hybrid cluster provided by the present invention, the task manager creates and simultaneously launches two identical task processes on the same server in the cluster, or creates and simultaneously launches two identical task processes on different servers in the cluster.

[0007] According to the present invention, a distributed computing system based on a hybrid cluster is provided, wherein the cluster manager, the task manager, and the task processes communicate indirectly through the cluster coordinator.

[0008] According to a distributed computing system based on a hybrid cluster provided by the present invention, the cluster manager creates a target cluster mode based on a stream computing topology configuration file, and distributes the stream computing topology configuration file to the task manager in the target cluster mode; wherein, the target cluster mode includes at least one of a first cluster mode, a second cluster mode, and a third cluster mode; The first cluster mode is a mode in which each server in the cluster runs a complete data flow and processing stream computing topology; The second cluster mode connects the current node with all downstream computing nodes of all servers in the cluster, and all computing nodes in the cluster perform computations in a parallel and asynchronous manner. The third cluster module is a mode that deploys specific computing nodes to one or more other servers besides the current server to achieve data aggregation or distribution across the entire network. It is a combination of the first cluster mode and the second cluster mode.

[0009] According to the present invention, a distributed computing system based on a hybrid cluster is provided, wherein the cluster coordinator includes a ZooKeeper cluster coordinator.

[0010] According to the distributed computing system based on a hybrid cluster provided by the present invention, the cluster manager is also used to control the start or stop of the flow computing topology, as well as the expansion or contraction of the cluster. The task manager is also used to control the restart of the task process.

[0011] This invention also provides a distributed computing method based on a hybrid cluster, comprising: Obtain the stream computing topology configuration file and distribute the stream computing topology configuration file to the compute node where the task manager is located; The task manager is controlled to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology; Based on the aforementioned stream computing topology, the task processes are controlled to execute distributed stream computing as independent processes.

[0012] According to a distributed computing method based on a hybrid cluster provided by the present invention, the step of connecting the task processes based on the stream computing topology configuration file to obtain the stream computing topology includes: The target cluster mode and the downstream task processes of the task process are determined based on the stream computing topology configuration file. Based on the target cluster mode and the downstream task processes, all the task processes are connected to obtain the stream computing topology.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distributed computing method based on hybrid clusters as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed computing method based on a hybrid cluster as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed computing method based on a hybrid cluster as described above.

[0016] This invention provides a distributed computing system, method, device, and storage medium based on a hybrid cluster. The cluster manager distributes stream computing topology configuration files, and the cluster coordinator collects the status of task processes and the task manager, sending the status to both. The task manager creates task processes and establishes a stream computing topology based on these processes. Each task process executes distributed stream computing and runs as an independent process. By limiting task processes to run as independent processes, resource contention and logic pollution caused by multiple business tasks sharing processes are avoided, increasing the stability of the entire computing cluster. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the distributed computing system based on a hybrid cluster provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the stream computing topology provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the first cluster mode provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the second cluster mode provided in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the third cluster mode provided in the embodiments of the present invention.

[0023] Figure 6 This is a schematic diagram illustrating the connection between the task manager and the task process provided in an embodiment of the present invention.

[0024] Figure 7 This is a flowchart illustrating the distributed computing method based on a hybrid cluster provided by the present invention.

[0025] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The following is combined Figures 1-8 This invention describes a distributed computing system, method, device, and storage medium based on a hybrid cluster.

[0028] Figure 1 This is a schematic diagram illustrating a distributed computing system based on a hybrid cluster, according to an exemplary embodiment. For example... Figure 1 As shown, in an exemplary embodiment, the distributed computing system based on a hybrid cluster includes: Cluster Manager 110 is used to distribute streaming computing topology configuration files to the Task Manager; Cluster coordinator 120 is used to collect the status of task processes and task managers, and send the status of the task managers to the cluster manager and the status of the task processes to the task managers. Task Manager 130 is used to create task processes based on the stream computing topology configuration file, establish a stream computing topology based on the task processes, and manage the task processes based on their status; wherein, the task processes serve as nodes in the stream computing topology and are used to perform distributed stream computing, running as independent processes.

[0029] In this embodiment of the invention, the cluster coordinator connects servers (also known as nodes) scattered in different geographical locations to form a spatially distributed but logically unified computing cluster, which provides services to users as a whole. The cluster coordinator balances the load to each server, which can handle higher access volumes.

[0030] A process is a single running activity of a program with independent functionality on a certain set of data. It can request and own system resources and is the most basic unit of allocation and execution. In modern thread-oriented computer architectures, a process is a container for threads, and a process is a memory space.

[0031] like Figure 1 As shown, a distributed computing system based on a hybrid cluster includes a cluster manager, a cluster coordinator, a task manager, and task processes. Typically, a distributed real-time stream computing system has a stream computing topology for data flow and processing, such as... Figure 2 As shown, data stream A and data stream B are computed by their respective task processes (workers).

[0032] In this embodiment of the invention, only one cluster manager runs on the cluster control node, responsible for distributing the stream computing topology configuration, starting and stopping the stream computing topology, and monitoring the cluster status. The cluster control node is responsible for the global management, resource scheduling, status monitoring, and coordination of the cluster.

[0033] The cluster coordinator is responsible for collecting the status of task managers and task processes (workers) in the cluster, and notifying the corresponding cluster managers and task managers of the collected status. The status of the task manager includes information about the task processes it manages, and the status of the task processes includes information such as the creation, destruction, name, and distributed stream computation being performed.

[0034] Only one task manager runs on each task compute node. The task manager is responsible for establishing the stream computing topology, automatically restarting task processes, and monitoring their status (which is obtained from the cluster). The task compute nodes execute the actual tasks (such as data processing and application execution). The task manager manages the task processes based on their status. Each task manager can manage at least one task process.

[0035] The task process (Worker) is an independent process that loads only one business module and runs only one business module logic.

[0036] In this embodiment of the invention, the cluster manager distributes the streaming computing topology configuration file to the task manager. The cluster coordinator collects the status of the task processes and the task manager, and sends the status to both the task manager and the cluster manager. The task manager creates task processes and establishes a streaming computing topology based on these task processes. Each task process executes distributed streaming computing and runs as an independent process. By limiting task processes to run as independent processes, resource contention and logic pollution caused by multiple business tasks sharing processes are avoided, thus increasing the stability of the entire computing cluster.

[0037] In an exemplary embodiment of the present invention, the cluster manager creates a target cluster mode based on a stream computing topology configuration file, and distributes the stream computing topology configuration file to the task manager in the target cluster mode; wherein, the target cluster mode includes at least one of a first cluster mode, a second cluster mode, and a third cluster mode; The first cluster mode is a mode in which each server in the cluster runs a complete data flow and processing stream computing topology; The second cluster mode connects the current node with all downstream computing nodes of all servers in the cluster, and all computing nodes in the cluster perform computations in a parallel and asynchronous manner. The third cluster mode is a mode that deploys specific computing nodes to one or more other servers besides the current server to achieve data aggregation or distribution across the entire network. It is a combination of the first and second cluster modes.

[0038] In this embodiment of the invention, the cluster mode can be configured through a stream computing topology configuration file. Furthermore, based on business characteristics, one or more task processes can be set to use a different cluster mode than other task processes. In other words, multiple cluster modes can exist in a stream computing topology.

[0039] This invention defines three cluster modes: the first cluster mode, the second cluster mode, and the third cluster mode, also known as the simple cluster mode, the full-network cluster mode, and the hybrid cluster mode, respectively. These three cluster modes can be flexibly configured according to the needs of the business scenario through a stream computing topology configuration file. Specific task processes can use full-network communication, while other task processes use local communication. The following is a description of the three cluster modes: In the first cluster mode, each server in the cluster independently and in parallel runs a complete data flow and stream computing topology. Specifically, as follows... Figure 3 As shown, the task processes in server 1 and server 2 do not involve data transfer or stream computing topology processing between each other. The first cluster mode simplifies the system structure to the greatest extent and does not require TCP communication between servers, which can minimize network traffic. Since the communication takes place within the same service, the communication efficiency is high. At the same time, since multiple servers run in parallel, the task processes are executed asynchronously and in parallel, which has the characteristics of large-scale cluster parallel and asynchronous operation. The overall efficiency is high and the disaster recovery is good. If one server goes down, the other servers will continue to run.

[0040] In the second cluster mode, the current task process communicates with all downstream task processes on all servers in the cluster, such as using persistent TCP connections. All processing nodes in the cluster operate in parallel and asynchronously. Specifically... Figure 4 As shown, communication connections can be established between task processes within server 1 and server 2, and between upstream and downstream task processes, even if they are located on different servers.

[0041] The second cluster mode has the capability to aggregate and distribute computations across the entire cluster, such as for alarm statistics or other applications requiring centralized data processing across the entire network. Each task process executes in parallel and asynchronously, possessing the characteristics of large-scale cluster parallel and asynchronous operation. Overall efficiency is high, and disaster recovery is good; if one server fails, the others continue to run.

[0042] In the third cluster mode, which is an extension of the first cluster mode, a target task process on the current server can be deployed to one or more other servers. The third cluster mode is similar to a hybrid of the first and second cluster modes. This mode allows the first cluster mode's cluster to achieve cross-server network aggregation or load balancing. The target task process, i.e., the specific computing node, can be determined from the computing nodes based on the task characteristics. Specifically... Figure 5 As shown, the task processes in service area 1 and server 2 establish communication connections with the downstream task processes located in server 3.

[0043] The third cluster mode combines the advantages of the first and second cluster modes, appropriately reducing cross-server transmission and making good use of the advantages of both modes.

[0044] In an exemplary embodiment of the present invention, the task process loads only one business module and runs one business module logic when performing distributed stream computing.

[0045] In this embodiment of the invention, a task process isolation mechanism is provided, meaning that only one task manager runs on each compute node in the cluster, responsible for creating and destroying task processes. The task manager creates a corresponding number of task processes based on the number of business modules to be loaded on the compute node, and loads the corresponding business modules according to the business module names passed to the task manager. Each task process runs as an independent process and loads only one specific business module, thus achieving task process isolation. Figure 6 As shown, when a certain business module encounters an anomaly, since the task process carrying that business module is an independent process, it will not affect other running task processes, and thus will not affect other businesses, achieving excellent anomaly isolation.

[0046] In an exemplary embodiment of the present invention, the task manager creates and simultaneously launches two identical task processes on the same server or different servers in the cluster, or creates and simultaneously launches two identical task processes on different servers in the cluster.

[0047] In this embodiment of the invention, a task disaster recovery mechanism is configured, including disaster recovery within the same server and disaster recovery between different servers. Disaster recovery within the same server refers to simultaneously creating and starting two identical task processes on the same server to prevent accidental operation by others or failure of one task process due to program abnormalities, so that another task can continue to be used. Disaster recovery between different servers mainly targets some processing nodes (task processes) in the third cluster mode and all processing nodes in the second cluster mode. If a downstream processing node fails, the current task node will distribute the message to the downstream processing nodes in the cluster across the entire network. When a task process on a server encounters a problem, the task manager on that server will automatically detect it and restart the corresponding task process.

[0048] In an exemplary embodiment of the present invention, the cluster manager, the task manager, and the task processes communicate indirectly through the cluster coordinator.

[0049] In this embodiment of the invention, the cluster manager, task manager, and task processes in the entire distributed cluster do not communicate directly with each other. Instead, they communicate indirectly through a global cluster coordinator using events, ensuring that the three components are highly independent and autonomous.

[0050] In an exemplary embodiment of the present invention, the cluster coordinator includes a ZooKeeper cluster coordinator.

[0051] In this embodiment of the invention, the ZooKeeper cluster coordinator is a distributed coordination service system composed of multiple ZooKeeper nodes (servers) used to solve problems such as data consistency, configuration management, distributed locks, service registration and discovery in distributed systems, thereby achieving high availability and strong consistency.

[0052] In an exemplary embodiment of the present invention, the cluster manager is also used to control the start or stop of the flow computing topology, as well as the expansion or contraction of the cluster; The task manager is also used to control the restart of the task process.

[0053] In this embodiment of the invention, the cluster manager is also used to control the start or stop of the streaming computing topology. Simultaneously, the cluster manager can also be used to control the expansion and contraction of the cluster. The task manager is also used to control the restart of task processes in disaster recovery scenarios.

[0054] In this embodiment of the invention, the existing distributed stream computing system suffers from problems such as a single cluster mode that cannot better and more comprehensively adapt to business scenarios, resource contention and logic pollution caused by shared processes of business tasks, leading to mutual interference between tasks. The technical solution provided by this invention allows for flexible configuration of the corresponding cluster mode according to the business scenario as needed. By limiting the task process to load only one business module and run only one business module logic, resource contention and logic pollution caused by shared processes of multiple business tasks are avoided, thereby increasing the stability of the entire computing cluster.

[0055] In this embodiment of the invention, flexible cluster mode configuration enables flexible support and configuration adaptation of multiple cluster modes, a feature not found in traditional distributed stream computing systems. It also allows for complex scenarios where different services can employ different cluster modes within the same stream computing topology, based on business characteristics. Business-level process isolation is employed, with each task process running as an independent process and loading only one specific business module, thus achieving task process isolation. When a business module malfunctions, because the task process carrying that module is an independent process, it will not affect other running task processes, and therefore will not impact other services, achieving excellent exception isolation. This results in efficient and stable distributed real-time stream processing, applicable to real-time data processing fields such as communications and finance.

[0056] Figure 7 This is a flowchart illustrating a distributed computing method based on a hybrid cluster, according to an exemplary embodiment. Figure 7 As shown in an exemplary embodiment, the distributed computing method based on a hybrid cluster includes steps 710 to 730, which are described in detail below.

[0057] Step 710: Obtain the stream computing topology configuration file and distribute the stream computing topology configuration file to the computing node where the task manager is located.

[0058] In this embodiment of the invention, after obtaining the stream computing topology configuration file, the cluster manager distributes the stream computing topology configuration file to all the computing nodes where the task managers of the cluster are located, and notifies all the task managers through the cluster that a new stream computing topology needs to be created.

[0059] Step 720: Control the task manager to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology.

[0060] In this embodiment of the invention, the task manager in the cluster determines whether to create a task process corresponding to the business component on the local machine based on the restrictions of the service where the business component is located in the stream computing topology configuration file. If it is necessary to create the task process, the corresponding task process is created, the dynamic library corresponding to the business component is loaded, and the connection between the task processes is established to obtain the stream computing topology.

[0061] Step 730: Based on the stream computing topology, control the task process to execute distributed stream computing as an independent process.

[0062] In this embodiment of the invention, the task manager records the name of the stream computing topology started on the local machine and the information of the task processes corresponding to all business components, and feeds back the corresponding stream computing topology startup information to the cluster coordinator, and then the cluster coordinator notifies the cluster manager to complete the creation of the stream computing topology.

[0063] Once the stream computing topology is created, task processes execute distributed stream computing as independent processes based on the stream computing topology.

[0064] In this embodiment of the invention, flexible cluster mode configuration enables flexible support and configuration adaptation of multiple cluster modes, a feature not found in traditional distributed stream computing systems. It also allows for complex scenarios where different services can employ different cluster modes within the same stream computing topology, based on business characteristics. Business-level process isolation is employed, with each task process running as an independent process and loading only one specific business module, thus achieving task process isolation. When a business module malfunctions, because the task process carrying that module is an independent process, it will not affect other running task processes, and therefore will not impact other services, achieving excellent exception isolation. This results in efficient and stable distributed real-time stream processing, applicable to real-time data processing fields such as communications and finance.

[0065] In an exemplary embodiment of the present invention, the step of connecting the task processes based on the stream computing topology configuration file to obtain the stream computing topology includes: The target cluster mode and the downstream task processes of the task process are determined based on the stream computing topology configuration file. Based on the target cluster mode and the downstream task processes, all the task processes are connected to obtain the stream computing topology.

[0066] In this embodiment of the invention, the target cluster mode and the downstream task process of the task process are determined based on the stream computing topology configuration file, the upstream and downstream relationship between the task processes is obtained, and the connection between the task processes is established according to the upstream and downstream relationship and the target cluster mode.

[0067] The target cluster mode includes at least one of the first cluster mode, the second cluster mode, and the third cluster mode; the first cluster mode is a mode in which each server in the cluster runs a complete data flow and processing flow computing topology; the second cluster module is a mode in which the current node and all downstream computing nodes of all servers in the cluster communicate with each other, and all computing nodes in the cluster execute computing in a parallel and asynchronous manner; the third cluster module is a mode in which specific computing nodes are deployed to one or more other servers besides the current server to realize the aggregation or distribution of data across the entire network, and is a mode that combines the first cluster mode and the second cluster mode.

[0068] The relevant introductions to the various cluster modes have been explained above and will not be repeated here.

[0069] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a distributed computing method based on a hybrid cluster. This method includes: obtaining a stream computing topology configuration file and distributing the stream computing topology configuration file to the computing nodes where the task manager is located. The task manager is controlled to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology; Based on the aforementioned stream computing topology, the task processes are controlled to execute distributed stream computing as independent processes.

[0070] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the distributed computing method based on a hybrid cluster provided by the above methods, the method including: obtaining a stream computing topology configuration file, and distributing the stream computing topology configuration file to the computing node where the task manager is located; The task manager is controlled to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology; Based on the aforementioned stream computing topology, the task processes are controlled to execute distributed stream computing as independent processes.

[0072] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the distributed computing method based on a hybrid cluster provided by the above methods, the method comprising: obtaining a stream computing topology configuration file and distributing the stream computing topology configuration file to the computing node where the task manager is located; The task manager is controlled to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology; Based on the aforementioned stream computing topology, the task processes are controlled to execute distributed stream computing as independent processes.

[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed computing system based on a hybrid cluster, characterized in that, include: The cluster manager is used to distribute streaming computing topology configuration files to the task manager; The cluster coordinator is used to collect the status of the task manager and the task processes, and send the status of the task manager to the cluster manager and the status of the task processes to the task manager. The task manager is used to create task processes based on the stream computing topology configuration file, establish stream computing topologies based on the task processes, and manage the task processes based on their status. The task processes serve as nodes in the stream computing topology and are used to perform distributed stream computing, running as independent processes.

2. The distributed computing system based on a hybrid cluster according to claim 1, characterized in that, The task process loads only one business module and runs only one business module logic when performing distributed stream computing.

3. The distributed computing system based on a hybrid cluster according to claim 1, characterized in that, The task manager can create and launch two identical task processes simultaneously on the same server in the cluster, or create and launch two identical task processes simultaneously on different servers in the cluster.

4. The distributed computing system based on a hybrid cluster according to claim 1, characterized in that, The cluster manager, the task manager, and the task processes communicate indirectly through the cluster coordinator.

5. The distributed computing system based on a hybrid cluster according to claim 1, characterized in that, The cluster manager creates a target cluster mode based on the stream computing topology configuration file, and distributes the stream computing topology configuration file to the task manager in the target cluster mode; wherein, the target cluster mode includes at least one of a first cluster mode, a second cluster mode, and a third cluster mode; The first cluster mode is a mode in which each server in the cluster runs a complete data flow and processing stream computing topology; The second cluster mode establishes communication connections between the current node and all downstream computing nodes on all servers in the cluster, and all computing nodes in the cluster execute computations in a parallel and asynchronous manner. The third cluster mode is a mode that deploys specific computing nodes to one or more other servers besides the current server to achieve data aggregation or distribution across the entire network. It is a combination of the first and second cluster modes.

6. The distributed computing system based on a hybrid cluster according to any one of claims 1 to 5, characterized in that, The cluster coordinator includes the ZooKeeper cluster coordinator.

7. The distributed computing system based on a hybrid cluster according to any one of claims 1 to 5, characterized in that, The cluster manager is also used to control the start or stop of the flow computing topology, as well as the expansion or reduction of the cluster. The task manager is also used to control the restart of the task process.

8. A distributed computing method based on hybrid clusters, characterized in that, include: Obtain the stream computing topology configuration file and distribute the stream computing topology configuration file to the compute node where the task manager is located; The task manager is controlled to create task processes according to the stream computing topology configuration file, and connect the task processes based on the stream computing topology configuration file to obtain the stream computing topology; Based on the aforementioned stream computing topology, the task processes are controlled to execute distributed stream computing as independent processes.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the distributed computing method based on a hybrid cluster as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed computing method based on a hybrid cluster as described in claim 8.

Images