Method and system for realizing Controller and Worker service combination under industrial Internet of Things

Abstract

The present application discloses a method and system for realizing the union of Controller and Worker services under the industrial Internet of Things, the method includes aggregating according to routing information, uniformly allocating Controller services and Worker services for the same routes; when adding a new route and starting a data transmission task, determining whether a Controller service with the same route is started, if so, updating the routing information of the started Controller service; if not, selecting idle Controller services and Worker services from the Controller service pool and Worker service pool respectively and assigning them to the new route, and starting the Controller service and Worker service corresponding to the new route. In the present application, when the Kafka cluster changes in the routing information, the Controller automatically updates the routing configuration without the intervention of the operation and maintenance personnel; automatically expanding new routing links according to routing rules to solve the scalability of Controller and Worker services.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission of industrial Internet of Things, and relates to a method and system for realizing the combination of Controller and Worker services under the industrial Internet of Things. Background Art

[0002] When the Industrial Internet of Things data transmission Controller and Worker services are running, as the number of Kafka cluster partitions and the number of Controller and Worker service deployments continue to increase, the overall Industrial Internet of Things data transmission system will encounter scalability and balance limitations, which will result in different Kafka clusters routing data transmission. Different Controller and Worker service clusters need to be deployed, causing Controller and Worker service redundancy and increasing operating costs. Summary of the invention

[0003] In order to solve the technical problems in the related technologies, this application provides a method and system for realizing the union of Controller and Worker services under the industrial Internet of Things. The technical solution is as follows:

[0004] In a first aspect, the present application provides a method for realizing the union of Controller and Worker services under the industrial Internet of Things, the method being applied to a system including a DC-Manager routing management service, a Controller service and a Worker service, the method comprising:

[0005] Creating a route aggregation method in the DC-Manager route management service, performing aggregation based on route information, and uniformly allocating Controller services and Worker services for the same routes, wherein the route information includes a Kafka cluster data source and a Kafka cluster target source;

[0006] A method for adding routing information is created in the DC-Manager routing management service. When a new route is added and a data transmission task is started, it is determined whether a Controller service with the same route is started. If a Controller service with the same route is started, the routing information of the started Controller service is updated; if no Controller service with the same route is started, an idle Controller service and a Worker service are selected from the Controller service pool and the Worker service pool respectively and assigned to the new route, and the Controller service and the Worker service corresponding to the new route are started.

[0007] Optionally, the method further comprises:

[0008] In the DC-Manager routing management service, the heartbeats of the Controller service and the Worker service are sensed from Zookeeper through the Helix service;

[0009] When the Controller service or Worker service is detected to be offline, the cluster rebalancing mechanism is triggered.

[0010] Optionally, the triggering of the cluster rebalancing mechanism includes:

[0011] When a Controller service is detected to be offline, an idle Controller service is selected from the Controller service pool and allocated to the route where the offline Controller service is located;

[0012] When a Worker service is detected to be offline, the Controller service is notified to allocate the corresponding amount of data processed by the remaining Worker services.

[0013] Optionally, the method further comprises:

[0014] When the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method is added for the Controller service to obtain Kafka cluster information. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

[0015] Optionally, the method further comprises:

[0016] When the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, modify the Worker service routing information request interface to receive the push from the DC-Manager routing management service or the Controller service to obtain the routing information.

[0017] In a second aspect, the present application provides a system for realizing the union of Controller and Worker services under the industrial Internet of Things, wherein:

[0018] The DC-Manager routing management service creates a routing aggregation method, aggregates according to routing information, and uniformly allocates Controller services and Worker services for the same routes. The routing information includes the Kafka cluster data source and the Kafka cluster target source.

[0019] The DC-Manager routing management service creates a method for adding routing information. When a new route is added and a data transmission task is started, it is determined whether a Controller service with the same route is started. If a Controller service with the same route is started, the routing information of the started Controller service is updated; if no Controller service with the same route is started, an idle Controller service and Worker service are selected from the Controller service pool and the Worker service pool respectively and assigned to the new route, and the Controller service and Worker service corresponding to the new route are started.

[0020] Optionally, the DC-Manager routing management service senses the heartbeats of the Controller service and the Worker service from Zookeeper through the Helix service;

[0021] The DC-Manager routing management service triggers a cluster rebalancing mechanism when it senses that the Controller service or the Worker service is offline.

[0022] Optionally, when the DC-Manager routing management service senses that a Controller service is offline, it selects an idle Controller service from the Controller service pool and allocates it to the route where the offline Controller service is located;

[0023] When the DC-Manager routing management service senses that the Worker service is offline, it notifies the Controller service to allocate the corresponding amount of data processed by the remaining Worker services.

[0024] Optionally, when the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method for the Controller service to obtain Kafka cluster information is added. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

[0025] Optionally, when the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, the Worker service modifies the routing information acquisition request interface and receives the push from the DC-Manager routing management service or the Controller service to acquire routing information.

[0026] According to the above technical solution, the present application can at least achieve the following beneficial effects:

[0027] When the Kafka cluster changes in the routing information, the Controller automatically updates the routing configuration without the need for operation and maintenance personnel to intervene; it automatically expands new routing links based on routing rules to solve the scalability of Controller and Worker services.

[0028] In addition, when the Controller or Worker service crashes or goes offline, the cluster rebalancing mechanism is automatically triggered to solve the balance of the Controller and Worker services.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0031] Figure 1 It is a flowchart of a method for realizing the union of Controller and Worker services under the industrial Internet of Things provided in one embodiment of the present application;

[0032] Figure 2 It is a schematic diagram of the framework of a system for realizing the union of Controller and Worker services under the industrial Internet of Things provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0034] Figure 1The flowchart of a method for realizing the union of Controller and Worker services under the industrial Internet of Things provided in one embodiment of the present application is applied to a system including a DC-Manager routing management service, a Controller service and a Worker service. The method may include the following steps:

[0035] Step 101, create a route aggregation method in the DC-Manager route management service, aggregate according to route information, and uniformly allocate Controller services and Worker services for the same routes;

[0036] The routing information mentioned here may include the Kafka cluster data source and the Kafka cluster target source. The Kafka cluster data source includes the source topic, and the Kafka cluster target source includes the target topic.

[0037] Step 102: Create a method for adding routing information in the DC-Manager routing management service. When adding a new route and starting a data transmission task, determine whether a Controller service with the same route is started.

[0038] Step 103: If a Controller service with the same route is started, update the route information of the started Controller service;

[0039] Step 104: If no Controller service with the same route is started, an idle Controller service and Worker service are selected from the Controller service pool and Worker service pool respectively and allocated to the newly added route, and the Controller service and Worker service corresponding to the newly added route are started.

[0040] In actual applications, the Controller service and Worker service usually need to send heartbeat data to Zookeeper. At this time, the DC-Manager routing management service can perceive the heartbeats of the Controller service and Worker service from Zookeeper through the Helix service. When it senses that the Controller service or Worker service is offline, it triggers the cluster rebalancing mechanism.

[0041] In a possible implementation of triggering a cluster rebalancing mechanism, when sensing that a Controller service is offline, an idle Controller service is selected from a Controller service pool and assigned to the route where the offline Controller service is located.

[0042] The Controller service pool mentioned here usually contains multiple Controller services, some of which are in the startup state and some of which are in the idle state. When a Controller service is detected to be down or offline, an idle Controller service can be selected from the Controller service pool and assigned to the route where the offline Controller service is located.

[0043] Similarly, when a Worker service is detected to be offline, the Controller service is notified to allocate the corresponding amount of data processed by the remaining Worker services.

[0044] The Worker service pool mentioned here usually contains multiple Worker services. When a Worker service is detected to be down or offline, the Controller service is notified to allocate the corresponding amount of data processed by the remaining Worker services.

[0045] For example, the Controller service originally allocates three sets of data to three Worker services. When one of the Worker services goes down, two sets of data can be allocated to one of the remaining two Worker services, and one set of data can be allocated to the other of the remaining two Worker services.

[0046] In actual applications, there may be situations where the Topic information of the Kafka cluster changes. In this case, it is necessary to correct the routing information related to the Topic in the Controller service.

[0047] In one possible implementation, when the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method is added for the Controller service to obtain Kafka cluster information. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

[0048] For the Worker service, when the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, the Worker service obtains the routing information request interface and receives the push from the DC-Manager routing management service or the Controller service to obtain the routing information.

[0049] In summary, the method for realizing the combination of Controller and Worker services under the industrial Internet of Things provided by this application is that when the Kafka cluster in the routing information changes, the Controller automatically updates the routing configuration without the need for operation and maintenance personnel to intervene; new routing links are automatically expanded according to routing rules to solve the scalability of Controller and Worker services.

[0050] In addition, when the Controller or Worker service crashes or goes offline, the cluster rebalancing mechanism is automatically triggered to solve the balance of the Controller and Worker services.

[0051] Figure 2 This is a schematic diagram of a framework of a system for realizing the union of Controller and Worker services under the industrial Internet of Things provided in one embodiment of the present application. The system may include a DC-Manager routing management service, a Controller service, and a Worker service.

[0052] Multiple Controller services form a Controller service resource pool, that is, the Controller service resource pool contains multiple Controller services. However, in actual applications, some Controller services in the Controller service resource pool are started, and some Controller services are idle.

[0053] Similarly, multiple Worker services form a Worker service resource pool, that is, the Worker service resource pool contains multiple Worker services. In actual applications, some Worker services in the Worker service resource pool are started, and some Worker services are idle.

[0054] In actual applications, the Controller service and the Worker service interact with each other through the Helix service.

[0055] Typically, a Controller service can correspond to at least one Worker service, for example, two Worker services or three Worker services.

[0056] The DC-Manager routing management service creates a routing aggregation method, aggregates according to routing information, and uniformly allocates Controller services and Worker services for the same routes. The routing information includes a Kafka cluster data source and a Kafka cluster target source.

[0057] The DC-Manager routing management service creates a method for adding routing information. When a new route is added and a data transmission task is started, it is determined whether a Controller service with the same route is started.

[0058] If a Controller service with the same route is started, the routing information of the started Controller service is updated; if no Controller service with the same route is started, idle Controller services and Worker services are selected from the Controller service pool and Worker service pool respectively and assigned to the newly added route, and the Controller service and Worker service corresponding to the newly added route are started.

[0059] In a possible implementation, the DC-Manager routing management service senses the heartbeats of the Controller service and the Worker service from Zookeeper through the Helix service; when the DC-Manager routing management service senses that the Controller service or the Worker service is offline, it triggers a cluster rebalancing mechanism.

[0060] Optionally, when sensing that a Controller service is offline, the DC-Manager routing management service selects an idle Controller service from the Controller service pool and allocates it to the route where the offline Controller service is located.

[0061] When the DC-Manager routing management service senses that the Worker service is offline, it notifies the Controller service to allocate the corresponding amount of data processed by the remaining Worker services.

[0062] In another possible implementation, when the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method for the Controller service to obtain Kafka cluster information is added. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

[0063] Optionally, when the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, the Worker service modifies the routing information acquisition request interface and receives the push from the DC-Manager routing management service or the Controller service to acquire routing information.

[0064] To summarize, the system provided by this application realizes the union of Controller and Worker services under the industrial Internet of Things. When the Kafka cluster in the routing information changes, the Controller automatically updates the routing configuration without the intervention of the operation and maintenance personnel; new routing links are automatically expanded according to the routing rules to solve the scalability of the Controller and Worker services.

[0065] In addition, when the Controller or Worker service crashes or goes offline, the cluster rebalancing mechanism is automatically triggered to solve the balance of the Controller and Worker services.

[0066] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention invented herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not invented by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0067] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for realizing the combination of Controller and Worker services in industrial Internet of Things, characterized in that: The method is applied to a system including a DC-Manager routing management service, a Controller service, and a Worker service, and the method includes: Creating a route aggregation method in the DC-Manager route management service, performing aggregation based on route information, and uniformly allocating Controller services and Worker services for the same routes, wherein the route information includes a Kafka cluster data source and a Kafka cluster target source; A method for adding routing information is created in the DC-Manager routing management service. When a new route is added and a data transmission task is started, it is determined whether a Controller service with the same route is started. If a Controller service with the same route is started, the routing information of the started Controller service is updated; if no Controller service with the same route is started, an idle Controller service and a Worker service are selected from the Controller service pool and the Worker service pool and assigned to the new route, and the Controller service and the Worker service corresponding to the new route are started; When a Controller service is detected to be offline, an idle Controller service is selected from the Controller service pool and allocated to the route where the offline Controller service is located; When a Worker service is detected to be offline, the Controller service is notified to allocate the corresponding amount of data processed by the remaining Worker services.

2. The method according to claim 1, characterized in that The method further comprises: In the DC-Manager routing management service, the heartbeats of the Controller service and the Worker service are sensed from Zookeeper through the Helix service; When the Controller service or Worker service is detected to be offline, the cluster rebalancing mechanism is triggered.

3. The method according to claim 1, characterized in that The method further comprises: When the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method is added for the Controller service to obtain Kafka cluster information. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

4. The method according to claim 1, characterized in that: The method further comprises: When the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, modify the Worker service routing information request interface to receive the push from the DC-Manager routing management service or the Controller service to obtain the routing information.

5. A system for realizing the union of Controller and Worker services under the industrial Internet of Things, characterized in that: The system includes DC-Manager routing management service, Controller service and Worker service, wherein: The DC-Manager routing management service creates a routing aggregation method, aggregates according to routing information, and uniformly allocates Controller services and Worker services for the same routes. The routing information includes the Kafka cluster data source and the Kafka cluster target source. The DC-Manager routing management service creates a method for adding routing information. When adding a new route and starting a data transmission task, it is determined whether a Controller service with the same route is started. If a Controller service with the same route is started, the routing information of the started Controller service is updated; if no Controller service with the same route is started, an idle Controller service and a Worker service are selected from the Controller service pool and the Worker service pool respectively and assigned to the new route, and the Controller service and the Worker service corresponding to the new route are started; When the DC-Manager routing management service senses that a Controller service is offline, it selects an idle Controller service from the Controller service pool and assigns it to the route where the offline Controller service is located; When the DC-Manager routing management service senses that the Worker service is offline, it notifies the Controller service to allocate the corresponding amount of data processed by the remaining Worker services.

6. The system according to claim 5, characterized in that The DC-Manager routing management service senses the heartbeats of the Controller service and the Worker service from Zookeeper through the Helix service; The DC-Manager routing management service triggers a cluster rebalancing mechanism when it senses that the Controller service or the Worker service is offline.

7. The system according to claim 5, characterized in that When the Controller service is selected by the DC-Manager routing management service and started according to the current routing information, a method for the Controller service to obtain Kafka cluster information is added. When the Topic information of the Kafka cluster changes, the routing information related to the Topic in the Controller service is corrected to balance the amount of data processed by multiple Worker services.

8. The system according to claim 5, characterized in that When the Worker service is selected by the DC-Manager routing management service and started according to the current routing information, the Worker service obtains routing information request interface, receives the push from the DC-Manager routing management service or the Controller service to obtain routing information.

Images