Pre-acquisition method and system for dynamic capacity expansion based on edge cluster system

Abstract

The invention relates to the technical field of data acquisition, and discloses a front acquisition method and system based on dynamic capacity expansion under an edge cluster system.The method comprises the steps that a front management node distributes channels to specified acquisition units, and an acquisition unit container elastically expands and contracts according to the number of the channels; distributing a collection task of a channel table to the collection unit container; and managing and monitoring the acquisition unit container. According to the invention, the pre-collection based on dynamic capacity expansion, such as pre-container elastic expansion, collection task distribution, container management and monitoring, can be realized, and a pre-collection technical support satisfying dynamic capacity expansion is provided for the construction of a cloud edge fusion intelligent scheduling operation platform (CEP) edge cluster system.

Description

Technical Field

[0001] The present invention relates to the field of data collection technology, and in particular to a pre-collection method based on dynamic expansion in an edge cluster system. Background Art

[0002] At present, the access of massive new energy sources has increased the frequency of data collection, and the amount of interactive data is huge, which has posed challenges to the data processing and storage capabilities of the pre-collection. The system may have bottlenecks when processing high-concurrency tasks, affecting the real-time and accuracy of data collection, and the system's scalability may be insufficient. The construction requirements of the edge cluster system of the cloud-edge fusion intelligent scheduling and operation platform (CEP) urgently need to carry out research and technical implementation of pre-collection methods and systems based on dynamic expansion. Summary of the invention

[0003] The present invention aims to at least solve the technical problems that the system may have bottlenecks when processing high-concurrency tasks, affecting the real-time and accuracy of data collection, and the system's scalability may be insufficient.

[0004] To this end, an object of the present invention is to propose a pre-collection method based on dynamic expansion in an edge cluster system, comprising the following steps:

[0005] The front-end management node allocates the channel to the specified collection unit, and the collection unit container is elastically scaled according to the number of the channels;

[0006] Allocating the collection tasks of the channel table to the collection unit container;

[0007] Manage and monitor collection unit containers.

[0008] Furthermore, the front-end management node allocates the channel to the designated collection unit, and the collection unit container is elastically scaled according to the number of the channels, including:

[0009] Get the total number of channels in the system;

[0010] Get the number of running collection unit containers;

[0011] The number of required collection unit containers is calculated, the number of required collection unit containers is compared with the number of the collection unit containers that have been in operation, and the collection unit containers are recycled or expanded according to the comparison result.

[0012] Furthermore, the calculation formula for the number of collection unit containers required is as follows:

[0013] The number of collection unit containers required = the total number of channels in the system / the number of single container management channels.

[0014] Further, calculating the number of required collection unit containers, comparing the number of required collection unit containers with the number of the already running collection unit containers, and recycling or expanding the collection unit containers according to the comparison result, including:

[0015] When the number of required collection unit containers is less than the number of the collection unit containers that have been in operation, the channels of the excess collection unit containers that have been in operation are transferred and evenly distributed to other collection unit containers, and then the excess collection unit containers that have been in operation are recycled;

[0016] When the number of required collection unit containers is greater than the number of the running collection unit containers, the collection unit container expansion is performed, and the redundant channels are allocated to the newly expanded collection unit containers.

[0017] Further, calculating the number of required collection unit containers, comparing the number of required collection unit containers with the number of the already running collection unit containers, and recycling or expanding the collection unit containers according to the comparison result, including:

[0018] The elastic expansion program runs on the front-end management node, and the collection unit container is recycled or expanded and contracted through the elastic expansion program.

[0019] Further, allocating the collection task of the channel table to the container includes:

[0020] Each record in the channel table corresponds to a collection task, and multiple collection tasks are evenly distributed to corresponding collection unit containers; the task allocation program runs on the front-end management node, is responsible for the scheduling of all collection tasks, generates allocation results, and writes the allocation results into the channel table. The collection unit container loads the channel belonging to the collection unit container according to the allocation results.

[0021] Furthermore, the collection unit container is managed and monitored, including:

[0022] Collect information about collection unit containers running on each front-end node;

[0023] Collect channel information and count the number of online channels and channel flow in each collection unit container;

[0024] Based on the previous node, the total number of online channels and the total communication traffic are counted.

[0025] The present invention provides a pre-collection system based on dynamic expansion in an edge cluster system, which is used to implement the pre-collection method based on dynamic expansion in an edge cluster system, including:

[0026] The front-end management module allocates the channel to the designated acquisition unit;

[0027] An extended recovery module enables the collection unit container to be elastically expanded and contracted according to the number of channels;

[0028] An allocation module allocates the collection tasks of the channel table to the collection unit containers;

[0029] The monitoring module is used to manage and monitor the collection unit container.

[0030] The present invention provides a pre-collection method and system based on dynamic expansion in an edge cluster system, which has the following beneficial effects:

[0031] The present invention discloses a front-end collection method and system based on dynamic capacity expansion in an edge cluster system. The front-end management node monitors the operating status of the processing unit in real time, and allocates the channel to the designated collection unit according to the load balancing algorithm. The collection unit only loads the front-end channel model allocated to the unit, realizes network communication with the station end and protocol message parsing and processing functions, and sends the processed data to background applications such as SCADA through a platform message bus. The collection unit adopts containerized deployment and elastically expands according to the number of accessed channels, and finally realizes elastic expansion of the front container, collection task allocation, container management and monitoring, and other front-end collection based on dynamic capacity expansion, so as to provide front-end collection technical support that meets dynamic capacity expansion for the construction of an edge cluster system of a cloud-edge fusion intelligent scheduling and operation platform (CEP). BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 This is a flow chart of a pre-collection method based on dynamic expansion in an edge cluster system according to the present invention;

[0034] Figure 2 It is a schematic diagram of a front-end acquisition system with dynamic expansion based on an edge cluster system according to the present invention. DETAILED DESCRIPTION

[0035] Various aspects and features of the present invention are described herein with reference to the accompanying drawings.

[0036] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of the embodiments. Other modifications within the scope and spirit of the present invention will occur to those skilled in the art.

[0037] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the general description of the invention given above and the detailed description of the embodiments given below, serve to explain the principles of the invention.

[0038] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the accompanying drawings.

[0039] It should also be understood that, although the invention has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics described in the claims and are therefore within the scope of protection defined thereby.

[0040] The above and other aspects, features and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0041] Specific embodiments of the present invention are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments claimed are merely examples of the present invention, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present invention with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present invention in a variety of ways with substantially any suitable detailed structure.

[0042] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0043] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0044] Example

[0045] like Figure 1 and Figure 2As shown, this embodiment provides a pre-collection method based on dynamic expansion in an edge cluster system, including the following steps:

[0046] Step S1: The front-end management node allocates the channel to the designated collection unit, and the collection unit container is elastically scaled according to the number of the channels;

[0047] Step S2: Allocating the collection tasks of the channel table to the collection unit container;

[0048] Step S3: Manage and monitor the collection unit container.

[0049] According to another specific embodiment of the present invention, based on step S1, the front-end management node allocates the channel to the designated collection unit, and the collection unit container is elastically scaled according to the number of the channels, including:

[0050] Get the total number of channels in the system;

[0051] Get the number of running collection unit containers;

[0052] The number of required collection unit containers is calculated, the number of required collection unit containers is compared with the number of the collection unit containers that have been in operation, and the collection unit containers are recycled or expanded according to the comparison result.

[0053] According to another specific embodiment of the present invention, based on the above embodiment, the calculation formula for the number of collection unit containers required is as follows:

[0054] The number of collection unit containers required = the total number of channels in the system / the number of single container management channels.

[0055] According to another specific embodiment of the present invention, based on the above embodiment, the number of required collection unit containers is calculated, the number of required collection unit containers is compared with the number of the collection unit containers that have been in operation, and according to the comparison result, the collection unit containers are recycled or expanded, including:

[0056] When the number of required collection unit containers is less than the number of the collection unit containers that have been in operation, the channels of the excess collection unit containers that have been in operation are transferred and evenly distributed to other collection unit containers, and then the excess collection unit containers that have been in operation are recycled;

[0057] When the number of required collection unit containers is greater than the number of the running collection unit containers, the collection unit container expansion is performed, and the redundant channels are allocated to the newly expanded collection unit containers.

[0058] According to another specific embodiment of the present invention, based on the above embodiment, the number of required collection unit containers is calculated, the number of required collection unit containers is compared with the number of the collection unit containers that have been in operation, and according to the comparison result, the collection unit containers are recycled or expanded, including:

[0059] The elastic expansion program runs on the front-end management node, and the collection unit container is recycled or expanded and contracted through the elastic expansion program.

[0060] According to another specific embodiment of the present invention, on the basis of step S2, allocating the collection task of the channel table to the container includes:

[0061] Each record in the channel table corresponds to a collection task, and multiple collection tasks are evenly distributed to corresponding collection unit containers; the task allocation program runs on the front-end management node, is responsible for the scheduling of all collection tasks, generates allocation results, and writes the allocation results into the channel table. The collection unit container loads the channel belonging to the collection unit container according to the allocation results.

[0062] According to another specific embodiment of the present invention, on the basis of step S3, the collection unit container is managed and monitored, including:

[0063] Collect the collection unit container information running on each front-end node, which can intuitively display the transfer status of the front-end container on each node;

[0064] Collect channel information and count the number of online channels and channel flow in each collection unit container;

[0065] Based on the previous node, the total number of online channels and the total communication traffic are counted.

[0066] The present invention provides a pre-collection system based on dynamic expansion in an edge cluster system, which is used to implement the pre-collection method based on dynamic expansion in an edge cluster system, including:

[0067] The front-end management module allocates the channel to the designated acquisition unit;

[0068] An extended recovery module enables the collection unit container to be elastically expanded and contracted according to the number of channels;

[0069] An allocation module allocates the collection tasks of the channel table to the collection unit containers;

[0070] The monitoring module is used to manage and monitor the collection unit container.

[0071] The present invention provides a pre-collection method and system based on dynamic expansion in an edge cluster system, which has the following beneficial effects:

[0072] The present invention discloses a front-end collection method and system based on dynamic capacity expansion in an edge cluster system. The front-end management node monitors the operating status of the processing unit in real time, and allocates the channel to the designated collection unit according to the load balancing algorithm. The collection unit only loads the front-end channel model allocated to the unit, realizes network communication with the station end and protocol message parsing and processing functions, and sends the processed data to background applications such as SCADA through a platform message bus. The collection unit adopts containerized deployment and elastically expands according to the number of accessed channels, and finally realizes elastic expansion of the front container, collection task allocation, container management and monitoring, and other front-end collection based on dynamic capacity expansion, so as to provide front-end collection technical support that meets dynamic capacity expansion for the construction of an edge cluster system of a cloud-edge fusion intelligent scheduling and operation platform (CEP).

[0073] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0074] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0075] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0076] Multiple embodiments of the present disclosure are described in detail above, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should all fall within the scope of protection required by the present disclosure.

Claims

1. A pre-collection method based on dynamic expansion in an edge cluster system, characterized in that: The following steps are involved: The front-end management node allocates the channel to the specified collection unit, and the collection unit container is elastically scaled according to the number of the channels; Allocating the collection tasks of the channel table to the collection unit container; Manage and monitor collection unit containers.

2. The pre-collection method based on dynamic expansion in edge cluster system according to claim 1 is characterized in that: The front-end management node allocates the channel to the specified collection unit, and the collection unit container is elastically scaled according to the number of channels, including: Get the total number of channels in the system; Get the number of running collection unit containers; The number of required collection unit containers is calculated, the number of required collection unit containers is compared with the number of the collection unit containers that have been in operation, and the collection unit containers are recycled or expanded according to the comparison result.

3. The pre-collection method based on dynamic expansion in edge cluster system according to claim 2 is characterized in that: The number of collection unit containers required is calculated as follows: The number of collection unit containers required = the total number of channels in the system / the number of single container management channels.

4. The pre-collection method based on dynamic expansion in edge cluster system according to claim 2 is characterized in that: Calculating the number of required collection unit containers, comparing the number of required collection unit containers with the number of the already running collection unit containers, and recycling or expanding the collection unit containers according to the comparison result, including: When the number of required collection unit containers is less than the number of the collection unit containers that have been in operation, the channels of the excess collection unit containers that have been in operation are transferred and evenly distributed to other collection unit containers, and then the excess collection unit containers that have been in operation are recycled; When the number of required collection unit containers is greater than the number of the running collection unit containers, the collection unit container expansion is performed, and the redundant channels are allocated to the newly expanded collection unit containers.

5. The pre-collection method based on dynamic expansion in edge cluster system according to claim 2 is characterized in that: Calculating the number of required collection unit containers, comparing the number of required collection unit containers with the number of the already running collection unit containers, and recycling or expanding the collection unit containers according to the comparison result, including: The elastic expansion program runs on the front-end management node, and the collection unit container is recycled or expanded and contracted through the elastic expansion program.

6. The pre-collection method based on dynamic expansion in edge cluster system according to claim 3 is characterized in that: Allocating the collection task of the channel table to the container includes: Each record in the channel table corresponds to a collection task, and multiple collection tasks are evenly distributed to corresponding collection unit containers; the task allocation program runs on the front-end management node, is responsible for the scheduling of all collection tasks, generates allocation results, and writes the allocation results into the channel table. The collection unit container loads the channel belonging to the collection unit container according to the allocation results.

7. The pre-collection method based on dynamic expansion in edge cluster system according to claim 4 is characterized in that: Management and monitoring of collection unit containers, including: Collect information about collection unit containers running on each front-end node; Collect channel information and count the number of online channels and channel flow in each collection unit container; Based on the previous node, the total number of online channels and the total communication traffic are counted.

8. A front-end acquisition system with dynamic expansion based on edge cluster system, characterized in that: The method for implementing the front-end collection method based on dynamic expansion in the edge cluster system according to any one of claims 1 to 7 comprises: The front-end management module allocates the channel to the designated acquisition unit; An extended recovery module enables the collection unit container to be elastically expanded and contracted according to the number of channels; An allocation module allocates the collection tasks of the channel table to the collection unit containers; The monitoring module is used to manage and monitor the collection unit container.

Images