Data acquisition and control engine system and method based on cloud edge-end architecture

Abstract

The invention provides a data acquisition and control engine system and method based on a cloud side end architecture, and the system comprises a service end which is used for providing a task list; the controller is used for monitoring the heartbeat change of the control center and modifying a task allocation scheme according to the task information change and the heartbeat change of the collector; the collector is used for collecting data information and reporting the operation state to the control center; the control center is used for deploying the controllers and the collectors, recording task lists needing to be distributed, selecting suitable groups according to identifiers of tasks and generating task information; and readjusting and distributing tasks to the collector according to the running state reported by the collector. According to the method, the high dependence of a large-scale data acquisition task on basic middleware under a micro-service architecture is greatly reduced, and the resource consumption of software use is reduced; in addition, due to the lack of a large number of middleware interaction and quantifiable performance estimation standards, daily operation and maintenance work is simpler, and the method has good practicability.

Description

Technical Field

[0001] The present invention relates to the field of task scheduling and orchestration, and in particular, to a data acquisition and control engine system and method based on a cloud-edge-terminal architecture. Background Art

[0002] Modern IT systems and devices are huge in number, and data sources are complex and diverse, including servers, network devices, storage systems, applications, etc. This makes it difficult for a single node to efficiently handle massive monitoring data collection tasks. Monitoring systems usually require real-time collection and analysis of system operating status in order to detect and handle faults in a timely manner.

[0003] However, the types of monitoring data and the collection methods vary for different devices, services, and applications, such as through SNMP, HTTP API, log files, etc., which further increases the complexity of management. During the data collection process, the priority and resource requirements of tasks also vary. Data collection for some devices requires higher frequency and computing resources, while other tasks are relatively lightweight. Monitoring data collection tasks are key to ensuring the stable operation of the system, so high availability is particularly important. As the monitoring scope continues to expand, the number of data sources and collection tasks continues to increase, and the existing collection solutions are difficult to meet the scalability and real-time requirements of the system. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a data acquisition and control engine system and method based on a cloud-edge-end architecture.

[0005] A data acquisition and control engine system based on a cloud-edge-end architecture provided by the present invention includes:

[0006] The business side is used to provide a task list;

[0007] The controller is used to monitor the heartbeat changes of the control center and modify the task allocation plan according to the changes in task information and the heartbeat changes of the collector;

[0008] Collector, used to collect data information and report the operation status to the control center;

[0009] The control center is used to coordinate controllers and collectors, record the task list that needs to be distributed, select the applicable group according to the task identifier, and generate task information; according to the operating status reported by the collector, readjust the distribution task list.

[0010] Preferably, it also includes a data consumption end for receiving the results generated by the collector.

[0011] Preferably, the service end can customize task grouping, and when grouping, the groups are split according to different network segments in the network scenario.

[0012] Preferably, the control center includes searching for matching groups within a preset value domain in the task information, and is applicable to network segment isolation in network scenarios and collector resource isolation in cloud resources.

[0013] Preferably, the control center also includes:

[0014] When distributing tasks or not distributing tasks, a minimum impact strategy is adopted; the minimum impact strategy includes not changing the result of the last task distribution, so that the change of the task received by the collector is minimal;

[0015] Scenarios when tasks are not distributed include:

[0016] Intra-group rebalancing: if the collector in a group changes, the tasks in the group will be adjusted;

[0017] Global rebalancing: if the grouping conditions change or the task list changes, all grouped tasks will be regrouped.

[0018] Preferably, when the collector reports the running status, if the status is unavailable, the control center takes back the assigned tasks and redistributes the tasks according to the currently available collector information.

[0019] Preferably, multiple types of collectors are supported, and each collector is of only one type; the collector can use multiple development languages ​​to implement an interactive interface with the platform.

[0020] A data acquisition and control engine method based on a cloud-edge-device architecture provided by the present invention includes:

[0021] Step S1: Recording a task list provided by the service end; the task list includes a task identifier;

[0022] Step S2: The control center selects an applicable group according to the task identifier;

[0023] Step S3: After the collector is connected to the platform, it is registered and regularly reports its health status to the control center;

[0024] Step S4: Instruct the control center to readjust the distribution task list according to the grouping status and the operating status reported by the collector.

[0025] Preferably, the control center adopts a minimum impact strategy when distributing tasks and when not distributing tasks; the minimum impact strategy includes not changing the result of the last task distribution, so that the task received by the collector has the minimum change;

[0026] Scenarios when tasks are not distributed include:

[0027] Intra-group rebalancing: if the collector in a group changes, the tasks in the group will be adjusted;

[0028] Global rebalancing: if the grouping conditions change or the task list changes, all grouped tasks will be regrouped.

[0029] Preferably, when the collector reports the running status, if the status is unavailable, the control center is instructed to take back the assigned tasks and redistribute the tasks according to the currently available collector information.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention utilizes the data acquisition and control engine to connect to various temporary or permanent task lists from the cloud data source in real time, and realizes dynamic task scheduling and adaptive rebalancing tasks; the edge collector can realize stand-alone operation or unified control of the receiving line by configuring its own operating mode, and executes the acquisition task and outputs the data results, thus meeting the needs of hot-swappable and reusable acquisition modules.

[0032] 2. The present invention greatly reduces the high dependence of large-scale data collection tasks on basic middleware under the microservice architecture, and reduces the resource consumption of software use; and due to the lack of a large number of middleware interactions and quantifiable performance estimation standards, daily operation and maintenance work is simpler and has better practicality.

[0033] 3. The present invention provides a set of development-friendly and flexible access standards, and due to its rich scalability, it makes the access of new services very simple and easy, and makes the system have better stability.

[0034] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0036] Figure 1 Schematic diagram of the architecture of the data acquisition and control engine system of the present invention.

[0037] Figure 2 This is an execution flow chart of the task of the present invention when it is issued.

[0038] Figure 3 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0040] Reference Figure 1 As shown, a data acquisition and control engine system based on a cloud-edge-end architecture includes:

[0041] The business end is used to provide a task list, which can customize task grouping and divide the task into groups according to different network segments in the network scenario.

[0042] The controller is used to monitor the heartbeat changes of the control center and modify the task allocation plan according to the changes in task information and the heartbeat changes of the collector;

[0043] The collector is used to collect data information and report the running status to the control center. When reporting the running status, if the status is unavailable, the control center will take back the assigned tasks and redistribute the tasks based on the currently available collector information.

[0044] The collector can use a variety of development languages ​​to implement interactive interfaces with the platform.

[0045] The control center is used to deploy controllers and collectors, record the task list that needs to be distributed, select the applicable group according to the task identifier, and generate task information; according to the operating status reported by the collector, readjust and distribute the task. Find matching groups within the preset value range in the task information, and apply to network segment isolation in network scenarios and collector resource isolation under cloud resources. When distributing tasks and not distributing tasks, it adopts the minimum impact strategy; the minimum impact strategy includes not changing the result of the last task distribution, so that the changes in the tasks received by the collector are minimized; the scenarios when not distributing tasks include: intra-group rebalancing, if the collector in a group changes, the tasks in the group are adjusted; global rebalancing, if the grouping conditions change or the task list changes, the tasks of all groups are regrouped.

[0046] The data consumer is used to receive the results generated by the collector.

[0047] The system supports multiple types of collectors, and each collector belongs to only one type.

[0048] The above are basic embodiments of the present invention. The technical solution of the present invention is further described below through two preferred embodiments.

[0049] Example 1

[0050] Reference Figure 2 and Figure 3 As shown, a data acquisition and control engine method based on a cloud-edge-end architecture includes:

[0051] The control center first needs to record the task list that needs to be distributed. The task list is provided by the upper-level business end. When the control center obtains the tasks that need to be distributed, it will select the applicable group according to the task identifier.

[0052] The business end can customize task grouping and select the grouping according to actual needs. For example, in the network scenario, different groups will be divided according to different network segments, such as 192.168.1.0 / 24 grouping and 10.200.1.0 / 24 grouping, or in the cloud resource scenario, grouping will be based on tenants and cloud accounts. The grouping conditions can be selected by yourself, and the control center will search for matching groups within the specified value range of the task information. This function can be used to meet the classic scenarios of network segment isolation in network scenarios and collector resource isolation in cloud resources.

[0053] When a collector needs to access the platform as part of a dispatchable system, it must first register with the control center and then periodically report its health status. The status of the collector directly affects the control center's selection of collectors when distributing tasks. If the collector is unavailable in the control center, the assigned tasks will be reclaimed and redistributed based on the currently available collector information.

[0054] Because of the sinking of collection tasks, each distribution or rebalancing may interfere with the progress of task execution, so the control center will adopt the minimum impact strategy when balancing and distributing, that is, try not to change the result of the last task distribution as much as possible, so that the changes in the tasks received by the collector are minimal. Rebalancing is based on the changing scenarios of status information and grouping information, and fixes the balancing scenarios into two: intra-group rebalancing and global rebalancing. As the name implies, if only the collector in a group changes, only the tasks in the group need to be adjusted; if the grouping conditions change or the task list changes, all grouped tasks need to be regrouped.

[0055] Based on existing experience, there must be many types of collectors. There will be a collector implementation for each standard of collection and docking resources. The needs of the business end will naturally be naturally coupled with different collectors, but the reusability of the collector should still be achieved as much as possible. At this time, you can start with the interactive protocol of the collector, such as snmp, ssh, http, etc., and make it compatible in data format instead of directly integrating business information into the collector. The type of collector can be customized, whether it is a business logic type or a standard protocol type, but a collector can only belong to one type, which determines its role in the acquisition and control platform and what tasks it can perform.

[0056] The development of the collector is not limited to the SpringCloud microservice approach of the cloud network architecture. The collector can be developed in any language. It can be easily accessed by implementing the corresponding interactive interface and logic when it is needed as an available resource on the collection and control platform.

[0057] The task source needs to be added to the task source management of the control center. After adding, the task information can be synchronized to the control center by manual triggering, and the control center will complete the purpose of sending it to the collector. Since the control center is a SpringCloud microservice, the task source can be configured by adding the service name to the url (http: / / instance-name / uri) or directly configured by http: / / ip:port / uri. The former will be called through feign, and the latter will be directly accessed through RestTemplate as a standard http.

[0058] Example 2

[0059] Reference Figure 1 As shown, a data acquisition and control engine system based on a cloud-edge-end architecture includes:

[0060] Task source module, control center module, collector module.

[0061] Among them, the task source module is used to publish the details of the tasks that need to be executed, including the objects to be collected, the information to be collected, the application identifier of the collection task, the result output configuration and other information.

[0062] The control center module is used to receive tasks, manage collector groups, maintain collector status, and automatically rebalance and distribute tasks to collectors.

[0063] The collector module is used to receive tasks issued by the control center module, start scheduled tasks or execute one-time tasks according to the specific content of the tasks, and regularly report the operating status to the control center, including startup duration, task execution status, tasks to be executed, and other information.

[0064] The system provided by the present invention can be connected to a variety of custom collectors to meet the needs of collecting various resources; it can expand the number of collectors to achieve dynamic expansion, and can manage collectors in groups, so that collectors in the same group share tasks, and the grouping conditions can be customized.

[0065] In the present invention, the business end can specify the corresponding collector to execute its own collection needs, and can perceive the state of the collector and dynamically allocate tasks according to the operation status. The system has a convenient business end task access method and can provide corresponding access standards.

[0066] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0067] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data acquisition and control engine system based on cloud-edge-end architecture, characterized in that: include: The business side is used to provide a task list; The controller is used to monitor the heartbeat changes of the control center and modify the task allocation plan according to the changes in task information and the heartbeat changes of the collector; Collector, used to collect data information and report the operation status to the control center; The control center is used to coordinate controllers and collectors, record the list of tasks that need to be distributed, select the applicable group according to the task identification, and generate task information; according to the operating status reported by the collector, readjust and distribute the tasks to it.

2. According to claim 1, the data acquisition and control engine system based on the cloud-edge-end architecture is characterized in that: It also includes a data consumer for receiving the results generated by the collector.

3. The data acquisition and control engine system based on cloud-edge-end architecture according to claim 1 is characterized in that: The service end can customize task grouping and perform group splitting according to different network segments in network scenarios when grouping.

4. The data acquisition and control engine system based on cloud-edge-end architecture according to claim 3 is characterized in that: The control center includes searching for matching groups within a preset value domain in the task information, and is applicable to network segment isolation in network scenarios and collector resource isolation in cloud resources.

5. The data acquisition and control engine system based on cloud-edge-end architecture according to claim 4 is characterized in that: The control center also includes: When distributing tasks or not distributing tasks, a minimum impact strategy is adopted; the minimum impact strategy includes not changing the result of the last task distribution, so that the change of the task received by the collector is minimal; Scenarios when tasks are not distributed include: Intra-group rebalancing: if the collector in a group changes, the tasks in the group will be adjusted; Global rebalancing: if the grouping conditions change or the task list changes, all grouped tasks will be regrouped.

6. The data acquisition and control engine system based on cloud-edge-end architecture according to claim 1 is characterized in that: When the collector reports the running status, if the status is unavailable, the control center will take back the assigned tasks and redistribute the tasks according to the currently available collector information.

7. The data acquisition and control engine system based on cloud-edge-end architecture according to claim 6 is characterized in that: Supports multiple types of collectors, and each collector is of only one type; the collector can use multiple development languages ​​to implement an interactive interface with the platform.

8. A data acquisition and control engine method based on a cloud-edge-device architecture, based on the data acquisition and control engine system based on a cloud-edge-device architecture according to any one of claims 1-7, characterized in that: include: Step S1: Recording a task list provided by the service end; the task list includes a task identifier; Step S2: The control center selects an applicable group according to the task identifier; Step S3: After the collector is connected to the platform, it is registered and regularly reports its health status to the control center; Step S4: Instruct the control center to readjust the distribution task list according to the grouping status and the operating status reported by the collector.

9. The data acquisition and control engine method based on cloud-edge-end architecture according to claim 8 is characterized in that: The control center adopts the minimum impact strategy when distributing tasks or not distributing tasks; the minimum impact strategy includes not changing the result of the last task distribution, so that the change of the task received by the collector is minimal; Scenarios when tasks are not distributed include: Intra-group rebalancing: if the collector in a group changes, the tasks in the group will be adjusted; Global rebalancing: if the grouping conditions change or the task list changes, all grouped tasks will be regrouped.

10. The data acquisition and control engine method based on cloud-edge-end architecture according to claim 8 is characterized in that: When the collector reports the running status, if the status is unavailable, the control center will take back the assigned tasks and redistribute the tasks according to the currently available collector information.

Images