Cloud side end online data system and verification method thereof

Abstract

The invention belongs to the technical field of data processing systems, and particularly relates to a cloud side online data system and a verification method thereof. Comprising a cloud platform service module, an edge service module and a terminal service module. The cloud platform service module is composed of server group equipment, data isolation equipment, security gateway equipment and data storage equipment; the system comprises a service collaboration component, a service management component and a system monitoring component. The edge service module is based on a power distribution network edge internet-of-things proxy structure and is composed of near-end edge data and network service equipment; comprising a service registration component, a service discovery component and a service updating component. The terminal service module is composed of data acquisition terminals of a power distribution network user side and a management and control side, and data transmission terminal operation data and environment data sensing ports or devices; comprising a terminal monitoring component and a user generation component. The invention provides a system scheme for improving the validity of data service under the situation of larger data concurrency.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing systems, and in particular, relates to a cloud-edge online data system and a verification method thereof. Background Art

[0002] With the advent of the big data era, while using data technology to enhance and optimize the business capabilities of various industries and increase the industry's market value, it also makes the majority of enterprises face an exponentially rapid increase in data processing needs. These data involve data generated by the company's own internal operation and management, as well as massive data related to the company in the network and industry. For large enterprises, as the company continues to develop, it is necessary to divide more levels of management levels in order to better disperse the pressure of business and data processing, and rely on this multi-level distributed management to establish data processing processes and systems. However, at present, most companies still lack effective means to establish efficient and accurate data processing systems based on the current structure. Summary of the invention

[0003] The purpose of the present invention is to provide a cloud-edge data processing solution for better optimizing data processing services, taking full account of actual needs and making full use of related technologies such as layered processing structure and cloud data services.

[0004] To achieve the above purpose, the present invention adopts the following technical solution.

[0005] A cloud-edge-end online data system, comprising a cloud platform service module, an edge service module, and a terminal service module;

[0006] The cloud platform service module consists of server group equipment, data isolation equipment, security gateway equipment, and data storage equipment; the cloud platform service module includes service coordination components, service management components, and system monitoring components;

[0007] The service coordination component establishes a gateway interface for service requests, service access, and service policy management; the service management component confirms service requests, allocates and adjusts service attribute information according to system instructions and / or preset specifications; the system monitoring component confirms the system operation status, collects system service availability, system resource usage, configures attributes and generates corresponding data interaction processes, tracks and confirms service allocation status and information, and generates corresponding service logs;

[0008] The edge service module is based on the edge IoT proxy structure of the distribution network and consists of proximal edge data and network service devices. The edge service module includes service registration components, service discovery components, and service update components.

[0009] The terminal service module is composed of data collection terminals, data transmission terminals and other operation data and environmental data perception ports or devices on the user side and the control side of the distribution network; it includes terminal monitoring components and user generation components.

[0010] For further improvement or preferred implementation of the aforementioned cloud-edge online data system, the service collaboration component is equipped with a request access gateway, a service aggregation gateway, a service intervention gateway, and a service policy gateway; the request access gateway completes the access service of each interface service request; the service aggregation gateway completes the joint service between one or more services; the service intervention gateway completes independent content services such as identity authentication and information identification for specific service content; the service policy gateway establishes a unified management policy service.

[0011] As a further improvement or preferred implementation plan of the aforementioned cloud-edge online data system, the service discovery component registers according to the IP address and port provided by the edge service module, monitors the corresponding IP address and port according to the database update component instructions, and feeds back the monitoring results.

[0012] As a further improvement or preferred implementation plan of the aforementioned cloud-edge online data system, the service update component generates, stores and updates the corresponding service IP database based on the service registration results of the service discovery component, and calls the service discovery component based on a preset period to monitor and update the service status in the service IP database.

[0013] Further improvement or preferred implementation plan of the aforementioned cloud-edge-end online data system, the service IP database includes at least a service ID data table and a service chain data table;

[0014] The service ID data table stores service registration information, service port, status update time, and tag information; the service chain data table stores the service chain data of standard jobs or tasks. The service chain data stores the IDs of a series of continuously executable services that complete standardized jobs and tasks; it also stores the generation information and update information of the corresponding service chain.

[0015] For further improvement or preferred implementation of the aforementioned cloud-edge-end online data system, the service registration component determines the corresponding edge computing service content based on its proximal edge data and the real-time status of the network service device, and uploads the corresponding IP address and device port information to the cloud platform service discovery component, and reports to the cloud platform service discovery component when the service content and status change;

[0016] The service discovery component updates and confirms the service availability status in real time, providing a basis for the operation of edge services; the service update component creates a corresponding service IP database for edge services and maintains the real-time nature of the corresponding database.

[0017] As a further improvement or preferred implementation plan of the aforementioned cloud-edge-end online data system, the edge service module is established based on the KubeEdge structure and consists of the edge cloud and the edge end;

[0018] The edge cloud includes edge controllers, device controllers, synchronization controllers, and cloud hubs; the edge includes: CloudCore components that detect edge status information and implement cloud-edge synchronization; EdgeCore components for edge container lifecycle management; and EventBus components that subscribe to and publish messages and pass data or events to corresponding subscribers in inter-thread communication scenarios.

[0019] For further improvement or preferred implementation of the aforementioned cloud-edge online data system, the cloud platform service module is established based on the Kubernetes structure and consists of multiple physical or virtual Kubernetes nodes running several Linux container sets; Kubernetes nodes are divided into control nodes and computing nodes, and the control nodes maintain the Linux container sets (including allocation and management of Linux container sets and computing resources required for applications or businesses); the computing node system actually runs, and the control nodes obtain instruction messages and forward them to the computing nodes.

[0020] Further improvement or preferred implementation plan of the aforementioned cloud-edge-end online data system, the terminal monitoring component monitors and analyzes the terminal implementation service status information, obtains the data information required to establish the service IP, and transmits it to the service discovery component;

[0021] The user generation component requests service ID data from the cloud platform service module as a user, and generates corresponding request instructions to obtain corresponding service content.

[0022] In order to confirm the validity of the system operation, the present invention also provides a verification method for verifying the cloud-edge online data system, the main steps of which include:

[0023] Stp1. Determine the test conditions

[0024] Determine the system service content and expected resource investment expectations based on the expected application scope; retrieve the existing service types and historical status information data of the distribution network to generate real-time service status control parameters;

[0025] Stp2. Determine the test content

[0026] According to the software and hardware environment configuration information of the current test scope, simulate the establishment of a cloud-edge online data system, obtain the establishment of a system test environment, and record the test environment configuration information;

[0027] Set cloud-edge service test projects of corresponding scale according to the characteristics of the distribution network cloud-edge business in the current application scope; record the test project information;

[0028] Stp3, feasibility verification

[0029] Conduct test analysis based on distribution network service types and historical status information data, and collect actual operation results of test items;

[0030] Stp4, run verification

[0031] Extract the test project operation results, obtain the actual execution success rate of each type of service and service chain, obtain the system operation load status data, and compare and determine whether the corresponding indicators meet the expected values;

[0032] If the expected expectations are met, further reduce resource input or further increase the difficulty of the test project content to further confirm the optimal resource input and the maximum load capacity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of the cloud-edge online data system;

[0034] Figure 2 It is a flowchart of the verification method of the cloud-edge online data system. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments.

[0036] The present invention relates to a cloud-edge-end online data system, which mainly includes a cloud platform service module, an edge service module and a terminal service module; the present invention provides a system solution for improving the effectiveness of data services in a scenario with a large amount of concurrent data.

[0037] Its composition structure is as follows Figure 1 As shown, it includes establishing a data control center based on cloud services, integrating, monitoring, and managing data service content, and assigning services according to data processing tasks. As the cloud platform service module of the control center of the entire system, the cloud platform service module is mainly composed of necessary hardware such as server group equipment, data isolation equipment, security gateway equipment, data storage equipment, and the data network between the hardware.

[0038] The cloud platform service module includes at least a service collaboration component, a service management component and a system monitoring component. In this embodiment, the cloud platform service module is established based on the Kubernetes structure and is composed of multiple physical or virtual Kubernetes nodes running several Linux container sets. Kubernetes nodes are divided into control nodes and computing nodes. The control node maintains the Linux container set (including the allocation and management of the Linux container set and computing resources required for the application or business). The computing node system actually runs, and at the same time, the control node obtains the instruction message and forwards it to the computing node.

[0039] The service collaboration component establishes a gateway interface for service request, service access, and service policy management to achieve collaborative deployment management of different edge services; the service collaboration component has built-in request access gateway, service aggregation gateway, service intervention gateway, and service policy gateway;

[0040] Request the access gateway to complete the access service of each interface service request; the request access gateway ensures the smooth access process of a large number of user data service requests, ensures the accuracy of data service requests, and can also perform user verification and confirmation when necessary to obtain resistance;

[0041] The service aggregation gateway completes the joint service between one or more services; the service aggregation network management establishes the interface of each service channel so as to realize the calling and allocation of multiple data services according to the spirit of service, and improve the execution efficiency of complex data services;

[0042] The service intervention gateway completes independent content services such as identity authentication and information identification for specific service content; the service intervention network management responds to emergencies that may be faced at different stages such as data service generation, execution and acceptance, and takes necessary service management measures in different situations to reduce the impact of emergencies, or verifies the data service content of data service objects without affecting the service itself;

[0043] The service policy gateway establishes a unified management policy service; it is used to establish a programmed processing strategy or specification for standardized data services or data service chains to improve the processing efficiency of a large number of concurrent or recurring specific data services;

[0044] The service management component confirms the service request, allocates and adjusts the service attribute information according to system instructions and or preset specifications;

[0045] The system monitoring component confirms the system operation status, collects system service availability, system resource usage, configuration properties and generates corresponding data interaction processes; tracks and confirms service allocation status and information, and generates corresponding service logs;

[0046] The system monitoring component is used to enable managers to collect information such as the operating status of the entire system and the occupancy of service resources, and provide it to managers in the form of various visual charts or data, so as to evaluate and analyze the system status at a glance, accurately control and manage system operation, timely discover and anticipate possible system conditions, and ensure system stability.

[0047] The edge service module is an intermediate service layer composed of proximal edge data and network service devices based on the edge IoT proxy structure of the distribution network.

[0048] As a preferred solution, in this embodiment, the edge service module is established based on the KubeEdge structure and consists of an edge cloud and an edge end; the edge cloud includes an edge controller, a device controller, a synchronization controller, and a cloud hub; the edge end includes: a CloudCore component for detecting edge status information and realizing cloud-edge synchronization; an EdgeCore component for edge container lifecycle management; an EventBus component for subscribing to and publishing messages, and transmitting data or events to corresponding subscribers in the scenario of inter-thread communication;

[0049] The edge service module includes service registration component, service discovery component, service update component,

[0050] The service registration component determines the corresponding edge computing service content based on its proximal edge data and the real-time status of the network service device, and uploads the corresponding IP address and device port information to the cloud platform service discovery component, and reports to the cloud platform service discovery component when the service content and status change;

[0051] The service discovery component updates and confirms the service availability status in real time, providing a basis for the operation of edge services, including: registering according to the IP address and port provided by the edge service module, monitoring the corresponding IP address and port according to the database update component instructions, and feeding back the monitoring results;

[0052] The service update component creates a corresponding service IP database for edge services and maintains the real-time performance of the corresponding database; including: generating, storing and updating the corresponding service IP database based on the service registration results of the service discovery component, and calling the service discovery component based on a preset period to monitor and update the service status in the service IP database;

[0053] The service IP database includes at least a class data table

[0054] Service ID data table, storing service registration information, service port, status update time, and tag information;

[0055] The service chain data table stores the service chain data of standard operations or tasks. The service chain data stores the IDs of a series of continuously executable services that complete the standardized operations and tasks; it also stores the generation information and update information of the corresponding service chain;

[0056] The terminal service module is composed of data acquisition terminals, data transmission terminals and other operation data and environmental data perception ports or devices on the user side and the control side of the distribution network; it is equipped with terminal monitoring components and user generation components.

[0057] Terminal monitoring monitors and analyzes the service status information of the terminal, obtains the data information required to establish the service IP, and transmits it to the service discovery component; the user generation component requests the service ID data from the cloud platform service module as a user, and generates corresponding request instructions to obtain the corresponding service content;

[0058] In order to confirm the validity of the system operation, the present invention also provides a verification method for verifying the cloud-edge online data system, such as Figure 2 As shown, the main steps include:

[0059] Stp1. Determine the test conditions

[0060] Determine the system service content and expected resource investment expectations based on the expected application scope; retrieve the existing service types and historical status information data of the distribution network to generate real-time service status control parameters;

[0061] Stp2. Determine the test content

[0062] According to the software and hardware environment configuration information of the current test scope, simulate the establishment of a cloud-edge online data system, obtain the establishment of a system test environment, and record the test environment configuration information;

[0063] Set cloud-edge service test projects of corresponding scale according to the characteristics of the distribution network cloud-edge business in the current application scope; record the test project information;

[0064] Stp3, feasibility verification

[0065] Conduct test analysis based on distribution network service types and historical status information data, and collect actual operation results of test items;

[0066] Stp4, run verification

[0067] Extract the test project operation results, obtain the actual execution success rate of each type of service and service chain, obtain the system operation load status data, and compare and determine whether the corresponding indicators meet the expected values;

[0068] If the expected expectations are met, further reduce resource input or further increase the difficulty of the test project content to further confirm the optimal resource input and the maximum load capacity of the system.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A cloud-edge-end online data system, characterized in that: Including cloud platform service module, edge service module, and terminal service module; The cloud platform service module consists of server group equipment, data isolation equipment, security gateway equipment, and data storage equipment; the cloud platform service module includes service coordination components, service management components, and system monitoring components; The service coordination component establishes a gateway interface for service requests, service access, and service policy management; the service management component confirms service requests, allocates and adjusts service attribute information according to system instructions and / or preset specifications; the system monitoring component confirms the system operation status, collects system service availability, system resource usage, configures attributes and generates corresponding data interaction processes, tracks and confirms service allocation status and information, and generates corresponding service logs; The edge service module is based on the edge IoT proxy structure of the distribution network and consists of proximal edge data and network service devices. The edge service module includes service registration components, service discovery components, and service update components. The terminal service module is composed of data collection terminals, data transmission terminals and other operation data and environmental data perception ports or devices on the user side and the control side of the distribution network; it includes terminal monitoring components and user generation components.

2. The cloud-edge-end online data system according to claim 1, characterized in that: The service coordination component has built-in request access gateway, service aggregation gateway, service intervention gateway, and service policy gateway; The request access gateway completes the access service of each interface service request; the service aggregation gateway completes the joint service between one or more services; the service intervention gateway completes independent content services such as identity authentication and information identification for specific service content; the service policy gateway establishes a unified management policy service.

3. The cloud-edge-end online data system according to claim 1, characterized in that: The service discovery component registers according to the IP address and port provided by the edge service module, monitors the corresponding IP address and port according to the database update component instructions, and feeds back the monitoring results.

4. The cloud-edge-end online data system according to claim 1, characterized in that: The service update component generates, stores and updates the corresponding service IP database based on the service registration results of the service discovery component, and calls the service discovery component based on a preset period to monitor and update the service status in the service IP database.

5. The cloud-edge-end online data system according to claim 1, characterized in that: The service IP database includes at least a service ID data table and a service chain data table; The service ID data table stores service registration information, service port, status update time, and tag information; the service chain data table stores service chain data for standard jobs or tasks, and the service chain data stores the IDs of a series of continuously executable services used to complete standardized jobs and tasks; it also stores the generation information and update information of the corresponding service chain.

6. The cloud-edge-end online data system according to claim 1, characterized in that: The service registration component determines the corresponding edge computing service content based on its proximal edge data and the real-time status of the network service device, and uploads the corresponding IP address and device port information to the cloud platform service discovery component, and reports to the cloud platform service discovery component when the service content and status change; The service discovery component updates and confirms the service availability status in real time, providing a basis for the operation of edge services; the service update component creates a corresponding service IP database for edge services and maintains the real-time nature of the corresponding database.

7. The cloud-edge-end online data system according to claim 1, characterized in that: The edge service module is built on the KubeEdge structure and consists of the edge cloud and edge end; The edge cloud includes edge controllers, device controllers, synchronization controllers, and cloud hubs; The edge side includes: CloudCore component for detecting edge status information and realizing cloud-edge synchronization; EdgeCore component for edge container lifecycle management; EventBus component for subscribing to and publishing messages and delivering data or events to corresponding subscribers in the scenario of inter-thread communication.

8. The cloud-edge-end online data system according to claim 1, characterized in that: The cloud platform service module is established based on the Kubernetes structure and consists of multiple physical or virtual Kubernetes nodes running several Linux container sets; Kubernetes nodes are divided into control nodes and computing nodes, and the control nodes are used to maintain the Linux container sets (including allocation and management of Linux container sets and computing resources required for applications or businesses); The computing nodes are used for the actual operation of the system, while the control nodes are used to obtain instruction messages and forward them to the computing nodes.

9. The cloud-edge-end online data system according to claim 1, characterized in that: The terminal monitoring component monitors and analyzes the terminal implementation service status information, obtains the data information required to establish the service IP, and transmits it to the service discovery component; The user generation component requests service ID data from the cloud platform service module as a user, and generates corresponding request instructions to obtain corresponding service content.

10. A verification method for verifying a cloud-edge online data system, characterized in that: The steps include: Stp1. Determine the test conditions Determine the system service content and expected resource investment expectations based on the expected application scope; retrieve the existing service types and historical status information data of the distribution network to generate real-time service status control parameters; Stp2. Determine the test content According to the software and hardware environment configuration information of the current test scope, simulate the establishment of a cloud-edge online data system, obtain the establishment of a system test environment, and record the test environment configuration information; Set cloud-edge-end service test projects of corresponding scale according to the characteristics of the distribution network cloud-edge-end business in the current application scope; Record test item information; Stp3, feasibility verification Conduct test analysis based on distribution network service types and historical status information data, and collect actual operation results of test items; Stp4, run verification Extract the test project operation results, obtain the actual execution success rate of each type of service and service chain, obtain the system operation load status data, and compare and determine whether the corresponding indicators meet the expected values; If the expected expectations are met, further reduce resource input or further increase the difficulty of the test project content to further confirm the optimal resource input and the maximum load capacity of the system.

Images