System, method and device for collaborative optimization of data quality of regulation and control cloud platform and medium

Abstract

The invention belongs to the field of data processing, and discloses a collaborative optimization system, method, equipment and medium for regulating and controlling data quality of a cloud platform. The collaborative optimization system for regulating and controlling the data quality of the cloud platform realizes global management of a data quality task by setting a data quality management module; through setting a node resource management module, rapid positioning and on-demand sharing of node resources are realized, unified monitoring and integration of the node resources are completed, flexible utilization of the node resources is carried out, collaborative distribution of data quality tasks among collaborative nodes is realized, and the calculation efficiency of the data quality tasks is improved; a node network management module is arranged to set a unified interface and a communication protocol for a leading node and a cooperative node, so that network delay between the nodes is reduced, and energy consumption generated by data transmission is reduced; the confidentiality, reliability and stability of data and application services are realized by setting a global security management module, and collaborative optimization of the data quality of the full-regulation cloud platform is realized.

Description

Technical Field

[0001] The present invention belongs to the field of data processing and relates to a system, method, device and medium for collaboratively optimizing data quality of a cloud platform. Background Art

[0002] Unified, complete, timely and accurate data is an important foundation for improving the coordinated control capabilities of large power grids and the level of lean dispatch management. In recent years, dispatches at all levels have continued to carry out comprehensive rectification of control data quality. Based on a unified two-level architecture control cloud platform, a unified model for all business of power grid dispatch has been built, breaking through the data interaction, application collaboration, and safe, real-time, and efficient service management between the two-level cloud nodes, realizing the horizontal integration of data between various professions and the vertical sharing of dispatches at all levels, laying a solid foundation for standardized data management.

[0003] At present, the technical solutions for collaborative optimization of data quality in the control cloud platform generally collect data in a centralized control station or cloud center, and perform unified processing and data verification evaluation in the cloud center. When problems are found in the data verification, the source system processes the data and re-sends it. However, in the face of increasingly common data-driven integrated analysis and decision-making scenarios, data quality problems at a certain node under the two-level architecture of the control cloud platform may cause deviations in global analysis and decision-making, and the two-level architecture manages different operating data ranges, making it impossible to directly conduct data quality verification of the entire network and monitor the source data governance in real time in the cloud center. In addition, with the surge in data volume and the increasingly high requirements for response time and security in data quality work, the current centralized computing method of the cloud center faces huge computing pressure, storage pressure, and network pressure.

[0004] To address this problem, edge computing has been proposed as a new computing paradigm. The collaborative nodes in the cloud-edge collaborative computing model based on edge computing have the ability to calculate and analyze. By performing calculations at the edge of the network, while achieving overall computing capacity expansion, it can effectively reduce network bandwidth and cloud center computing and storage resource usage, and can be well applied to the data quality work of the cloud platform. However, the existing cloud platform data quality optimization methods based on cloud-edge collaborative technology mostly focus on the management of data quality computing tasks, rules, and data. How to coordinate and control the collaborative nodes of the cloud to maximize the utilization of each collaborative node and achieve more efficient data quality collaborative optimization has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a system, method, device and medium for collaborative optimization of cloud platform data quality.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] According to a first aspect of the present invention, a data quality collaborative optimization system for a control cloud platform is provided, wherein the control cloud platform includes a dominant node and a plurality of collaborative nodes; the data quality collaborative optimization system for the control cloud platform includes: a data quality management module for managing data quality tasks of the dominant node and the collaborative nodes, managing data collection scopes and data processing rules of the collaborative nodes, and managing data quality evaluation results of the collaborative nodes; a node resource management module for locating node resources of the collaborative nodes, monitoring node resources of the collaborative nodes, obtaining node load rates of the collaborative nodes and determining collaborative nodes that communicate with the dominant node according to the node load rates, and distributing data quality tasks to the collaborative nodes according to the node resources; a node network management module for setting a unified interface and communication protocol for the dominant node and the collaborative nodes; a global security management module for managing information transmission security between collaborative nodes and between collaborative nodes and the dominant node, monitoring and auditing collaborative nodes, managing collaborative nodes, and performing data encryption storage.

[0008] Optionally, the locating node resources of the collaborative node includes: obtaining the resource locator of each node resource of the collaborative node through a node-service collaborative coding method, and locating each node resource of the collaborative node according to the resource locator; wherein the node-service collaborative coding method includes: at the node level, encoding the area to which the node resource belongs, the node name, the node type, and the access / suspend status to obtain a first code; at the service level, encoding the service interface name, service IP address, service port number, service PID, and service type of the node resource to obtain a second code; combining the first code and the second code to obtain the resource locator of the node resource.

[0009] Optionally, the node resources of the monitoring collaborative node include: obtaining the resource load information of the operating system and each process of the collaborative node in real time by accessing the operating system kernel file of the collaborative node, and obtaining the real-time values ​​of monitoring indicators of the server and virtual machine of the collaborative node; wherein the monitoring indicators include one or more of the following: CPU utilization, memory utilization, disk utilization, network traffic, disk I / O read and write information, process memory utilization information, system log, number of database connections, distribution information and status of each data node, and port traffic of each network device.

[0010] Optionally, the method of obtaining the node load rate of the collaborative node and determining the collaborative node that communicates with the dominant node based on the node load rate includes: obtaining the node load rate of the collaborative node; wherein the node load rate is the CPU load rate or the memory load rate; based on the node load rate of the collaborative node, communicating between the dominant node and the collaborative node whose node load rate is lower than a preset node load rate threshold, or communicating between the dominant node and the collaborative node in order of the node load rates from low to high.

[0011] Optionally, distributing the data quality tasks to the collaborative nodes according to the node resources includes: based on the node resources, using gRPC-based wide area service sharing technology to disperse the data quality tasks to the collaborative nodes according to geographical locations and scheduling agencies.

[0012] Optionally, the management of information transmission security between collaborative nodes and between collaborative nodes and dominant nodes includes: managing the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes through network division, business and management separation, malicious code prevention and identity authentication; the monitoring and auditing of collaborative nodes includes: conducting account review of collaborative nodes, resource locking and unlocking, and log auditing.

[0013] Optionally, the management of collaborative nodes includes: mutual recognition of identities between the dominant node and the collaborative node, and registration of collaborative nodes, node information maintenance, whitelist management and IP configuration; wherein the node information includes node name, node description, node type, flag, message subject, operation time and owner; the data encryption storage includes: using CRC check to verify the data, and using RSA64-bit encryption algorithm for encrypted storage.

[0014] In a second aspect, the present invention provides a method for collaborative optimization of data quality of a control cloud platform based on the above-mentioned collaborative optimization system for data quality of a control cloud platform, comprising: managing data quality tasks of dominant nodes and collaborative nodes, managing data collection scope and data processing rules of collaborative nodes, and managing data quality evaluation results of collaborative nodes through a data quality management module; locating node resources of collaborative nodes, monitoring node resources of collaborative nodes, obtaining node load rates of collaborative nodes and determining collaborative nodes that communicate with dominant nodes according to the node load rates, and distributing data quality tasks to collaborative nodes according to the node resources; setting a unified interface and communication protocol for dominant nodes and collaborative nodes through a node network management module; managing information transmission security between collaborative nodes and between collaborative nodes and dominant nodes through a global security management module, monitoring and auditing collaborative nodes, managing collaborative nodes, and performing data encryption storage.

[0015] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for collaboratively optimizing the quality of cloud platform data when executing the computer program.

[0016] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for collaborative optimization of cloud platform data quality are implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The data quality collaborative optimization system of the control cloud platform of the present invention is aimed at the control cloud platform designed with a two-level architecture based on the dominant node and the collaborative node. The global management of data quality tasks is realized by setting a data quality management module, and the node resource management module is realized by setting a node resource management module to realize the rapid positioning and on-demand sharing of node resources, complete the unified monitoring and integration of node resources, and make flexible use of node resources, so as to realize the collaborative allocation of data quality tasks among various collaborative nodes, and improve the calculation efficiency of data quality tasks; by setting a node network management module to set a unified interface and communication protocol for the dominant node and the collaborative node, the network delay between nodes is reduced, thereby reducing the energy consumption generated by data transmission; by setting a global security management module, the confidentiality, reliability and stability of data and application services are realized. The collaborative optimization system comprehensively considers the influencing factors such as the node resources of the collaborative nodes and the network transmission between the nodes, and at the same time establishes a global security management mechanism under the two-level architecture, so as to realize the unified optimization and scheduling of the node resources of each collaborative node, alleviate the calculation pressure of the control cloud platform and the bandwidth pressure of the wide area data network, and realize the collaborative optimization of the data quality of the whole platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a structural block diagram of the collaborative optimization system for data quality of the control cloud platform according to an embodiment of the present invention.

[0020] Figure 2 This is a structural block diagram of a node resource management module according to an embodiment of the present invention.

[0021] Figure 3 This is a structural block diagram of the global security management module according to an embodiment of the present invention.

[0022] Figure 4 This is a flow chart of a method for collaborative optimization of data quality on a control cloud platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0026] See also Figure 1 In one embodiment of the present invention, a control cloud platform data quality collaborative optimization system is provided, which is applicable to a two-level architecture control cloud platform including a dominant node and a plurality of collaborative nodes, so as to improve the computing efficiency of the control cloud platform data quality tasks and reduce its computing cost.

[0027] Specifically, the cloud platform data quality collaborative optimization system of the present invention includes a data quality management module, a node resource management module, a node network management module and a global security management module.

[0028] Among them, the data quality management module is used to manage the data quality tasks of the dominant node and the collaborative node, manage the data collection scope and data processing rules of the collaborative node, and manage the data quality evaluation results of the collaborative node; the node resource management module is used to locate the node resources of the collaborative node, monitor the node resources of the collaborative node, obtain the node load rate of the collaborative node and determine the collaborative node that communicates with the dominant node according to the node load rate, and distribute the data quality tasks to the collaborative node according to the node resources; the node network management module is used to set a unified interface and communication protocol for the dominant node and the collaborative node; the global security management module is used to manage the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes, monitor and audit collaborative nodes, manage collaborative nodes, and perform data encryption storage.

[0029] The data quality collaborative optimization system of the control cloud platform of the present invention is aimed at the control cloud platform designed with a two-level architecture based on the dominant node and the collaborative node. The global management of data quality tasks is realized by setting a data quality management module, and the node resource management module is realized by setting a node resource management module to realize the rapid positioning and on-demand sharing of node resources, complete the unified monitoring and integration of node resources, and make flexible use of node resources, so as to realize the collaborative allocation of data quality tasks among various collaborative nodes, and improve the calculation efficiency of data quality tasks; by setting a node network management module to set a unified interface and communication protocol for the dominant node and the collaborative node, the network delay between nodes is reduced, thereby reducing the energy consumption generated by data transmission; by setting a global security management module, the confidentiality, reliability and stability of data and application services are realized. The collaborative optimization system comprehensively considers the influencing factors such as the node resources of the collaborative nodes and the network transmission between the nodes, and at the same time establishes a global security management mechanism under the two-level architecture, so as to realize the unified optimization and scheduling of the node resources of each collaborative node, alleviate the calculation pressure of the control cloud platform and the bandwidth pressure of the wide area data network, and realize the collaborative optimization of the data quality of the whole platform.

[0030] Explanatory, the control cloud platform is based on cloud computing technology concepts such as virtualization, distribution and service. It is a cloud service platform for power grid dispatching business. Its architecture design reflects the characteristics of hardware resource virtualization, data standardization and application service. It is an important technical means to support power grid operation and control management. Combined with the production organization model of control business, the control cloud platform follows the hierarchical deployment model that is compatible with the principle of "unified dispatching, hierarchical management", and builds a two-level deployment control cloud system of "1 dominant node + N collaborative nodes" across dispatching organizations. The dominant node is responsible for managing and controlling the operation of each collaborative node, and realizing interconnection and information exchange with each collaborative node. The collaborative node cooperates with the dominant node to collect and aggregate data. On the basis of a unified architecture, the dominant node and the collaborative node can operate independently, forming an organic and collaborative whole, realizing source-end maintenance, automatic integration, unified service and global sharing of various information resources.

[0031] The collaborative optimization of the data quality of the control cloud platform refers to the optimization of the data quality of the control cloud platform by collaboratively optimizing the processing process of the data quality tasks of the control cloud platform. Among them, data quality tasks generally include data cleaning, indicator calculation, and data quality assessment.

[0032] Exemplarily, the data quality management module has three submodules: a global task scheduling module, a global rule management module, and a global data quality assessment module. The global task scheduling module is used to manage the data quality tasks of the dominant node and the collaborative node, such as the start or end of the data quality tasks of the dominant node and the collaborative node; the global rule management module is used to manage the data collection scope and data processing rules of the collaborative node, such as the creation and modification of the data collection scope and data processing rules of the collaborative node; the global data quality assessment module is used to manage the data quality assessment results of the collaborative node, such as unified management and integration of the data quality assessment results sent by each collaborative node, and intelligent comprehensive evaluation of the integrated multi-source data quality assessment results. The three submodules work together to achieve integrated calculation of data quality tasks for the entire network.

[0033] See also Figure 2 Exemplarily, the node resource management module is provided with four sub-modules: a resource location service module, a resource monitoring service module, a load balancing service module and a distributed computing service module.

[0034] Among them, the resource positioning service module is used to locate the node resources of the collaborative node, the resource monitoring service module is used to monitor the node resources of the collaborative node, the load balancing service module is used to obtain the node load rate of the collaborative node and determine the collaborative node that communicates with the dominant node based on the node load rate, and the distributed computing service module is used to distribute data quality tasks to the collaborative nodes based on the node resources.

[0035] In a possible implementation manner, locating the node resources of the collaborative node includes: acquiring a resource locator of each node resource of the collaborative node through a node-service collaborative coding method, and locating each node resource of the collaborative node according to the resource locator.

[0036] Among them, the node-service collaborative encoding method includes: at the node level, encoding the area to which the node resource belongs, the node name, the node type and the access / suspend status to obtain a first code; at the service level, encoding the service interface name, service IP address, service port number, service PID and service type of the node resource to obtain a second code; combining the first code and the second code to obtain the resource locator of the node resource.

[0037] Specifically, by constructing a globally unique resource locator for the control cloud platform, based on the interactive protocol model agreed upon by the control cloud platform wide area network, the resource locator can be used to achieve rapid positioning and on-demand sharing of node resources, supporting computing resource allocation and data sharing in a distributed collaborative operation environment.

[0038] In a possible implementation, monitoring the node resources of the collaborative node includes: obtaining real-time information on the resource load occupied by the operating system and each process of the collaborative node by accessing the operating system kernel file of the collaborative node, and obtaining real-time values ​​of monitoring indicators of the server and virtual machine of the collaborative node.

[0039] Among them, the monitoring indicators include one or more of the following: CPU usage, memory occupancy, disk occupancy, network traffic, disk I / O read and write information, process memory usage information, system logs, number of database connections, distribution information and status of each data node, and port traffic of each network device.

[0040] Specifically, by monitoring the node resources of the collaborative nodes, a basis is provided for the subsequent collaborative allocation of computing tasks to idle collaborative nodes. In addition, the above-mentioned monitoring indicators can also be stored in a database at the same time, and the risks and alarms of each collaborative node can be intelligently predicted through data mining and analysis methods. For example, when physical resources and virtual resources fail, prompts are given to handle the failure in a timely manner.

[0041] In a possible implementation, the step of obtaining the node load rate of the collaborative node and determining the collaborative node that communicates with the dominant node based on the node load rate includes: obtaining the node load rate of the collaborative node; wherein the node load rate is a CPU load rate or a memory load rate; based on the node load rate of the collaborative node, communicating between the dominant node and the collaborative node whose node load rate is lower than a preset node load rate threshold, or communicating between the dominant node and the collaborative node in order of the node load rates from low to high.

[0042] Specifically, through technologies such as elastic horizontal expansion, the flexible allocation of computing and storage resources required for grid-integrated computing applications can be met, and resource integration and elastic utilization can be truly realized, providing sufficient storage and computing resources for the data quality computing tasks of the entire network. By collecting the execution status of processes and tasks, the CPU and memory resources of the collaborative nodes can be obtained, the CPU load rate or memory load rate can be calculated, and the collaborative nodes with lower CPU load rate or memory load rate can be communicated to achieve collaborative optimization of software and hardware resource utilization.

[0043] In a possible implementation, distributing the data quality tasks to the collaborative nodes according to the node resources includes: based on the node resources, using gRPC-based wide area service sharing technology to disperse the data quality tasks to the collaborative nodes according to geographical locations and scheduling organizations.

[0044] Specifically, multi-node parallel processing is used to improve the processing efficiency of millions of data and meet the real-time requirements of integrated analysis of power grid data quality. By decomposing data quality tasks, based on a 10G high-speed synchronous network, and using gRPC-based wide-area service sharing technology, on-demand calling of collaborative nodes and data sharing between collaborative nodes are realized. The leading node is used for unified scheduling, and the data quality that requires a large amount of computing and storage resources is divided according to geographical location and scheduling organization. At the same time, combined with the node resources of the collaborative nodes, it is dispersed to multiple collaborative nodes nearby for calculation, thereby improving data quality optimization performance and high availability.

[0045] Explanatory, the node network management module is used to set a unified interface and communication protocol for the dominant node and the collaborative node, solve the compatibility issues between different hardware and communication protocols of each collaborative node, reduce the transmission delay and energy consumption generated when the data quality calculation tasks are scheduled between different nodes, and ensure the data quality management function in the dominant node and each collaborative node and data communication, especially in poor network environments such as narrow communication bandwidth and unstable signals, to ensure the execution of data quality tasks.

[0046] See also Figure 3 , exemplarily, the global security management module has four sub-modules: information transmission security module, node monitoring and auditing module, node management module and data storage security module.

[0047] Among them, the information transmission security module is used to manage the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes, the node monitoring and auditing module is used to monitor and audit collaborative nodes, the node management module is used to manage collaborative nodes, and the data storage security module is used to perform data encryption storage.

[0048] In one possible implementation, the management of information transmission security between collaborative nodes and between collaborative nodes and dominant nodes includes: managing the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes through network division, business and management separation, malicious code prevention and identity authentication, so as to achieve the security of information transmission between the dominant node and collaborative nodes of the cloud platform and between each collaborative node.

[0049] In a possible implementation, the monitoring and auditing of the collaborative node includes: performing account review, resource locking and unlocking, and log auditing of the collaborative node.

[0050] Specifically, by conducting account review, resource locking and unlocking, and log auditing of collaborative nodes, potential security risks of collaborative nodes can be discovered and resolved in a timely manner.

[0051] In a possible implementation, the management of the collaborative node includes: mutual recognition of identities between the leading node and the collaborative node, registration of the collaborative node, node information maintenance, whitelist management, and IP configuration.

[0052] The node information includes node name, node description, node type, flag, message subject, operation time and owner.

[0053] In a possible implementation, the data encryption storage includes: verifying the data using CRC verification, and encrypting and storing the data using RSA 64-bit encryption algorithm.

[0054] Specifically, the system uses the RSA 64-bit encryption algorithm to encrypt and store data, and the system uses CRC verification to prevent data from being tampered with, thereby achieving secure storage of data.

[0055] The collaborative optimization system for data quality of a control cloud platform of the present invention, based on the characteristics of cloud computing and edge computing, proposes a collaborative optimization system for data quality of a control cloud platform under a two-level architecture of a control cloud that considers node resource management, node network management, and global security management. It comprehensively considers influencing factors such as computing and storage resources of collaborative nodes, network delay between nodes, and transmission energy consumption, and at the same time establishes a global security management mechanism under the two-level cloud architecture, uniformly optimizes and schedules the computing and storage resources of collaborative nodes of each unit, alleviates the computing pressure of the control cloud center and the bandwidth pressure of the wide area data network, and realizes the integrated calculation of data quality of the entire network.

[0056] See also Figure 4 In another embodiment of the present invention, a method for collaboratively optimizing data quality of a control cloud platform is provided, which can be implemented based on the above-mentioned collaborative optimization system for data quality of a control cloud platform. Specifically, the method for collaboratively optimizing data quality of a control cloud platform includes the following steps:

[0057] S1: Manage the data quality tasks of the leading node and the collaborative node through the data quality management module, manage the data collection scope and data processing rules of the collaborative node, and manage the data quality assessment results of the collaborative node.

[0058] S2: Locate the node resources of the collaborative node through the node resource management module, monitor the node resources of the collaborative node, obtain the node load rate of the collaborative node and determine the collaborative node that communicates with the dominant node according to the node load rate, and distribute the data quality tasks to the collaborative node according to the node resources.

[0059] S3: Set a unified interface and communication protocol for the leading node and the collaborative node through the node network management module.

[0060] S4: Manage the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes through the global security management module, monitor and audit collaborative nodes, manage collaborative nodes, and perform data encryption storage.

[0061] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0062] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to regulate the operation of the cloud platform data quality collaborative optimization method.

[0063] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for collaborative optimization of data quality of the cloud platform in the above embodiment.

[0064] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A collaborative optimization system for regulating cloud platform data quality, characterized in that: The control cloud platform includes a leading node and several collaborative nodes; the control cloud platform data quality collaborative optimization system includes: Data quality management module, used to manage the data quality tasks of the leading node and the collaborative node, manage the data collection scope and data processing rules of the collaborative node, and manage the data quality evaluation results of the collaborative node; A node resource management module, used to locate node resources of collaborative nodes, monitor node resources of collaborative nodes, obtain node load rates of collaborative nodes and determine collaborative nodes that communicate with the leading node according to the node load rates, and distribute data quality tasks to collaborative nodes according to the node resources; The node network management module is used to set a unified interface and communication protocol for the leading node and the collaborative node; The global security management module is used to manage the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes, monitor and audit collaborative nodes, manage collaborative nodes, and perform data encryption storage.

2. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The node resources of the positioning coordination node include: Acquire resource locators of each node resource of the collaborative node through a node-service collaborative coding method, and locate each node resource of the collaborative node according to the resource locators; Among them, the node-service collaborative encoding method includes: at the node level, encoding the area to which the node resource belongs, the node name, the node type and the access / suspend status to obtain a first code; at the service level, encoding the service interface name, service IP address, service port number, service PID and service type of the node resource to obtain a second code; combining the first code and the second code to obtain the resource locator of the node resource.

3. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The node resources of the monitoring coordination node include: By accessing the operating system kernel file of the collaborative node, the resource load information of the collaborative node's operating system and each process is obtained in real time, and the real-time values ​​of the monitoring indicators of the collaborative node's server and virtual machine are obtained; Among them, the monitoring indicators include one or more of the following: CPU usage, memory occupancy, disk occupancy, network traffic, disk I / O read and write information, process memory usage information, system logs, number of database connections, distribution information and status of each data node, and port traffic of each network device.

4. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The acquiring the node load rate of the collaborative node and determining the collaborative node communicating with the dominant node according to the node load rate comprises: Obtain the node load rate of the collaborative node; the node load rate is the CPU load rate or the memory load rate; According to the node load rates of the collaborative nodes, the dominant node communicates with the collaborative nodes whose node load rates are lower than a preset node load rate threshold, or the dominant node communicates with the collaborative nodes in order of the node load rates from low to high.

5. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The distributing the data quality tasks to the collaborative nodes according to the node resources comprises: According to the node resources, the gRPC-based wide-area service sharing technology is adopted to disperse the data quality tasks to the collaborative nodes according to the geographical location and scheduling organization.

6. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The management of information transmission security between collaborative nodes and between collaborative nodes and dominant nodes includes: Through network segmentation, business and management separation, malicious code prevention and identity authentication, manage the information transmission security between collaborative nodes and between collaborative nodes and dominant nodes; The monitoring and auditing coordination node includes: Conduct account review, resource locking and unlocking, and log auditing for collaborative nodes.

7. The control cloud platform data quality collaborative optimization system according to claim 1 is characterized in that: The management coordination node includes: Mutual recognition of identities between leading nodes and collaborative nodes, as well as registration of collaborative nodes, node information maintenance, whitelist management, and IP configuration; where node information includes node name, node description, node type, logo, message subject, operation time, and owner; The data encryption storage comprises: CRC check is used to verify the data, and RSA 64-bit encryption algorithm is used for encrypted storage.

8. A method for collaborative optimization of data quality of a control cloud platform based on the collaborative optimization system for data quality of a control cloud platform according to any one of claims 1 to 7, characterized in that: include: The data quality management module manages the data quality tasks of the leading node and the collaborative node, manages the data collection scope and data processing rules of the collaborative node, and manages the data quality assessment results of the collaborative node; Locating node resources of the collaborative nodes through the node resource management module, monitoring the node resources of the collaborative nodes, obtaining the node load rate of the collaborative nodes and determining the collaborative nodes that communicate with the leading node according to the node load rate, and distributing the data quality tasks to the collaborative nodes according to the node resources; Set up unified interfaces and communication protocols for leading nodes and collaborative nodes through the node network management module; The global security management module manages the security of information transmission between collaborative nodes and between collaborative nodes and dominant nodes, monitors and audits collaborative nodes, manages collaborative nodes, and performs data encryption storage.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for collaborative optimization of cloud platform data quality control as described in claim 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the method for collaborative optimization of cloud platform data quality control as claimed in claim 8 are implemented.

Images