Regulation cloud platform data quality collaborative optimization system, method, device and medium

Abstract

The application belongs to the field of data processing, and discloses a regulation and control cloud platform data quality collaborative optimization system, method, device and medium. The regulation and control cloud platform data quality collaborative optimization system realizes global management of data quality tasks by setting a data quality management module, realizes quick positioning and on-demand sharing of node resources by setting a node resource management module, completes unified monitoring and integration of node resources, realizes flexible use of node resources, realizes collaborative distribution of data quality tasks among collaborative nodes, and improves the calculation efficiency of data quality tasks. The regulation and control cloud platform data quality collaborative optimization system sets a node network management module to set a unified interface and communication protocol for the leading node and the collaborative node, reduces the network delay between nodes, thereby reducing the energy consumption generated by data transmission. The regulation and control cloud platform data quality collaborative optimization system sets a global security management module to realize the confidentiality, reliability and stability of data and application services, and realizes the collaborative optimization of the whole regulation and control cloud platform data quality.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data processing, and relates to a regulation and control cloud platform data quality collaborative optimization system, method, device and medium. BACKGROUND

[0002] Unified, complete, timely and accurate data is an important basis for improving the collaborative control capability of large power grids and the lean management level of dispatching. In recent years, all levels of dispatching have been continuously carrying out comprehensive rectification of regulation and control data quality. Based on the two-level architecture of the regulation and control cloud platform, a unified model of power grid dispatching is constructed, the data interaction, application collaboration, and safe, real-time and efficient service management between the two-level cloud nodes are broken through, the horizontal integration of data among various professions and the vertical sharing of all levels of dispatching are realized, and a solid foundation is laid for data standardized management.

[0003] At present, the technical scheme for collaborative optimization of data quality of the regulation and control cloud platform is generally to centrally collect data to the regulation and control master station or the cloud center, and to perform unified processing and data checking and evaluation at the cloud center. When problems are found in data checking, the source system processes the data again and re-uploads it. However, in the face of more and more common data-driven integrated analysis and decision-making scenarios, data quality problems of a node under the two-level architecture of the regulation and control cloud platform may cause deviation of global analysis and decision-making, and the range of operation data managed by the two-level architecture is different, so it is impossible to directly carry out data quality checking of the whole network and real-time monitoring of source data governance at the cloud center. Moreover, with the explosive growth of data and the increasing requirements of data quality work on response time and security, the current cloud center centralized computing mode faces huge computing, storage and network pressures.

[0004] To solve the problem, edge computing as a new computing paradigm is proposed. The collaborative nodes in the cloud-edge collaborative computing model based on edge computing have the ability of computing and analysis, and by performing computing at the edge of the network, the overall computing capacity is expanded while the network bandwidth and the occupation of cloud center computing and storage resources are effectively reduced, which can be well applied to the data quality work of the regulation and control cloud platform. However, the existing regulation and control cloud platform data quality optimization method based on cloud-edge collaborative technology mostly focuses on the management of data quality computing tasks, rules and data, and how to coordinate and control the collaborative nodes of the regulation and control cloud to maximize the use of each collaborative node and further realize more efficient data quality collaborative optimization becomes a problem to be solved urgently. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a regulation and control cloud platform data quality collaborative optimization system, method, device and medium.

[0006] To achieve the above purpose, the following technical scheme is adopted:

[0007] In a first aspect, the application provides a cloud platform data quality collaborative optimization system, the cloud platform comprising a leading node and a plurality of collaborative nodes; the cloud platform data quality collaborative optimization system comprising: a data quality management module for managing data quality tasks of the leading node and the collaborative nodes, managing data collection ranges and data processing rules of the collaborative nodes, and managing data quality evaluation results of the collaborative nodes; a node resource management module for locating node resources of the collaborative nodes, monitoring the node resources of the collaborative nodes, obtaining node load rates of the collaborative nodes and determining the collaborative nodes in communication with the leading node according to the node load rates, and distributing data quality tasks to the collaborative nodes according to the node resources; a node network management module for setting a unified interface and communication protocol for the leading node and the collaborative nodes; a global security management module for managing information transmission security between the collaborative nodes and between the collaborative nodes and the leading node, monitoring and auditing the collaborative nodes, managing the collaborative nodes, and performing data encryption storage.

[0008] Optionally, the locating of the node resources of the collaborative nodes comprises: obtaining resource locators of the node resources of the collaborative nodes through a node-service collaborative coding method, and locating the node resources of the collaborative nodes according to the resource locators; wherein the node-service collaborative coding method comprises: encoding a region to which a node resource belongs, a node name, a node type, and an access / suspension state at a node level to obtain a first code; encoding a service interface name, a service IP address, a service port number, a service PID, and a service type at a service level to obtain a second code; and combining the first code and the second code to obtain a resource locator of the node resource.

[0009] Optionally, the monitoring of the node resources of the collaborative nodes comprises: obtaining resource load information of operating systems and processes of the collaborative nodes in real time by accessing operating system kernel files of the collaborative nodes, to obtain real-time values of monitoring indexes of servers and virtual machines of the collaborative nodes; wherein the monitoring indexes comprise one or more of the following: CPU usage, memory occupancy, disk occupancy, network traffic, disk I / O read and write information, process memory occupancy information, system logs, database connection numbers, data node distribution information and states, and port traffic of network devices.

[0010] Optionally, the obtaining of the node load rates of the collaborative nodes and the determination of the collaborative nodes in communication with the leading node according to the node load rates comprise: obtaining the node load rates of the collaborative nodes; wherein the node load rates are CPU load rates or memory load rates; and according to the node load rates of the collaborative nodes, performing communication between the leading node and the collaborative nodes whose node load rates are lower than a preset node load rate threshold, or performing communication between the leading node and the collaborative nodes in order from low to high node load rates.

[0011] Optionally, the distributing the data quality task to the collaborative node according to the node resource comprises: according to the node resource, using a gRPC-based wide-area service sharing technology, dispersing the data quality task to the collaborative node according to geographical position and a scheduling mechanism.

[0012] Optionally, the managing the information transmission security between the collaborative nodes and between the collaborative node and the leading node comprises: managing the information transmission security between the collaborative nodes and between the collaborative node and the leading node through network division, industry management separation, malicious code prevention and identity authentication; and the monitoring and auditing the collaborative node comprises: performing account auditing, resource locking and unlocking and log auditing of the collaborative node.

[0013] Optionally, the managing the collaborative node comprises: identity mutual authentication between the leading node and the collaborative node, and registration, node information maintenance, white list management and IP configuration of the collaborative node; wherein the node information comprises node name, node description, node type, flag, message topic, operation time and owner; and the performing data encryption storage comprises: using CRC check to check the data, and using RSA 64-bit encryption algorithm to perform encryption storage.

[0014] In the second aspect, the application provides a regulation and control cloud platform data quality collaborative optimization method based on the regulation and control cloud platform data quality collaborative optimization system, comprising: managing data quality tasks of the leading node and the collaborative node by the data quality management module, managing data collection range and data processing rules of the collaborative node, and managing data quality evaluation results of the collaborative node; positioning node resources of the collaborative node by the node resource management module, monitoring the node resources of the collaborative node, obtaining node load rates of the collaborative node and determining the collaborative nodes in communication with the leading node according to the node load rates, and distributing data quality tasks to the collaborative nodes according to the node resources; setting a unified interface and communication protocol for the leading node and the collaborative node by the node network management module; managing information transmission security between the collaborative nodes and between the collaborative node and the leading node by the global security management module, monitoring and auditing the collaborative node, managing the collaborative node, and performing data encryption storage.

[0015] In the third aspect, the application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the regulation and control cloud platform data quality collaborative optimization method when executing the computer program.

[0016] In the fourth aspect, the application provides a computer readable storage medium, which stores a computer program, wherein the computer program implements the steps of the regulation and control cloud platform data quality collaborative optimization method when executed by a processor.

[0017] Compared with the prior art, the present application has the following beneficial effects:

[0018] The regulation and control cloud platform data quality collaborative optimization system of the present application is for a regulation and control cloud platform designed based on a two-level architecture of a leading node and a collaborative node, realizes global management of data quality tasks by setting a data quality management module, realizes rapid positioning and on-demand sharing of node resources by setting a node resource management module, completes unified monitoring and integration of node resources, realizes flexible utilization of node resources, realizes collaborative distribution of data quality tasks among collaborative nodes, and improves the calculation efficiency of data quality tasks; a node network management module is set to set a unified interface and communication protocol for the leading node and the collaborative node, reduces the network delay between nodes, thereby reducing the energy consumption generated by data transmission; a global security management module is set to realize the confidentiality, reliability and stability of data and application services. The collaborative optimization system considers the node resources of the collaborative nodes and the network transmission between nodes, establishes a global security management mechanism under the two-level architecture, can realize unified optimization and scheduling of the node resources of the collaborative nodes, relieves the calculation pressure of the regulation and control cloud platform and the bandwidth pressure of the wide area data network, and realizes collaborative optimization of the data quality of the whole platform. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The regulation and control cloud platform data quality collaborative optimization system structure block diagram of the embodiment of the present application.

[0020] Figure 2 The node resource management module structure block diagram of the embodiment of the present application.

[0021] Figure 3 The global security management module structure block diagram of the embodiment of the present application.

[0022] Figure 4 The regulation and control cloud platform data quality collaborative optimization method flow chart of the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable personnel in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings:

[0026] See Figure 1 In one embodiment of the present invention, a collaborative optimization system for regulating cloud platform data quality is provided, which is applicable to a two-level architecture regulating cloud platform including a master node and several collaborative nodes, so as to improve the computational efficiency of data quality tasks of regulating cloud platform and reduce its computational cost.

[0027] Specifically, the cloud platform data quality collaborative optimization system of the present invention includes a data quality management module, a node resource management module, a node network management module, and a global security management module.

[0028] The system includes the following modules: a data quality management module for managing data quality tasks of the leading node and collaborating nodes, managing the data collection scope and data processing rules of collaborating nodes, and managing the data quality assessment results of collaborating nodes; a node resource management module for locating and monitoring the node resources of collaborating nodes, obtaining the node load rate of collaborating nodes and determining the collaborating nodes that communicate with the leading node based on the node load rate, and distributing data quality tasks to collaborating nodes based on the node resources; a node network management module for setting up unified interfaces and communication protocols for the leading node and collaborating nodes; and a global security management module for managing the security of information transmission between collaborating nodes and between collaborating nodes and the leading node, monitoring and auditing collaborating nodes, managing collaborating nodes, and performing encrypted data storage.

[0029] The application discloses a cloud platform data quality collaborative optimization system, which is designed for a two-level architecture of a cloud platform based on a leading node and collaborative nodes, realizes global management of data quality tasks by setting a data quality management module, realizes rapid positioning and on-demand sharing of node resources by setting a node resource management module, completes unified monitoring and integration of node resources, realizes flexible utilization of node resources, realizes collaborative distribution of data quality tasks among collaborative nodes, and improves the calculation efficiency of data quality tasks; a node network management module is set to set a unified interface and communication protocol for the leading node and the collaborative nodes, reduces the network delay between nodes, thereby reducing the energy consumption generated by data transmission; a global security management module is set to realize the confidentiality, reliability and stability of data and application services. The collaborative optimization system considers the node resources of the collaborative nodes and the network transmission between nodes, establishes a global security management mechanism under the two-level architecture, realizes unified optimization and scheduling of the node resources of the collaborative nodes, relieves the calculation pressure of the cloud platform and the bandwidth pressure of the wide area data network, and realizes collaborative optimization of the data quality of the whole platform.

[0030] Explanatorily, the cloud platform is based on virtualization, distribution and service, and is a cloud service platform for power grid dispatching business, and the architecture design embodies the characteristics of hardware resource virtualization, data standardization and application service, and is an important technical means for supporting power grid operation and dispatching management. In combination with the dispatching business production organization mode, the cloud platform follows a hierarchical deployment mode suitable for the principle of unified dispatching and hierarchical management, and builds a two-level deployment cloud system of "one leading node + N collaborative nodes" across dispatching institutions. The leading node is responsible for managing and controlling the operation of the collaborative nodes, and realizes interconnection and information exchange with the collaborative nodes. The collaborative nodes cooperate with the leading node to collect and converge data. The leading node and the collaborative nodes can independently operate on the basis of the unified architecture, form an organic whole, realize source-end maintenance, automatic integration, unified service and global sharing of various information resources.

[0031] The collaborative optimization of the data quality of the cloud platform refers to optimizing the data quality of the cloud platform by optimizing the processing process of the data quality tasks of the cloud platform. The data quality tasks generally include data cleaning, index calculation and data quality evaluation.

[0032] Exemplarily, the data quality management module sets three sub-modules of a global task scheduling module, a global rule management module and a global data quality evaluation module. The global task scheduling module is used for managing data quality tasks of the leading node and the collaborative node, such as the start or end of the data quality tasks of the leading node and the collaborative node; the global rule management module is used for managing the data collection range and the data processing rule of the collaborative node, such as the new establishment and modification of the data collection range and the data processing rule of the collaborative node; and the global data quality evaluation module is used for managing the data quality evaluation result of the collaborative node, such as uniformly managing and integrating the data quality evaluation result uploaded by each collaborative node, and intelligently comprehensively evaluating the integrated multi-source data quality evaluation result. The three sub-modules cooperatively operate to realize integrated calculation of the data quality tasks of the whole network.

[0033] Referring to Figure 2 Exemplarily, the node resource management module sets four sub-modules of a resource positioning service module, a resource monitoring service module, a load balancing service module and a distributed computing service module.

[0034] The resource positioning service module is used for positioning the node resources of the collaborative node, the resource monitoring service module is used for monitoring the node resources of the collaborative node, the load balancing service module is used for acquiring the node load rate of the collaborative node and determining the collaborative node in communication with the leading node according to the node load rate, and the distributed computing service module is used for distributing the data quality tasks to the collaborative node according to the node resources.

[0035] In a possible implementation, the positioning of the node resources of the collaborative node includes: acquiring resource locators of each node resource of the collaborative node through a node-service collaborative coding method, and positioning each node resource of the collaborative node according to the resource locators.

[0036] The node-service collaborative coding method includes: at the node level, coding the region to which the node resource belongs, the node name, the node type and the access / suspension state to obtain a first code; at the service level, coding the service interface name, the service IP address, the service port number, the service PID and the service type of the node resource to obtain a second code; and combining the first code and the second code to obtain the resource locator of the node resource.

[0037] Specifically, by constructing the globally unique resource locator of the regulation and control cloud platform, based on the interactive protocol mode agreed by the wide area network of the regulation and control cloud platform, the resource locator can realize quick positioning and on-demand sharing of the node resources, and support the allocation of computing resources and data sharing in the distributed collaborative operation environment.

[0038] In a possible implementation, the monitoring the node resources of the coordination node comprises: obtaining resource load information of an operating system and each process of the coordination node in real time by accessing an operating system kernel file of the coordination node, and obtaining real-time values of monitoring indexes of servers and virtual machines of the coordination node.

[0039] The monitoring indexes comprise one or more of the following: CPU usage, memory occupancy, disk occupancy, network traffic, disk I / O read and write information, process memory occupancy information, system logs, database connection numbers, data node distribution information and states, and network device port traffic.

[0040] Specifically, the node resources of the coordination node are monitored to provide a basis for subsequent cooperative allocation of computing tasks to idle coordination nodes. In addition, the monitoring indexes can also be stored in a database, and the risks and alarms of the coordination nodes can be intelligently predicted by data mining analysis methods, for example, when physical resources and virtual resources fail, the failure can be timely reminded.

[0041] In a possible implementation, the obtaining the node load rate of the coordination node and determining the coordination node that communicates with the leading node according to the node load rate comprises: obtaining the node load rate of the coordination node; wherein the node load rate is a CPU load rate or a memory load rate; and according to the node load rate of the coordination node, performing communication between the leading node and the coordination node whose node load rate is lower than a preset node load rate threshold, or performing communication between the leading node and the coordination node in order from low to high according to the node load rate.

[0042] Specifically, the flexible allocation of computing and storage resources required by power grid integrated computing applications is met by technologies such as elastic horizontal expansion, and the integration and elastic use of resources are truly realized, and sufficient storage and computing resources are provided for whole-network data quality computing tasks. By collecting the execution of processes and tasks, the conditions of resources such as CPU and memory of the coordination nodes are obtained, the CPU load rate or the memory load rate is calculated, and communication with the coordination node with a lower CPU load rate or memory load rate is performed, to realize cooperative optimization of software and hardware resource utilization.

[0043] In a possible implementation, the distributing the data quality task to the coordination node according to the node resources comprises: according to the node resources, using a wide-area service sharing technology based on gRPC, distributing the data quality task to the coordination node according to geographical positions and scheduling mechanisms.

[0044] Specifically, a multi-node parallel processing mode is adopted to improve the processing efficiency of data in the order of millions and meet the real-time requirements of power grid data quality integrated analysis. By decomposing data quality tasks, based on a gigabit resource high-speed synchronous network, using a wide-area service sharing technology based on gRPC, the on-demand calling of collaborative nodes and data sharing between collaborative nodes are realized, and the data quality requiring a large amount of computing and storage resources is divided according to geographical location and scheduling agencies. At the same time, combined with the node resources of the collaborative nodes, the data is dispersed to multiple collaborative nodes for calculation in proximity, thereby improving the data quality optimization performance and high availability.

[0045] Explanatorily, the node network management module is used to set a unified interface and communication protocol for the leading node and the collaborative nodes, solve the compatibility problem between different hardware and communication protocols of each collaborative node, reduce the transmission delay and energy consumption generated when the data quality calculation task is scheduled between different nodes, and ensure the data communication of the data quality management function between the leading node and each collaborative node. Especially in poor network environments with narrow communication bandwidth and unstable signals, the execution of the data quality task is ensured.

[0046] Referring to Figure 3 Exemplarily, the global security management module sets four sub-modules, namely, an information transmission security module, a node monitoring and auditing module, a node management module, and a data storage security module.

[0047] Among them, the information transmission security module is used to manage the information transmission security between the collaborative nodes and between the collaborative nodes and the leading node, the node monitoring and auditing module is used to monitor and audit the collaborative nodes, the node management module is used to manage the collaborative nodes, and the data storage security module is used to perform data encryption storage.

[0048] In a possible implementation, the management of the information transmission security between the collaborative nodes and between the collaborative nodes and the leading node includes: managing the information transmission security between the collaborative nodes and between the collaborative nodes and the leading node through network division, industry management separation, malicious code prevention, and identity authentication, to realize the security of information transmission between the leading node and the collaborative nodes of the regulation and control cloud platform and between the collaborative nodes.

[0049] In a possible implementation, the monitoring and auditing of the collaborative nodes includes: performing account auditing, resource locking and unlocking, and log auditing of the collaborative nodes.

[0050] Specifically, by performing account auditing, resource locking and unlocking, and log auditing of the collaborative nodes, potential security risks of the collaborative nodes are discovered in time and solved in time.

[0051] In a possible implementation, the management of the coordination node comprises identity mutual recognition between the leading node and the coordination node, registration of the coordination node, node information maintenance, white list management and IP configuration.

[0052] The node information comprises node name, node description, node type, flag, message topic, operation time and owner.

[0053] In a possible implementation, the data encryption storage comprises CRC check of the data and RSA 64-bit encryption algorithm for encryption storage.

[0054] Specifically, the system uses the RSA 64-bit encryption algorithm for encryption storage of the data, and the system uses the CRC check to prevent the data from being tampered, so as to realize the safe storage of the data.

[0055] The application discloses a regulation and control cloud platform data quality collaborative optimization system, based on the characteristics of cloud computing and edge computing, a regulation and control cloud platform data quality collaborative optimization system under a two-level architecture of regulation and control cloud considering node resource management, node network management and global security management is proposed, the computing and storage resources of the coordination node, the network delay between nodes, the transmission energy consumption and other influencing factors are considered, and a global security management mechanism under the two-level cloud architecture is established, the computing and storage resources of the coordination nodes of each unit are uniformly optimized and dispatched, the computing pressure of the regulation and control cloud center and the bandwidth pressure of the wide area data network are relieved, and integrated calculation of the data quality of the whole network is realized.

[0056] Referring to Figure 4 In another embodiment of the application, a regulation and control cloud platform data quality collaborative optimization method is provided, which can be realized based on the above regulation and control cloud platform data quality collaborative optimization system, and specifically, the regulation and control cloud platform data quality collaborative optimization method comprises the following steps:

[0057] S1: the data quality management module is used to manage the data quality tasks of the leading node and the coordination node, manage the data acquisition range and the data processing rules of the coordination node, and manage the data quality evaluation results of the coordination node.

[0058] S2: the node resource management module is used to locate the node resources of the coordination node, monitor the node resources of the coordination node, acquire the node load rate of the coordination node and determine the coordination node in communication with the leading node according to the node load rate, and distribute the data quality tasks to the coordination node according to the node resources.

[0059] S3: the node network management module is used to set a unified interface and a communication protocol for the leading node and the coordination node.

[0060] S4: managing the information transmission security between the collaborative nodes and between the collaborative nodes and the leading node through the global security management module, monitoring and auditing the collaborative nodes, managing the collaborative nodes, and performing data encryption storage.

[0061] The division of the modules in the embodiments of the present application is illustrative, and is merely a logical function division, and in actual implementation, there can be another division manner, and in addition, each function module in each embodiment of the present application can be integrated in one processor, or can be a separate physical existence, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software function module.

[0062] In still another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used to regulate the operation of the cloud platform data quality collaborative optimization method.

[0063] In still another embodiment of the present application, the present application also provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, used for storing programs and data. It can be understood that the computer readable storage medium here can include the built-in storage medium in the computer device, and of course can also include the extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium here can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the method for regulating and controlling the cloud platform data quality collaborative optimization in the above embodiments.

[0064] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks Figure 1 The means for implementing the function specified in one or more flows and / or blocks.

[0066] These computer program instructions can also be stored in a computer readable storage medium capable of directing the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction means, which implements the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocksFigure 1 the function specified in the one or more blocks.

[0067] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flows Figure 1 the flows or the flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0068] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered in the protection scope of the claims of the present application.

Claims

1. A collaborative optimization system for regulating data quality on a cloud platform, characterized in that, The control cloud platform includes a master node and several collaborative nodes; the control cloud platform data quality collaborative optimization system includes: The data quality management module is used to manage the data quality tasks of the leading node and collaborating nodes, manage the data collection scope and data processing rules of the collaborating nodes, and manage the data quality assessment results of the collaborating nodes. The node resource management module is used to locate the node resources of collaborative nodes, monitor the node resources of collaborative nodes, obtain the node load rate of collaborative nodes and determine the collaborative nodes that communicate with the leading node based on the node load rate, and distribute data quality tasks to collaborative nodes based on the node resources. The node network management module is used to set up a unified interface and communication protocol for the leading node and collaborating nodes; The global security management module is used to manage the security of information transmission between collaborative nodes and between collaborative nodes and the master node, monitor and audit collaborative nodes, manage collaborative nodes, and perform encrypted data storage. The node resources of the location coordination node include: The resource locator of each node resource of the collaborative node is obtained through the node-service co-coding method, and the resource of each node of the collaborative node is located based on the resource locator; The node-service co-coding method includes: at the node level, encoding the region to which the node resource belongs, the node name, the node type, and the access / suspend status to obtain the first encoding; at the service level, encoding the service interface name, service IP address, service port number, service PID, and service type of the node resource to obtain the second encoding; combining the first encoding and the second encoding to obtain the resource locator of the node resource; The monitoring of the node resources of the collaborative node includes: by accessing the operating system kernel file of the collaborative node, obtaining in real time the operating system and resource load information of each process of the collaborative node, and obtaining the real-time values ​​of the monitoring indicators of the server and virtual machine of the collaborative node.

2. The cloud platform data quality collaborative optimization system according to claim 1, characterized in that, in, Monitoring metrics include one or more of the following: CPU utilization, memory usage, disk usage, network traffic, disk I / O read and write information, process memory usage information, system logs, database connection count, distribution information and status of each data node, and traffic of each port of network devices.

3. The cloud platform data quality collaborative optimization system according to claim 1, characterized in that, The step of obtaining the node load rate of the cooperating nodes and determining the cooperating nodes that communicate with the dominant node based on the node load rate includes: Obtain the node load rate of the collaborating nodes; where the node load rate is either the CPU load rate or the memory load rate. Based on the node load rate of the collaborating nodes, communication is conducted between the leading node and collaborating nodes whose node load rate is lower than a preset node load rate threshold, or communication is conducted between the leading node and collaborating nodes in order of node load rate from low to high.

4. The cloud platform data quality collaborative optimization system according to claim 1, characterized in that, The step of distributing data quality tasks to collaborative nodes based on the node resources includes: Based on the node resources, a wide area service sharing technology based on gRPC is used to distribute data quality tasks to collaborative nodes according to geographical location and scheduling mechanism.

5. The cloud platform data quality collaborative optimization system according to claim 1, characterized in that, The security of information transmission between management collaboration nodes and between collaboration nodes and the dominant node includes: By segmenting the network, separating business management, preventing malicious code, and verifying identity, the security of information transmission between collaborative nodes and between collaborative nodes and the leading node is managed. The monitoring and auditing collaboration nodes include: Perform account verification, resource locking and unlocking, and log auditing for collaborative nodes.

6. The cloud platform data quality collaborative optimization system according to claim 1, characterized in that, The management collaboration nodes include: The system includes mutual identity recognition between the leading node and collaborating nodes, as well as the registration, maintenance, whitelist management, and IP configuration of collaborating nodes. Node information includes node name, node description, node type, flag, message subject, operation time, and owner. The data encryption and storage includes: The data is verified using CRC check and encrypted using the RSA 64-bit encryption algorithm.

7. A method for collaborative optimization of data quality in a cloud platform based on the collaborative optimization system for data quality in a cloud platform according to any one of claims 1 to 6, characterized in that, include: The data quality management module manages the data quality tasks of the leading node and collaborating nodes, manages the data collection scope and data processing rules of the collaborating nodes, and manages the data quality assessment results of the collaborating nodes. The node resource management module locates the node resources of collaborative nodes, monitors the node resources of collaborative nodes, obtains the node load rate of collaborative nodes, determines the collaborative nodes that communicate with the leading node based on the node load rate, and distributes data quality tasks to collaborative nodes based on the node resources. The node network management module sets up a unified interface and communication protocol for the leading node and collaborating nodes; The global security management module manages the security of information transmission between collaborative nodes and between collaborative nodes and the master node, monitors and audits collaborative nodes, manages collaborative nodes, and performs encrypted data storage.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the collaborative optimization method for regulating cloud platform data quality as described in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the collaborative optimization method for regulating cloud platform data quality as described in claim 7.

Images