Multi-source heterogeneous data management method and system for new energy mass data

Abstract

The invention discloses a multi-source heterogeneous data management method and system for new energy mass data, and the system comprises a data standardization layer, a data driving layer, a task scheduling layer, an operator management layer and a service interface layer which are sequentially communicated through a data bus. Unified management of SCADA data, CMS data, relational database data and time sequence database data is realized, the data is subjected to standardized processing by adopting a data quality evaluation model, and the data access efficiency is improved; distributed storage of data is realized through a dynamic fragmentation strategy, and the utilization rate of storage resources is optimized; task scheduling is performed based on a multi-factor scoring mechanism, so that the task processing efficiency is improved; and a standardized operator management mechanism is adopted, so that the expansibility of the system is enhanced. All levels of the system are in close cooperation through a data processing bus to form a complete data processing chain, and automation and intellectualization of data processing are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy big data, and in particular relates to a multi-source heterogeneous data management method and system for new energy massive data. Background Art

[0002] The new energy industry generates a large amount of heterogeneous data, including SCADA system data of wind farms, CMS status monitoring data, business data in relational databases, and historical data in time series databases. These data are multi-source, heterogeneous, and real-time, and need to be efficiently processed through a unified data management method. The data management system needs to handle a series of complex operations such as data format conversion, data distribution and storage, and task scheduling and execution, and is an important basis for the analysis and application of new energy data.

[0003] At present, the new energy field mainly uses two types of data management systems: one is the data management platform in the field of industrial control systems, such as solutions from Siemens, Emerson and other companies. This type of system is strongly coupled with a specific hardware platform and does not have the versatility of data processing; the other is a general big data processing platform, which requires professional technicians to develop and maintain, and has limitations when processing data types unique to the new energy field. Both types of systems face problems such as low data access efficiency, poor scalability, and complex operation and maintenance when processing massive heterogeneous data in new energy.

[0004] Due to the characteristics of new energy data such as multi-source heterogeneity, strong real-time nature and large data volume, existing technologies find it difficult to achieve unified and efficient management of data. In particular, in terms of data standardization processing, there is a lack of a unified processing mechanism for the characteristics of new energy data, resulting in low data processing efficiency and difficulty in system expansion, which cannot meet the needs of the rapid development of the new energy industry. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-source heterogeneous data management method and system for new energy massive data, so as to overcome the problems of low efficiency, poor scalability and complex operation and maintenance in the prior art for processing new energy massive heterogeneous data.

[0006] In order to solve the above problems, the present invention adopts the following technical solutions: A multi-source heterogeneous data management system for new energy mass data includes a data standardization layer, a data drive layer, a task scheduling layer, an operator management layer, and a service interface layer. Adjacent layers are connected through a data bus, wherein: Data standardization layer, used to identify the format, standardize and evaluate the quality of data; Data driver layer, used for data distribution and storage management; The task scheduling layer is used to prioritize data and allocate resources; The operator management layer is used to provide full life cycle management of operators; The service interface layer is used to provide a unified data access interface and service management.

[0007] Furthermore, the data sources in the data standardization layer include equipment operation data generated by the SCADA system, status monitoring data collected by the CMS system, business data stored in the relational database, and historical data recorded in the time series database. Furthermore, the data standardization layer includes a format recognition module, a field mapping module, a data transformation module and a quality assessment module which are connected in sequence. Furthermore, the quality assessment is implemented through a data quality assessment model, and the data quality assessment model formula is:

[0008] Among them, Q represents the data quality score, n represents the number of evaluation indicators, represents the indicator weight coefficient, represents the indicator evaluation function, Indicates the actual value of the indicator. Furthermore, the data driver layer includes a data slicing module, a memory allocation module and a data distribution module which are connected in sequence. Furthermore, the data driver layer performs data distribution and storage management through a sharding strategy. Furthermore, the calculation formula of the sharding strategy is:

[0009] In the formula, S represents the calculated shard size; M represents the total amount of data to be processed; N represents the number of currently available computing nodes; L represents the current load factor of the system; is the basic sharding coefficient; is the load regulation factor. Furthermore, the task scheduling layer includes a task parsing module, a resource allocation module and an execution monitoring module connected in sequence. The task scheduling layer uses a multi-factor scoring mechanism to prioritize tasks. The task priority calculation formula is:

[0010] In the formula, P represents the final priority score of the task; T represents the timeliness factor of the task; R represents the resource demand factor; U represents the user level factor; , and is the weight coefficient, and the sum of the three is 1. Furthermore, the operator management layer includes an operator registration module, an operator configuration module, an operator calling module and an operator monitoring module which are connected in sequence. In a second aspect, a method for managing multi-source heterogeneous data for new energy mass data includes the following steps: Parse the access data, establish field mapping relationships and conduct quality assessment through the data standardization layer; Slice the data after quality assessment and allocate storage space; Analyze the data after sharding to extract task description information, evaluate task priority and allocate resources; Determine the processing operator according to the task type, perform parameter configuration and environment initialization, and monitor the running status in real time during the operator execution; Summarize and integrate the processing results of each operator, unify the data format, add metadata information, and output the processing results through the service interface layer.

[0011] Compared with the prior art, the present invention has the following beneficial technical effects: The invention provides a multi-source heterogeneous data management system for new energy mass data, comprising a data standardization layer, a data driving layer, a task scheduling layer, an operator management layer and a service interface layer which are sequentially connected by a data bus, wherein the data processing bus provides a reliable message routing, flow control and state synchronization mechanism to ensure efficient collaboration between various components of the system; the data standardization layer performs format recognition and standardization processing on SCADA data, CMS data, relational database data and time series database data, and adopts a data quality assessment model to perform quality assessment; the data driving layer performs data distribution and storage management through a sharding strategy, and the sharding strategy is dynamically adjusted according to data characteristics and system load; the task scheduling layer adopts a multi-factor scoring mechanism to perform priority sorting and resource allocation on tasks; the operator management layer provides full life cycle management of operators, including operator registration, parameter configuration, execution management and state monitoring; the service interface layer provides a unified data access interface and service management through an API gateway; and by establishing a multi-level data management architecture, unified management of SCADA data, CMS data, relational database data and time series database data is realized.

[0012] The data standardization layer uses a data quality assessment model to standardize SCADA data, CMS data, etc. The data-driven layer distributes and manages data through sharding strategies. The task scheduling layer uses a multi-factor scoring mechanism to prioritize tasks. The operator management layer provides operator life cycle management. The service interface layer provides a unified access interface through an API gateway. The system connects each level through a data processing bus to achieve data exchange and control instruction transmission, forming a complete data processing chain, realizing the automation and intelligence of data processing, and improving the management efficiency of massive heterogeneous data of new energy.

[0013] Preferably, the data standardization layer uses a data quality assessment model to standardize the data. The system can accurately assess the data quality level, provide a basis for subsequent processing, and improve data access efficiency.

[0014] Preferably, the data driver layer performs data distribution and storage management through a sharding strategy, effectively balancing system processing capabilities and resource utilization efficiency.

[0015] Preferably, the task scheduling layer performs priority sorting based on a multi-factor scoring mechanism. The system reasonably arranges the execution order of tasks according to the calculated priority scores to ensure that key tasks are processed first.

[0016] Preferably, the operator management layer adopts a standardized operator management mechanism to achieve full life cycle management of data processing operators, thereby enhancing the scalability of the system.

[0017] The present invention provides a multi-source heterogeneous data management method for new energy mass data, which realizes the unified management of SCADA data, CMS data, relational database data and time series database data through multi-level architecture design and standardized processing flow, solves the technical problems of inconsistent data format, low processing efficiency, poor scalability and the like in the prior art, significantly improves the processing efficiency and management level of new energy data, realizes the efficient management of massive heterogeneous data in the new energy field, solves the technical problems of low data processing efficiency, poor scalability and the like, and has significant practical value and promotion significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an overall architecture diagram of a multi-source heterogeneous data management system for new energy mass data in an embodiment of the present invention; Figure 2 This is a flow chart of the data sharding strategy in an embodiment of the present invention; Figure 3 It is a task scheduling flow chart in an embodiment of the present invention; Figure 4 This is the operator management structure diagram in the present invention; Figure 5It is the service interface hierarchy diagram in the present invention; Figure 6 The present invention is a flow chart of a method for managing multi-source heterogeneous data for new energy mass data. DETAILED DESCRIPTION

[0019] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail in the following specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] The present invention provides a multi-source heterogeneous data management system for new energy mass data, such as Figure 1 As shown in the figure, the system adopts a multi-level architecture design to realize the modularization and standardization of data processing. The system includes five core functional levels from top to bottom, and each level is interconnected through a data processing bus; Among them, the service interface layer is at the top layer, providing a unified access entrance for external systems; the operator management layer is at the second layer, used for the full life cycle management of data processing operators; the task scheduling layer is at the third layer, used for data priority sorting and resource allocation; the data driver layer is at the fourth layer, responsible for the distributed storage and management of data; the data standardization layer is at the bottom layer, used for data format recognition, standardization processing and quality assessment, to achieve standardized processing of multi-source heterogeneous data; data exchange and control instruction transmission are carried out between the various levels of the system through the data processing bus, which provides reliable message routing, flow control and state synchronization mechanisms to ensure efficient collaboration between the various components of the system.

[0025] The multi-source heterogeneous data management method and system provided by the present invention realize the efficient management of massive heterogeneous data in the new energy field, solve the technical problems such as low data processing efficiency and poor scalability, and have significant practical value and promotion significance.

[0026] In one embodiment of the present invention, a multi-source heterogeneous data management system for new energy mass data is provided. Figure 1 As shown in the figure, the system adopts a multi-level architecture design to realize the modularization and standardization of data processing. The system includes five core functional levels from top to bottom, and each level is interconnected through the data processing bus. Among them, the service interface layer is located at the top layer, which consists of the basic service interface, the extended service interface and the management service interface, providing a unified access entrance for the external system; the operator management layer is located at the second layer, including four functional modules: operator registration, operator configuration, operator call and operator monitoring, to achieve the full life cycle management of data processing operators; the task scheduling layer is located at the third layer, through the three functional modules of task parsing, resource allocation and execution monitoring, to complete the task scheduling and resource management; the data driver layer is located at the fourth layer, consisting of three functional modules: data sharding, memory allocation and data distribution, responsible for the distributed storage and management of data; the data standardization layer is located at the bottom layer, including four functional modules: format recognition, field mapping, data conversion and quality assessment, to achieve standardized processing of multi-source heterogeneous data. Data exchange and control instruction transmission are carried out between the various levels of the system through the data processing bus. The data processing bus provides a reliable message routing, flow control and state synchronization mechanism to ensure efficient collaboration between the various components of the system.

[0027] The data standardization layer is the basic layer of the system, responsible for data access and standardization processing. It includes the format recognition module, field mapping module, data transformation module and quality assessment module connected in sequence. This layer receives data from different sources through multiple types of data source adapters and performs data format conversion and field mapping. The data source adapter supports multiple data sources such as SCADA system, CMS system, relational database and time series database, and can accurately identify and parse various data formats. The format conversion engine is responsible for converting data in different formats into the system standard format to ensure data consistency and processability.

[0028] The data driver layer implements the distributed storage and management functions of data, including the sequentially connected data sharding module, memory allocation module and data distribution module. Figure 2 As shown in Figure 1, this layer processes and distributes data through the shard manager. The storage engine is responsible for the physical storage of data, supports multiple storage media, and provides an efficient data reading and writing mechanism. The scheduler is responsible for the scheduling and execution of data processing tasks to ensure efficient data processing.

[0029] The task scheduling layer is responsible for the task management and resource scheduling of the entire system, including the task parsing module, resource allocation module and execution monitoring module connected in sequence. Figure 3 As shown in the figure, the task parser is responsible for parsing the task description file and extracting task parameters and dependencies. The resource manager maintains the system resource pool and dynamically allocates computing resources according to task requirements to achieve efficient resource utilization. This layer ensures that key tasks are processed in a timely manner through the task priority evaluation mechanism.

[0030] The operator management layer provides full life cycle management for operators, including the operator registration module, operator configuration module, operator call module, and operator monitoring module that are connected in sequence. Figure 4 As shown in Figure 1, the operator registration center maintains the operator registration information, including operator type, function description, parameter definition, version information, etc. The operator call engine is responsible for the instantiation and operation management of the operator to ensure that the operator can correctly perform data processing tasks.

[0031] The service interface layer is the unified entrance for the system's external services, including the basic service interface module, the extended service interface module, and the management service interface module. Figure 5 As shown in the figure, this layer provides a standardized service interface through the API gateway, supporting functions such as data query, task submission, and status query. The interface management module is responsible for version control and document management of the interface to ensure the standardization and availability of the interface.

[0032] In another embodiment of the present invention, a method for managing multi-source heterogeneous data for new energy mass data is provided. Figure 6As shown in the figure, this method adopts a standardized processing flow and realizes efficient management of massive heterogeneous data through five key stages: data standardization, data-driven, task scheduling, operator processing, and service response.

[0033] A multi-level data management architecture is established, which includes a data standardization layer, a data-driven layer, a task scheduling layer, an operator management layer, and a service interface layer. The data standardization layer performs format recognition and standardization processing on SCADA data, CMS data, relational database data, and time series database data, and uses a data quality assessment model to perform quality assessment. The data-driven layer distributes and manages data through a sharding strategy, and the sharding strategy is dynamically adjusted according to data characteristics and system load. The task scheduling layer uses a multi-factor scoring mechanism to prioritize tasks and allocate resources. The operator management layer provides full life cycle management of operators, including operator registration, parameter configuration, execution management, and status monitoring. The service interface layer provides a unified data access interface and service management through an API gateway.

[0034] First, in the data standardization stage, the system receives raw data from different data sources and performs format recognition and standardization. The data sources include equipment operation data generated by the SCADA system, status monitoring data collected by the CMS system, business data stored in the relational database, and historical data recorded in the time series database. By establishing a unified data quality assessment model, the data is comprehensively evaluated for quality. The calculation formula of the assessment model is:

[0035] Among them, Q represents the data quality score, ranging from 0 to 100; n represents the number of evaluation indicators, including four dimensions: completeness, accuracy, timeliness and consistency; Indicates the indicator weight coefficient, which is determined by the specific application scenario; Represents the evaluation function of each indicator, mapping the original indicator value to the standardized interval; Represents the actual value of the indicator; through this evaluation model, the system can accurately evaluate the data quality level and provide a basis for subsequent processing.

[0036] In the data-driven stage, the system dynamically calculates the optimal data sharding strategy based on data characteristics and system load. The shard size is calculated using an adaptive formula:

[0037] Where S is the calculated shard size in megabytes; M is the total amount of data to be processed; N is the number of currently available computing nodes; L is the current load factor of the system, ranging from 0 to 1; is the basic sharding coefficient, used to adjust the base sharding size; is the load adjustment coefficient, which is used to dynamically adjust the shard size according to the system load condition. This sharding strategy can effectively balance the system processing capacity and resource utilization efficiency.

[0038] The task scheduling stage uses a multi-factor scoring mechanism to prioritize tasks. The task priority calculation formula is:

[0039] In the formula, P represents the final priority score of the task; T represents the timeliness factor of the task, which reflects the urgency of the task; R represents the resource demand factor, which reflects the demand for computing resources of the task; U represents the user level factor, which reflects the priority level of the user who submitted the task; , and is the weight coefficient, and the sum of the three is 1. The system arranges the execution order of tasks reasonably according to the calculated priority scores to ensure that key tasks are processed first.

[0040] In the operator processing stage, the system first selects the appropriate processing operator according to the task type. Processing operators include data preprocessing operators, feature extraction operators, analysis and modeling operators, and result output operators. The system obtains operator information from the operator registration center to perform parameter configuration and environment initialization. During the operator execution process, the system monitors its operating status in real time, including indicators such as resource usage, processing progress, and execution efficiency, to ensure the normal operation of the operator.

[0041] The service interface layer includes: API gateway, used to provide data query interface, task submission interface and status query interface; Interface management module, used for interface version control and document management; The service management module is used to monitor the service operation status and manage service degradation strategies.

[0042] The service response phase is responsible for collecting and returning processing results. The system first summarizes and integrates the processing results of each operator, unifies the data format, adds necessary metadata information, and finally returns the processing results to the requester through a standardized interface. During the entire process, the system maintains real-time monitoring of data transmission to ensure the reliability and security of data transmission.

[0043] A method for managing multi-source heterogeneous data for new energy mass data includes the following steps: Parse the access data, establish field mapping relationships and conduct quality assessment through the data standardization layer; Slice the data after quality assessment and allocate storage space; Analyze the data after sharding to extract task description information, evaluate task priority and allocate resources; Determine the processing operator according to the task type, perform parameter configuration and environment initialization, and monitor the running status in real time during the operator execution; Summarize and integrate the processing results of each operator, unify the data format, add metadata information, and output the processing results through the service interface layer.

[0044] Detailed, such as Figure 6 As shown in the figure, in the actual operation process, each level of the system cooperates closely to form a complete data processing chain. First, in the data standardization stage, the system parses the multi-source heterogeneous data such as SCADA data and CMS data through the data format identification module, establishes a unified field mapping relationship through the field mapping conversion module, and finally the data quality assessment module evaluates the data quality. Then in the data driving stage, the calculation shard size module dynamically determines the optimal sharding parameters according to the data scale and system load, the allocation memory module divides the storage space for the data shards, and the data distribution module completes the distributed storage of the data. After entering the task scheduling stage, the task parsing module extracts the task description information, the priority calculation module evaluates the task priority level, the resource evaluation module analyzes the resource demand situation, and the task allocation module performs resource scheduling according to the evaluation results. In the operator processing stage, the operator selection module determines the processing operator according to the task type, the parameter configuration module completes the operator initialization, the execution processing module runs the data processing operation, and the status monitoring module tracks the execution status in real time. Finally, in the service response stage, the result collection module summarizes the processing results, the formatting module unifies the data format, and the result return module sends the processing results to the requester. The entire process realizes automation and intelligent data processing through strict process control and status management.

[0045] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited thereto. Within the technical concept of the present invention, the technical solution of the present invention can be subjected to a variety of simple modifications, including the combination of various technical features in any other suitable manner, and these simple modifications and combinations should also be regarded as the contents disclosed by the present invention and belong to the protection scope of the present invention.

Claims

1. A multi-source heterogeneous data management system for massive new energy data, characterized in that: It includes data standardization layer, data drive layer, task scheduling layer, operator management layer and service interface layer in sequence. Adjacent layers are connected through data buses, where: Data standardization layer, used to identify the format, standardize and evaluate the quality of data; Data driver layer, used for data distribution and storage management; The task scheduling layer is used to prioritize data and allocate resources; The operator management layer is used to provide full life cycle management of operators; The service interface layer is used to provide a unified data access interface and service management.

2. A multi-source heterogeneous data management system for new energy mass data according to claim 1, characterized in that: The data sources in the data standardization layer include equipment operation data generated by the SCADA system, status monitoring data collected by the CMS system, business data stored in the relational database, and historical data recorded in the time series database.

3. A multi-source heterogeneous data management system for new energy mass data according to claim 1, characterized in that: The data standardization layer includes a format recognition module, a field mapping module, a data conversion module and a quality assessment module which are connected in sequence.

4. A multi-source heterogeneous data management system for new energy mass data according to claim 1, characterized in that: The quality assessment is implemented through a data quality assessment model, and the data quality assessment model formula is: Among them, Q represents the data quality score, n represents the number of evaluation indicators, represents the indicator weight coefficient, represents the indicator evaluation function, Indicates the actual value of the indicator.

5. The multi-source heterogeneous data management system for new energy mass data according to claim 1 is characterized in that: The data driver layer includes a data slicing module, a memory allocation module and a data distribution module which are connected in sequence.

6. A multi-source heterogeneous data management system for new energy mass data according to claim 1, characterized in that: The data driver layer performs data distribution and storage management through sharding strategy.

7. A multi-source heterogeneous data management system for new energy mass data according to claim 6, characterized in that: The calculation formula of the sharding strategy is: In the formula, S represents the calculated shard size; M represents the total amount of data to be processed; N represents the number of currently available computing nodes; L represents the current load factor of the system; is the basic sharding coefficient; is the load regulation factor.

8. The multi-source heterogeneous data management system for new energy mass data according to claim 1 is characterized in that: The task scheduling layer includes a task parsing module, a resource allocation module and an execution monitoring module connected in sequence. The task scheduling layer uses a multi-factor scoring mechanism to prioritize tasks. The task priority calculation formula is: In the formula, P represents the final priority score of the task; T represents the timeliness factor of the task; R represents the resource demand factor; U represents the user level factor; , and is the weight coefficient, and the sum of the three is 1.

9. The multi-source heterogeneous data management system for new energy mass data according to claim 1, characterized in that: The operator management layer includes an operator registration module, an operator configuration module, an operator calling module and an operator monitoring module which are connected in sequence.

10. A method for managing multi-source heterogeneous data for new energy mass data, characterized in that: The multi-source heterogeneous data management system for new energy mass data according to any one of claims 1 to 7 comprises the following steps: Parse the access data, establish field mapping relationships and conduct quality assessment through the data standardization layer; Slice the data after quality assessment and allocate storage space; Analyze the data after sharding to extract task description information, evaluate task priority and allocate resources; Determine the processing operator according to the task type, perform parameter configuration and environment initialization, and monitor the running status in real time during the operator execution; Summarize and integrate the processing results of each operator, unify the data format, add metadata information, and output the processing results through the service interface layer.

Images