Data center platform based on multi-layer architecture centralized management

Abstract

The invention discloses a data center platform based on multi-layer architecture centralized management, which relates to the technical field of data management and comprises a basic resource layer, a data resource layer, a data platform layer, an application platform layer and a front-end service interaction layer, the basic resource layer is connected with the data resource layer, the data resource layer is connected with the data platform layer, the data platform layer is connected with the application platform layer, and the application platform layer is connected with the front-end service interaction layer; the data platform layer divides an application program into a plurality of independent service units through a micro-service architecture, and each independent service unit communicates through a RESTful API (Application Program Interface); the matched supporting system comprises a system mechanism system, a standard specification system, an operation and maintenance guarantee system and a safety protection system. According to the system, a multi-layer architecture and micro-service design is adopted, and efficient management and safety application of large-scale energy data are realized through organic combination of technical characteristics such as intelligent data processing, hierarchical storage, multiple safety protection and personalized service.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data center platform based on multi-layer architecture and centralized management. Background Art

[0002] As energy consumption continues to grow, the scale and complexity of energy data are also expanding. In order to meet this challenge, the traditional data center management model can no longer meet the current large-scale and diverse energy data management needs. Against the backdrop of a surge in energy data volume and diverse data sources, traditional centralized data centers are unable to cope with the processing power and flexibility, and are unable to adapt to the new requirements of data management.

[0003] In order to solve the problem of centralized management of energy data, improve the efficiency of data sharing and integration, ensure system security and reliability, and enhance the scalability and flexibility of the system, the Guizhou Energy Data Center Scenario Construction Project proposed an innovative technical solution. By building a comprehensive energy data center platform, this solution can not only centrally manage and process large-scale energy data, but also realize the effective sharing and integration of data, and improve the availability and value of data. At the same time, the platform adopts advanced security technologies and measures to protect the security and privacy of the data center when processing sensitive data. In addition, the design of the platform takes into account the needs of business growth and changes. Through microservice architecture and modular design, it ensures the system's rapid adaptability and expansion capabilities to meet possible challenges in the future. This technical background provides a solid foundation for the implementation of the Guizhou Energy Data Center Scenario Construction Project. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the present invention provides a data center platform based on a multi-layer architecture and centralized management, which can solve the problems mentioned in the background technology.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a data center platform based on a multi-layer architecture and centralized management, comprising: a basic resource layer, a data resource layer, a data platform layer, an application platform layer, and a front-end business interaction layer; wherein,

[0007] The basic resource layer is connected to the data resource layer, the data resource layer is connected to the data platform layer, the data platform layer is connected to the application platform layer, and the application platform layer is connected to the front-end business interaction layer;

[0008] The data platform layer divides the application into multiple independent service units through a microservice architecture, and each independent service unit communicates through a RESTful API;

[0009] The supporting system includes a system of institutional mechanisms, a system of standards and specifications, an operation and maintenance guarantee system, and a security protection system. The supporting system is respectively connected to the basic resource layer, the data resource layer, the data platform layer, the application platform layer, and the front-end business interaction layer.

[0010] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: a data acquisition unit is arranged between the basic resource layer and the data resource layer, and the data acquisition unit includes a data extraction module and a data preprocessing module;

[0011] When the data extraction module collects data, the data preprocessing module cleans the collected data, and the data preprocessing module transmits the cleaned data to the data resource layer.

[0012] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the data platform layer includes a data storage module and a data analysis module;

[0013] If a data storage request is received, the data storage module stores the data in a structured data area or an unstructured data area according to the data type;

[0014] If a data analysis request is received, the data analysis module retrieves the data to be analyzed from the data storage module.

[0015] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: each of the independent service units includes a first API interface, and a communication connection is established between the first API interfaces of two adjacent independent service units through a RESTful API;

[0016] If the first independent service unit sends a service call request to the second independent service unit, the second independent service unit returns service response data.

[0017] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the standard specification system includes a data resource display specification unit, a data classification storage specification unit and a data application specification unit; if the data platform layer receives new data, the data classification storage specification unit will classify the data; if the data needs to be displayed, the data resource display specification unit will select the corresponding display template according to the display scenario.

[0018] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the standard specification system includes a data resource display specification unit, a data classification storage specification unit and a data application specification unit;

[0019] The data classification storage specification unit establishes a three-level classification system for data, specifically:

[0020] If the data belongs to the first level, it corresponds to the basic business data storage area;

[0021] If the data belongs to the second level, then the corresponding analysis and mining data storage area;

[0022] If the data belongs to the third level, it corresponds to the derived application data storage area;

[0023] The data resource display specification unit allocates different data display templates according to data levels, and the data application specification unit establishes a data mapping table for cross-region data access.

[0024] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the security protection system includes a data encryption unit and an access control unit; the data encryption unit includes a multiple encryption mechanism;

[0025] If the data is structured data, field-level encryption is used; if the data is unstructured data, file-level encryption is used;

[0026] The access control unit establishes a multi-dimensional access control matrix, which includes a user dimension, a data dimension, and a time dimension, and determines data access rights according to calculation results of the access control matrix.

[0027] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the security protection system also includes a security monitoring unit; the security monitoring unit establishes a user access behavior model, and the user access behavior model includes time characteristics, frequency characteristics and data association characteristics;

[0028] If the deviation value between the detected user access behavior and the user access behavior model exceeds a first preset threshold, the security monitoring unit generates an abnormal warning signal; if the density of the abnormal warning signal exceeds a second preset threshold, the access control unit automatically adjusts the authority level of the access control matrix.

[0029] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the operation and maintenance guarantee system includes a system monitoring unit and a fault handling unit; the system monitoring unit establishes a system health evaluation model, and the system health evaluation model includes resource utilization index, performance index and stability index;

[0030] If the evaluation result of the system health evaluation model is lower than a third preset threshold, the self-healing mechanism of the fault processing unit is triggered; the self-healing mechanism includes dynamic resource adjustment and service degradation processing.

[0031] As a preferred solution of the data center platform based on multi-layer architecture centralized management described in the present invention, wherein: the front-end business interaction layer includes a data display module and a user interaction module; the user interaction module establishes a user portrait model, and the user portrait model includes user role characteristics, data access characteristics and business operation characteristics;

[0032] The data display module automatically adjusts the data display strategy according to the user portrait model. If the user portrait model shows that the user prefers data analysis, the data analysis results are displayed first. If the user portrait model shows that the user prefers original data, the original data set is displayed first.

[0033] Beneficial effects of the present invention: The present invention realizes hierarchical management and processing of energy data through multi-layer architecture design, and the design based on microservice architecture improves the flexibility and scalability of the system; through automated data collection and preprocessing mechanism, combined with three-level classification storage strategy, data management efficiency is improved; differentiated encryption strategy and multi-dimensional access control, combined with user behavior monitoring, a complete security protection system is constructed; through system health assessment and self-healing mechanism, the stability of system operation is improved; based on personalized display strategy of user portrait, the user experience is optimized. The organic combination of these technical features enables the present invention to effectively solve the management problems of large-scale energy data, and provide efficient data services while ensuring data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 This is an overall structural diagram of a data center platform based on multi-layer architecture and centralized management proposed by the present invention. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0038] Example 1, reference Figure 1 , which is an embodiment of the present invention, provides a data center platform with centralized management based on a multi-layer architecture.

[0039] It adopts a layered architecture design, including the basic resource layer, data resource layer, data platform layer, application platform layer and front-end business interaction layer, and is equipped with a corresponding supporting system. Each layer is interconnected through standardized interfaces, forming a complete data management ecosystem.

[0040] Among them, the basic resource layer is connected to the data resource layer, the data resource layer is connected to the data platform layer, the data platform layer is connected to the application platform layer, and the application platform layer is connected to the front-end business interaction layer. The data platform layer divides the application into multiple independent service units through the microservice architecture, and each independent service unit communicates through the RESTful API. The supporting system includes the system of institutional mechanisms, the system of standards and specifications, the operation and maintenance guarantee system, and the security protection system. The supporting system is connected to the basic resource layer, the data resource layer, the data platform layer, the application platform layer, and the front-end business interaction layer respectively.

[0041] Specifically, the basic resource layer is mainly responsible for the collection and storage of raw data, including the operation data of various energy equipment, environmental monitoring data, user consumption data, etc. These data come from various sources, including but not limited to smart meters, environmental sensors, energy trading systems, etc. The basic resource layer adopts a distributed storage architecture to ensure efficient storage and fast access to massive data.

[0042] A data acquisition unit is provided between the data resource layer and the basic resource layer. The data acquisition unit includes a data extraction module and a data preprocessing module. Among them, the data extraction module is responsible for extracting raw data from the basic resource layer. After the data extraction module collects the data, the data preprocessing module will clean the data, including removing noise data, filling missing values, and standardizing. The preprocessed data is then transmitted to the data resource layer for further management and storage.

[0043] The data platform layer is the core processing layer of the entire system, including the data storage module and the data analysis module. The data storage module stores the data in the structured data area or the unstructured data area according to the data type. Specifically, standardized data such as equipment operating parameters and user energy consumption data are stored in the structured data area, while unstructured data such as pictures, videos, and documents are stored in the unstructured data area. If a data storage request is received, the data storage module stores the data in the structured data area or the unstructured data area according to the data type; when the system receives a data analysis request, the data analysis module will retrieve the corresponding data to be analyzed from the data storage module for processing.

[0044] The data platform layer adopts a microservice architecture design to divide the application into multiple independent service units. Each independent service unit is equipped with a first API interface, and adjacent independent service units establish a communication connection through the RESTful API between their first API interfaces. For example, when the first independent service unit needs to call the service of the second independent service unit, it will send a service call request through the RESTful API, and the second independent service unit will return the corresponding service response data after receiving the request.

[0045] The supporting system includes the system of institutional mechanisms, the system of standards and specifications, the system of operation and maintenance guarantee, and the system of security protection. Among them, the standard and specification system includes the data resource display specification unit, the data classification and storage specification unit, and the data application specification unit. The data classification and storage specification unit has established a three-level classification system: the first level corresponds to the basic business data storage area, which stores the basic data generated by daily business; the second level corresponds to the analysis and mining data storage area, which stores the processed and analyzed data; the third level corresponds to the derivative application data storage area, which stores the application data further processed based on the basic data and analysis data. The data resource display specification unit will assign different display templates according to the level of the data, while the data application specification unit is responsible for establishing a data mapping table for cross-region data access to ensure the interoperability between data of different levels.

[0046] The security protection system consists of a data encryption unit, an access control unit, and a security monitoring unit. The data encryption unit uses multiple encryption mechanisms and adopts different encryption strategies for different types of data: for structured data, field-level encryption is used, and sensitive fields can be encrypted separately; for unstructured data, file-level encryption is used to ensure the security of the entire file. The access control unit establishes a multi-dimensional access control matrix, including user dimension, data dimension, and time dimension. The system determines the user's data access rights based on the calculation results of this matrix.

[0047] The security monitoring unit has established a user access behavior model, which includes time features, frequency features, and data association features. The system monitors the user's access behavior in real time. When the deviation value between the detected behavior and the model exceeds the first preset threshold (for example, set to 0.8), the security monitoring unit will generate an abnormal warning signal. If the density of abnormal warning signals exceeds the second preset threshold (for example, more than 3 warnings are generated within 10 minutes), the access control unit will automatically adjust the permission level of the access control matrix to strengthen security protection.

[0048] The operation and maintenance guarantee system includes a system monitoring unit and a fault handling unit. The system monitoring unit establishes a system health assessment model, which comprehensively considers resource utilization indicators (such as CPU usage, memory occupancy, etc.), performance indicators (such as response time, concurrent processing capabilities, etc.) and stability indicators (such as system running time, error rate, etc.). When the evaluation result of the system health assessment model is lower than the third preset threshold (for example, set to 75%), the self-healing mechanism of the fault handling unit will be triggered. This mechanism includes dynamic adjustment of resources (such as automatic capacity expansion, load balancing, etc.) and service degradation processing (such as shutting down non-core services, limiting concurrent requests, etc.).

[0049] The front-end business interaction layer includes a data display module and a user interaction module. The user interaction module establishes a user portrait model, which includes user role characteristics (such as administrators, ordinary users, etc.), data access characteristics (such as common data types, access frequency, etc.) and business operation characteristics (such as common functions, operating habits, etc.). The data display module will automatically adjust the data display strategy according to the user portrait model: for users who prefer data analysis, the system will give priority to displaying data analysis results, statistical charts, etc.; for users who prefer raw data, the system will give priority to displaying raw data sets and provide flexible data retrieval and export functions.

[0050] In specific applications, when the system receives a user request, it first performs identity authentication and permission verification through the user interaction module of the front-end business interaction layer. After verification, a personalized display strategy is determined based on the user portrait model. If a user requests access to certain data, the system will determine whether the user has the corresponding permission based on the access control matrix. After obtaining authorization, the system calls the corresponding service unit through the microservice architecture to obtain the required data from the data storage module. If data analysis is required, the data analysis module will process and return the result. During the whole process, the system monitoring unit continuously monitors the system status to ensure the stable operation of the service.

[0051] Through the above technical solutions, the data center platform provided in this embodiment realizes efficient management and secure access to energy data, meets the personalized needs of different users, and has strong scalability and fault tolerance.

[0052] In summary, the present invention realizes hierarchical management and processing of energy data through multi-layer architecture design. The design based on microservice architecture improves the flexibility and scalability of the system; through automated data collection and preprocessing mechanism, combined with three-level classification storage strategy, data management efficiency is improved; differentiated encryption strategy and multi-dimensional access control, combined with user behavior monitoring, a complete security protection system is constructed; through system health assessment and self-healing mechanism, the stability of system operation is improved; based on personalized display strategy of user portrait, the user experience is optimized. The organic combination of these technical features enables the present invention to effectively solve the management problems of large-scale energy data and provide efficient data services while ensuring data security.

[0053] Example 2 is an embodiment of the present invention, which provides a data center platform based on a multi-layer architecture and centralized management. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0054] In order to verify the technical effect of the present invention, this embodiment selects the actual business scenario of a provincial energy group for comparative testing. The test environment uses the same hardware configuration: the server is configured with Intel Xeon E5-2680v4 CPU, 256GB RAM, and the storage device is an NVMe SSD array. The test data contains 3 months of energy production and consumption data, with a total data volume of approximately 500TB, involving structured and unstructured data. The comparative test selected a traditional centralized data management platform (comparison system A) and a data management platform based on a simple microservice architecture (comparison system B) as references. The test content includes multiple dimensions such as data processing performance, system availability, security protection effect, and resource utilization. This embodiment adopts industry-standard testing methods to comprehensively evaluate system performance through stress testing, fault injection, security penetration testing and other means. During the test, a variety of actual scenarios such as high concurrent access, data anomalies, and network fluctuations were simulated to ensure the authenticity and reliability of the test results.

[0055] The test lasted for 30 days and collected more than 1 million test samples. In terms of performance testing, concurrent user requests ranging from 100 to 10,000 were simulated; in terms of security testing, various attack tests including SQL injection, cross-site scripting, denial of service, etc. were conducted; in terms of reliability testing, the system's fault tolerance and recovery capabilities were evaluated by randomly shutting down service nodes, simulating network delays, triggering data anomalies, etc. At the same time, this embodiment also continuously monitors the system's resource consumption, including key indicators such as CPU usage, memory usage, and storage I / O. To ensure the objectivity of the test, all test cases and data are verified by a third-party testing organization. After the test data is standardized, an industry-recognized evaluation model is used for comprehensive analysis and evaluation.

[0056] Table 1 System performance comparison data

[0057] Test indicators Traditional centralized platform Simple microservice platform Platform of the present invention Performance improvement ratio Data processing speed (GB / s) 0.8 1.2 2.5 108.33％ System response time (ms) 850 650 320 50.77％ Concurrent processing capacity (number of requests per second) 2000 3500 8000 128.57％ Fault recovery time (minutes) 45 30 12 60.00％ Data loading delay (seconds) 25 18 8 55.56％ Resource utilization (%) 45 60 85 41.67％ System availability (%) 99.9 99.95 99.99 0.04％ Safety protection success rate (%) 92 95 99.5 4.74％ Data processing accuracy (%) 95 97 99.8 2.89％ Storage space utilization (%) 65 75 90 20.00％

[0058] Through the analysis of the test data, the following conclusions can be drawn: the present invention is significantly superior to the comparison system in all key indicators. Specifically, in terms of data processing speed, the present invention reaches 2.5GB / s, which is 108.33% higher than the simple microservice platform; the system response time is shortened to 320ms, which is more than 50% higher than the average comparison system; the concurrent processing capacity reaches 8000 requests / second, which is 128.57% higher than the simple microservice platform; the fault recovery time is reduced to 12 minutes, which is more than 60% shorter than the average comparison system; the system resource utilization rate is increased to 85%, which is 41.67% higher than the traditional platform; in terms of security, the protection success rate reaches 99.5%, which is about 4.74% higher than the comparison system; the data processing accuracy rate reaches 99.8%, which is about 2.89% higher; the storage space utilization rate is increased to 90%, which is 20% higher than the traditional platform. These data fully demonstrate the significant advantages of the present invention in terms of performance, reliability, security and resource utilization efficiency. It is particularly noteworthy that the system availability has reached a high level of 99.99%, which is a very competitive indicator in large-scale data processing systems, indicating that the present invention has extremely high stability and reliability. At the same time, the test data also shows that with the increase in the number of concurrent users and the amount of data processed, the performance advantage of the present invention is more obvious, indicating that the platform has good scalability and stability.

[0059] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data center platform based on a multi-layer architecture and centralized management, characterized in that: It includes basic resource layer, data resource layer, data platform layer, application platform layer, and front-end business interaction layer; among them, The basic resource layer is connected to the data resource layer, the data resource layer is connected to the data platform layer, the data platform layer is connected to the application platform layer, and the application platform layer is connected to the front-end business interaction layer; The data platform layer divides the application into multiple independent service units through a microservice architecture, and each independent service unit communicates through a RESTful API; The supporting system includes a system of institutional mechanisms, a system of standards and specifications, an operation and maintenance guarantee system, and a security protection system. The supporting system is respectively connected to the basic resource layer, the data resource layer, the data platform layer, the application platform layer, and the front-end business interaction layer.

2. The data center platform based on multi-layer architecture centralized management as claimed in claim 1, characterized in that: A data acquisition unit is provided between the basic resource layer and the data resource layer, and the data acquisition unit includes a data extraction module and a data preprocessing module; When the data extraction module collects data, the data preprocessing module cleans the collected data, and the data preprocessing module transmits the cleaned data to the data resource layer.

3. The data center platform based on multi-layer architecture centralized management as claimed in claim 2, characterized in that: The data platform layer includes a data storage module and a data analysis module; If a data storage request is received, the data storage module stores the data in a structured data area or an unstructured data area according to the data type; If a data analysis request is received, the data analysis module retrieves the data to be analyzed from the data storage module.

4. The data center platform based on multi-layer architecture centralized management as claimed in claim 3, characterized in that: Each of the independent service units includes a first API interface, and a communication connection is established between the first API interfaces of two adjacent independent service units via a RESTful API; If the first independent service unit sends a service call request to the second independent service unit, the second independent service unit returns service response data.

5. The data center platform based on multi-layer architecture centralized management as claimed in claim 4, characterized in that: The standard specification system includes a data resource display specification unit, a data classification and storage specification unit and a data application specification unit; if the data platform layer receives new data, the data classification and storage specification unit will classify the data; if the data needs to be displayed, the data resource display specification unit will select a corresponding display template according to the display scenario.

6. The data center platform based on multi-layer architecture centralized management as claimed in claim 5, characterized in that: The standard specification system includes a data resource display specification unit, a data classification storage specification unit and a data application specification unit; The data classification storage specification unit establishes a three-level classification system for data, specifically: If the data belongs to the first level, it corresponds to the basic business data storage area; If the data belongs to the second level, then the corresponding analysis and mining data storage area; If the data belongs to the third level, it corresponds to the derived application data storage area; The data resource display specification unit allocates different data display templates according to data levels, and the data application specification unit establishes a data mapping table for cross-region data access.

7. The data center platform based on multi-layer architecture centralized management as claimed in claim 6, characterized in that: The security protection system includes a data encryption unit and an access control unit; the data encryption unit includes a multiple encryption mechanism; If the data is structured data, field-level encryption is used; if the data is unstructured data, file-level encryption is used; The access control unit establishes a multi-dimensional access control matrix, which includes a user dimension, a data dimension, and a time dimension, and determines data access rights according to calculation results of the access control matrix.

8. The data center platform based on multi-layer architecture centralized management as claimed in claim 7, characterized in that: The security protection system also includes a security monitoring unit; the security monitoring unit establishes a user access behavior model, and the user access behavior model includes time characteristics, frequency characteristics and data association characteristics; If the deviation value between the detected user access behavior and the user access behavior model exceeds a first preset threshold, the security monitoring unit generates an abnormal warning signal; If the density of the abnormal warning signal exceeds a second preset threshold, the access control unit automatically adjusts the authority level of the access control matrix.

9. The data center platform based on multi-layer architecture centralized management as claimed in claim 8, characterized in that: The operation and maintenance guarantee system includes a system monitoring unit and a fault handling unit; the system monitoring unit establishes a system health evaluation model, and the system health evaluation model includes resource utilization indicators, performance indicators and stability indicators; If the evaluation result of the system health evaluation model is lower than a third preset threshold, the self-healing mechanism of the fault processing unit is triggered; the self-healing mechanism includes dynamic resource adjustment and service degradation processing.

10. The data center platform based on multi-layer architecture centralized management as claimed in claim 9, characterized in that: The front-end business interaction layer includes a data display module and a user interaction module; the user interaction module establishes a user portrait model, and the user portrait model includes user role characteristics, data access characteristics and business operation characteristics; The data display module automatically adjusts the data display strategy according to the user portrait model. If the user portrait model shows that the user prefers data analysis, the data analysis results are displayed first. If the user portrait model shows that the user prefers original data, the original data set is displayed first.

Images