Data hierarchical fusion storage system, storage method and related device

Abstract

The invention belongs to a storage system, and provides a data hierarchical fusion storage system, a data hierarchical fusion storage method and a related device aiming at the technical problems of tedious work and low data processing efficiency of a big data warehouse adopted for mass data storage, processing and analysis at present. Comprising an original data layer, a data initial processing layer, a data deep processing layer and a data application layer which are arranged layer by layer, data synchronization is carried out among the layers, layered and hierarchical storage is carried out on the original data layer, cooked data processing is carried out in the data initial processing layer, and data are summarized and aggregated in the data deep processing layer; and finally, connecting with an external business application interface through the data application layer. Uniform storage and flexible processing of multi-source heterogeneous data are achieved in a hierarchical and hierarchical mixed storage mode, near-real-time afflux of service data is achieved in a streaming computing mode, and batch processing of mass offline data is achieved in a batch computing mode.

Description

Technical Field

[0001] The present application relates to a storage system, and specifically to a data layered fusion storage system, a storage method and related devices. Background Art

[0002] As the scale of the power system continues to expand and its level of intelligence continues to increase, grid regulation is facing more and more challenges. The efficient development of grid regulation business has put forward higher requirements on the ability to efficiently collect, store, and analyze massive data.

[0003] At present, the storage, processing and analysis of massive data are mainly based on the big data warehouse of the Hadoop platform, relying on Hadoop, Spark, Hive, HBase and other components. HBase is mainly used for the storage of large-scale structured data. As a query and analysis engine for large-scale structured data, Hive provides a query language similar to SQL, HiveQL, for data analysis and query on the Hadoop cluster. Finally, Spark's memory-based distributed streaming computing framework is used to support multiple data processing modes during analysis and calculation. However, using HBase as a NoSQL database has high requirements for data consistency and quality, and more additional work is required to ensure the quality and accuracy of the data. In addition, relying on Hive as the data query engine of HBase to search and query massive data, the query performance is greatly affected by the structural design when scanning a large amount of data, especially when complex queries and aggregation operations are required, the performance may be affected, which seriously affects the data processing efficiency of streaming data processing frameworks such as Spark. Summary of the invention

[0004] The present application aims to provide a data hierarchical fusion storage system, storage method and related devices to address the technical problems of cumbersome work and low data processing efficiency in the current big data warehouses used for massive data storage, processing and analysis.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application proposes a data hierarchical fusion storage system, including: The original data layer is used to access the native data and store it in layers and grades; the native data includes structured data, semi-structured data and unstructured data; The data initial processing layer is used to synchronize the native data stored in the original data layer, and to initially process the multi-source data in the structured data in the original data layer into cooked data and store it; The data deep processing layer is used to synchronously store the data in the data initial processing layer, and summarize and aggregate the mature data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data, and unstructured data. After obtaining high-density data, it is stored through multi-type storage components; The data application layer is used to synchronize high-density data in the data deep processing layer, store it through multiple types of storage components, and connect to the external business application interface.

[0006] Furthermore, the original data layer uses storage components Hudi and HDFS to perform hierarchical storage on the accessed native data.

[0007] Furthermore, the layered and graded storage includes: the structured data sub-tables are accessed and stored in the storage component Hudi, and the semi-structured data and unstructured data are accessed and stored in the storage component HDFS in the form of files.

[0008] Furthermore, the structured data sub-table is accessed and stored in the storage component Hudi, including: After the structured data is generated, it is stored in tables according to different power grid objects in the form of online message packets or binary files, and is partitioned and stored after evaluation based on data service time and data volume.

[0009] Furthermore, the semi-structured data is stored in the storage component HDFS in the form of files, including: Semi-structured data is organized and stored according to business categories, business subcategories, and business time.

[0010] Furthermore, the preliminarily processing the multi-source data in the structured data in the original data layer into cooked data includes: Perform data cleansing, data instrumentation, and data aggregation on multi-source data.

[0011] Furthermore, the storage in the data initial processing layer adopts the storage component Hudi; The storage in the data deep processing layer adopts at least one of the storage component HBase, the storage component ClickHouse and the storage component MPP; The storage in the data application layer adopts the storage component ClickHouse and / or the storage component MPP.

[0012] Furthermore, it also includes: Integrate real-time data streams and historically stored batch data to obtain integrated data; The memory computing component Flink is used to realize integrated data stream and batch processing and complete synchronization.

[0013] Furthermore, the use of the memory computing component Flink to achieve data stream batch integrated processing includes: The real-time data processing, window computing, and streaming aggregation functions of the in-memory computing component Flink process and integrate real-time data streams.

[0014] In a second aspect, the present application proposes a data hierarchical fusion storage method, comprising: Access native data and perform hierarchical storage; the native data includes structured data, semi-structured data and unstructured data; Synchronously store native data, and process multi-source data in structured data into mature data and store it; Synchronize data, summarize and aggregate the mature data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data, and unstructured data, and store them through multiple types of storage components after obtaining high-density data; Synchronize high-density data, store it through multiple types of storage components, and connect to external business application interfaces.

[0015] In a third aspect, the present application proposes an electronic device, comprising: a memory and one or more processors; the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the steps of the above-mentioned data hierarchical fusion storage method.

[0016] In a fourth aspect, the present application proposes a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned data hierarchical fusion storage method are implemented.

[0017] Compared with the prior art, this application has the following beneficial effects: The present application proposes a data layered fusion storage system, including an original data layer, a data initial processing layer, a data deep processing layer and a data application layer, which are arranged layer by layer. Data synchronization is performed between the layers. The original data layer is first stored in layers and levels, and the data is processed in the data initial processing layer. The data is summarized and aggregated in the data deep processing layer, and finally connected to the external business application interface through the data application layer. The present application realizes the unified storage and flexible processing of multi-source heterogeneous data through a layered and hierarchical hybrid storage method, realizes the near real-time import of business data through streaming computing, and realizes batch processing of massive offline data through batch computing.

[0018] The present application also proposes a data hierarchical fusion storage method, an electronic device and a computer storage medium, which have all the advantages of the above-mentioned data hierarchical fusion storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of a hierarchical fusion storage system for data in this application; Figure 2 A schematic diagram of the hierarchical fusion storage method of data in this application; Figure 3 A schematic diagram of the hierarchical fusion storage method of data in this application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0023] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0024] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invented product is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0025] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0026] In the description of the embodiments of the present application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0027] With the continuous expansion of the scale of the power system and the improvement of its intelligence, power grid regulation is facing more and more challenges and complexities. The efficient development of power grid regulation business has put forward higher requirements for the efficient collection, storage, and analysis of massive data. Specifically, in terms of data collection, from the perspective of breadth, panoramic analysis requires the collection of data from all aspects of the power system, such as power generation, transmission, and distribution. From the perspective of depth, real-time calculation and analysis of power grid operation status based on massive long-term data, including load conditions, voltage stability, and line load analysis and display, so as to timely discover potential problems and take corresponding regulation measures to ensure the safe and stable operation of the power grid, all put forward higher requirements for multiple data sources and data collection efficiency. Second, in terms of data storage, the types of data are richer. Different data sources such as power grid operation data and power market data, and data in different formats need to fully participate in the big data mining and analysis process to provide full data support for more accurate and intelligent decision-making analysis. Third, in terms of data analysis, through in-depth analysis of power system data, power grid models and prediction models can be established to predict power demand, load fluctuations, etc., and provide decision support for power grid regulators.

[0028] At present, the efficient collection, storage, analysis and calculation of the above-mentioned massive data are still the focus and frontier of research. There is a lack of mature technology to achieve a balance among the above three. Therefore, it is necessary to carry out relevant technical research on massive control and operation data.

[0029] In traditional methods, the storage, processing, and analysis of massive data are mainly based on the big data warehouse of the Hadoop platform, relying on components such as Hadoop, Spark, Hive, and HBase. Among them, HBase and Hive are mainly used for large-scale data storage, and Spark and Hadoop are mainly used for large-scale data processing and analysis. The main technical solutions used in building a big data platform are as follows: First, HBase is used as the main storage for large-scale structured data. A reasonable Rowkey is designed according to actual business needs and high-performance random read and write capabilities are provided. Second, Hive is used as the query and analysis engine for large-scale structured data. A SQL-like query language, HiveQL, is provided for data analysis and query on the Hadoop cluster. Hive provides parallel data processing capabilities that are compatible with traditional SQL. Finally, Spark's memory-based distributed streaming computing framework is used during analysis and calculation to support a variety of data processing modes, including batch processing, stream processing, and machine learning. These technologies can support the needs of large-scale data storage, processing, and analysis, and have high reliability, scalability, and flexibility. However, this solution mainly relies on HBase's structured data capabilities to store data from diverse sources, and there are two bottlenecks: (1) As a NoSQL database, HBase has high requirements for data consistency and quality. During data import, cleaning, and processing, additional work is required to ensure data quality and accuracy, especially the need to perform a good structural design before data is stored in the database, which cannot adapt well to the rapid access of increasingly complex and diverse data. (2) Relying on Hive as the data query engine of HBase to search and query massive data, although HBase has excellent random read and write performance, the query performance is greatly affected by the structural design when scanning large amounts of data. In particular, when complex queries and aggregation operations are required, the performance may be affected, which seriously affects the data processing efficiency of streaming data processing frameworks such as Spark.

[0030] Based on the above situation, the present application proposes a data layered fusion storage system, storage method and related devices, and the present application is described in detail below in conjunction with embodiments and drawings.

[0031] As a first embodiment of a data hierarchical fusion storage system of the present application, the system may include the following hierarchical settings from low to high: The original data layer is used to access native data and perform hierarchical storage; the native data includes structured data, semi-structured data, and unstructured data. The original data layer is the cornerstone of the entire data hierarchical fusion storage system. It is responsible for directly accessing and storing native data from different sources. These data may be structured (such as table data in the database), semi-structured (such as data in JSON, XML, etc.) or unstructured (such as text, images, videos, etc.). According to factors such as data type, size, and access frequency, data is divided into different layers and levels for storage to improve storage efficiency and access speed.

[0032] It should be noted that structured data refers to data with a clear pattern or structure, usually stored in a relational database in the form of a table, where each row represents a record and each column represents a field. Semi-structured data refers to data that has a certain structure but is not as strict and regular as structured data. It usually contains tags or key-value pairs, making the data self-descriptive. Unstructured data refers to data without a fixed structure or pattern, usually complex data objects stored in binary form.

[0033] The data initial processing layer is used to synchronize the native data stored in the original data layer, and initially process the multi-source data in the structured data in the original data layer into mature data and store it. The data initial processing layer is located above the original data layer, and is mainly responsible for the initial processing of the structured data in the original data layer, converting it into mature data that is easier to analyze and use. The structured data from different sources is cleaned, converted and integrated to eliminate data redundancy and inconsistency. The integrated data is initially analyzed and calculated, such as data screening, format conversion, etc., to generate mature data.

[0034] The data deep processing layer is used to synchronously store the data in the data initial processing layer, and summarize and aggregate the cooked data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data, and unstructured data. After obtaining high-density data, it is stored through multiple types of storage components. The data deep processing layer further summarizes and aggregates the cooked data in the data initial processing layer and other types of raw data to generate a higher density and more valuable data set. Complex data processing algorithms and models can be used to comprehensively analyze cooked data, non-multi-source structured data, semi-structured data, and unstructured data to extract valuable information and patterns. Then, according to the characteristics of the data and usage requirements, select the appropriate data storage component for storage.

[0035] The data application layer is used to synchronize high-density data in the data deep processing layer. It is stored through multiple types of storage components and connected to external business application interfaces. The data application layer is the top layer of the data layered fusion storage system. It is directly connected to the external business application interface to provide high-quality data support for business applications. The data application layer can maintain real-time synchronization with the data deep processing layer to ensure that business applications can obtain the latest high-density data. It can also select appropriate data storage and access methods according to the needs of business applications, and provide flexible data interfaces and services. In actual applications, it can be integrated with business applications through APIs, SDKs, etc. to achieve seamless docking and sharing of data. It provides data analysis tools and visualization interfaces to help business users better understand and use data and improve decision-making efficiency and quality.

[0036] This application is aimed at the control and operation data that is mainly structured data, but also has semi-structured and unstructured files. It analyzes the structure, frequency, and real-time characteristics of the data, proposes a layered and hierarchical hybrid storage model, and uses multiple types of data underlying storage to support efficient data access, processing, analysis, and query.

[0037] like Figure 1 As shown in FIG. 1 , a schematic diagram of the data layered fusion storage system of the present application is shown. Figure 2 The figure is a schematic diagram of the data layered fusion storage method of the present application. The embodiment is further described in detail as follows.

[0038] 1. Unified access to native data.

[0039] The original data is uniformly connected to the original data layer of hierarchical storage. The original data includes the massive data generated by each system in the process of power grid regulation and operation. After the massive data is stored, it becomes the historical operation data of power grid regulation and control. The original data is mainly highly structured data, and also contains some semi-structured data and some unstructured data.

[0040] The original data layer relies on the big data storage components Hudi and HDFS, where Hudi supports loose structured table storage. After the structured data in the massive data is generated, it is received and parsed and stored in the form of online message packets or binary files. The structured data in the original data layer is stored in tables according to the organizational structure, power equipment container, abstract container, primary energy object, power generation equipment, transmission equipment, substation equipment and other power grid objects, and is stored in partitions after evaluating the data business time and data volume. Semi-structured data and unstructured data are stored in HDFS in the form of files. Semi-structured data is stored in the organization of business categories, business subcategories, and business time. The above-mentioned business categories, business subcategories, business time and other meta-information are uniformly received and written into the Hudi structured table when the data is accessed.

[0041] 2. Integrated processing of data stream and batch.

[0042] After completing the access and storage of the original data, it is necessary to perform different degrees of processing tasks on different types of data according to the data source and business type. After the data is processed into mature data, high-quality data is provided for the upper-level business.

[0043] Stream-batch integrated processing integrates the real-time data stream of raw data and the batch data processing that has been stored historically. It is implemented using the in-memory computing component Flink. It processes real-time or near real-time data based on the real-time data processing, window computing, and streaming aggregation functions of the Flink component. The real-time data stream comes from the real-time data access of the raw data layer, and the batch data comes from the data or offline files that have been accessed by the raw data layer.

[0044] It should be noted that Flink can directly process data streams that are accessed in real time from the original data layer. These data streams may come from various data sources, such as IoT devices, website logs, social media streams, etc. Their characteristics are that data is continuously generated and needs to be processed in real time to obtain immediate insights. Flink can process these real-time data with low latency through its efficient stream processing engine, support complex event processing, pattern matching and other operations, so that enterprises can quickly respond to market changes and user behaviors. Window calculation allows aggregate analysis of data over a period of time. Flink provides a variety of window types, including rolling windows, sliding windows, session windows, etc., which can be defined based on time, quantity or other conditions. Through window calculation, real-time data streams can be aggregated and counted, such as calculating the number of user visits per minute, the total transaction amount per hour, etc., so as to achieve real-time monitoring and analysis of business activities. Streaming aggregation allows data to be aggregated during the data flow, such as summation, average calculation, maximum / minimum value search, etc. Flink's streaming aggregation function can handle complex aggregation logic and provide accurate aggregation results while maintaining low latency.

[0045] 3. Process mature data.

[0046] Structured data may include multi-source data and ordinary data. For multi-source data, multi-source data cleaning, data detection, and data aggregation tasks are configured and executed in sequence to process the multi-source data into high-quality mature data.

[0047] For common types of structured operating data, configure and execute data cleaning tasks and data synchronization tasks in sequence to process the structured operating data into high-quality mature data.

[0048] For common types of semi-structured data and unstructured data, data labeling tasks can be configured and executed in sequence, and semi-structured data can be associated with structured data operation data according to the business labels configured by business applications.

[0049] In the process of processing the above data into cooked data, you can configure and execute batch and stream data synchronization tasks according to business needs, and synchronize the data to different storage layers of the hierarchical storage model.

[0050] 4. Data deep processing layer storage.

[0051] The data deep processing layer stores high-density data after data is summarized and aggregated according to business needs. The data deep processing layer has a close relationship with the upper-layer application and involves high real-time interactive queries. It is designed to support storage based on basic storage components such as ClickHouse, Hbase, and MPP.

[0052] 5. Data application layer storage.

[0053] The data application layer is the data layer directly facing business applications. Its data structure and data access conform to the design pattern of traditional Web applications, and its changes are also quite diverse. It is designed based on basic storage components such as ClickHouse and MPP to support storage, and provide high-performance data writing, multi-table association, and wide table query performance.

[0054] Data fusion storage stores processed and processed data in a hierarchical data storage architecture for subsequent data analysis and application. This process aims to provide a unified data storage solution to facilitate data management and query. In addition, data fusion storage is mainly implemented through data synchronization tasks. According to the hierarchical and hierarchical storage fusion storage design, the data initial processing layer, data deep processing layer and data application layer are set above the original data layer. Among them, the data initial processing layer is the first-level mature data after the data is initially processed such as multi-source data fusion processing and dirty data cleaning. Its structure and function are not much different from the original data layer and are still based on Hudi storage. According to the classification of power grid control and operation data, for data accessed by multiple data sources, the data synchronization task will synchronize the multi-source data cleaning and detection results to the data initial processing layer, and design the corresponding structured table storage in Hudi; synchronize the results of data aggregation to the data deep processing layer, which is stored by ClickHouse, and can also be synchronized directly from the data initial processing layer to the application data layer according to demand. For ordinary structured data, the data synchronization task will directly synchronize the results of data cleaning and data processing to the data deep processing layer, or directly synchronize to the data application layer according to actual needs, and store it by ClickHouse or MPP. According to the idea of ​​data fusion storage mentioned above, after configuring and executing the data synchronization task, the massive amount of operating data will be able to provide optimized query and analysis capabilities to the outside world through different basic storage components.

[0055] like Figure 3 FIG. 1 is a schematic diagram of a hierarchical fusion storage method for data in the present application, which may include: S101, access native data and perform hierarchical storage; the native data includes structured data, semi-structured data and unstructured data.

[0056] S102, synchronously storing native data, preliminarily processing multi-source data in structured data into mature data and storing it.

[0057] S103, synchronize data, summarize and aggregate the cooked data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data and unstructured data, and store the high-density data through multi-type storage components.

[0058] S104, synchronize high-density data, store it through multiple types of storage components, and connect it to an external business application interface.

[0059] In some embodiments of the data layered fusion storage method of the present application, the accessing of native data and performing layered and hierarchical storage may include: using storage component Hudi and storage component HDFS to perform layered and hierarchical storage on the accessed native data.

[0060] In some embodiments of the data layered fusion storage method of the present application, the layered and graded storage may include: the structured data sub-tables are accessed and stored in the storage component Hudi, and the semi-structured data and unstructured data are accessed and stored in the storage component HDFS in the form of files.

[0061] In some embodiments of the data hierarchical fusion storage method of the present application, the structured data is accessed and stored in the storage component Hudi, which may include: after the structured data is generated, it is stored in tables in the form of online message packets or binary files in the manner of different power grid objects, and is partitioned and stored after evaluating the data service time and data volume. The semi-structured data is stored in the storage component HDFS in the form of files, which may include: the semi-structured data is organized and stored according to the major business categories, minor business categories, and business time.

[0062] In some embodiments of the data layered fusion storage method of the present application, the preliminary processing of the multi-source data in the structured data in the original data layer into cooked data may include: performing data cleaning, data detection and data aggregation on the multi-source data.

[0063] In some embodiments of the data layered fusion storage method of the present application, it may also include: integrating real-time data streams and historically stored batch data to obtain integrated data; using the memory computing component Flink to implement data stream batch integrated processing to complete synchronization.

[0064] It should be noted that in the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of each module is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another device, or some features can be ignored or not executed. The module described as a separate component may or may not be physically separated. The component displayed as a module may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0065] In addition, each module in each embodiment of the present invention may be integrated into a processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0066] An embodiment of the present application also provides an electronic device, which may include one or more processors, a memory, and a communication interface.

[0067] The memory, the communication interface and the processor are coupled, for example, the memory, the communication interface and the processor may be coupled together via a bus.

[0068] The communication interface is used for data transmission with other devices. The memory stores computer program code. The computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the steps of the data hierarchical fusion storage method in the embodiment of the present application.

[0069] Wherein, the processor can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the present disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, and the like. The processor can be used to support electronic devices to execute the method steps provided in the above embodiments.

[0070] The bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The above bus may be divided into an address bus, a data bus, a control bus, etc.

[0071] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the data hierarchical fusion storage method described in any of the above embodiments are implemented.

[0072] The computer-readable storage medium involved in the present application includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field.

[0073] This application can be applied to the massive historical data generated by the operation of power grid regulation. With this data as the core, the structural characteristics of multi-source heterogeneous data and the real-time processing requirements of business are analyzed, and a hierarchical storage model for access, processing, analysis, and query of massive power grid regulation and operation data is designed. A hierarchical hybrid storage architecture is introduced to solve the problem of HBase's single storage capacity. The data flow process is optimized by relying on the computing method that integrates distributed streaming computing and batch computing, and near-real-time access and distributed parallel processing of massive power grid regulation data are achieved. The problems of inconvenient storage and untimely calculation of multi-source heterogeneous data are solved, and the development of offline analysis and calculation, online analysis and calculation and other businesses in power grid operation are supported.

[0074] It should also be noted that the present application can not only be used for the massive historical data generated by the power grid regulation and operation, but can also be used for data storage in other scenarios, and provides a storage method.

[0075] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data tiered fusion storage system, characterized in that: Includes the following levels from low to high: The original data layer is used to access the native data and store it in layers and grades; the native data includes structured data, semi-structured data and unstructured data; The data initial processing layer is used to synchronize the native data stored in the original data layer, and to initially process the multi-source data in the structured data in the original data layer into cooked data and store it; The data deep processing layer is used to synchronously store the data in the data initial processing layer, and summarize and aggregate the mature data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data, and unstructured data. After obtaining high-density data, it is stored through multi-type storage components; The data application layer is used to synchronize high-density data in the data deep processing layer, store it through multiple types of storage components, and connect to the external business application interface.

2. The data layered fusion storage system according to claim 1, characterized in that: The original data layer uses the storage component Hudi and the storage component HDFS to perform hierarchical storage on the accessed native data.

3. The data layered fusion storage system according to claim 2, characterized in that: The layered and graded storage includes: the structured data sub-tables are accessed and stored in the storage component Hudi, and the semi-structured data and unstructured data are accessed and stored in the storage component HDFS in the form of files.

4. The data layered fusion storage system according to claim 3, characterized in that: The structured data sub-tables are accessed and stored in the storage component Hudi, including: After the structured data is generated, it is stored in tables according to different power grid objects in the form of online message packets or binary files, and is partitioned and stored after evaluation based on data service time and data volume.

5. The data layered fusion storage system according to claim 4, characterized in that: The semi-structured data is stored in the storage component HDFS in the form of files, including: Semi-structured data is organized and stored according to business categories, business subcategories, and business time.

6. The data layered fusion storage system according to claim 1, characterized in that: The preliminarily processing the multi-source data in the structured data in the original data layer into cooked data includes: Perform data cleansing, data instrumentation, and data aggregation on multi-source data.

7. The data layered fusion storage system according to claim 1, characterized in that: The storage in the initial data processing layer uses the storage component Hudi; The storage in the data deep processing layer adopts at least one of the storage component HBase, the storage component ClickHouse and the storage component MPP; The storage in the data application layer adopts the storage component ClickHouse and / or the storage component MPP.

8. The data layered fusion storage system according to claim 1, characterized in that: Also includes: Integrate real-time data streams and historically stored batch data to obtain integrated data; The memory computing component Flink is used to realize integrated data stream and batch processing and complete synchronization.

9. The data layered fusion storage system according to claim 8, characterized in that: The use of the memory computing component Flink to achieve integrated data stream and batch processing includes: The real-time data processing, window computing, and streaming aggregation functions of the in-memory computing component Flink process and integrate real-time data streams.

10. A data layered fusion storage method, characterized in that: include: Access native data and perform hierarchical storage; the native data includes structured data, semi-structured data and unstructured data; Synchronously store native data, and process multi-source data in structured data into mature data and store it; Synchronize data, summarize and aggregate the mature data after the initial processing of multi-source data, non-multi-source data in structured data, semi-structured data, and unstructured data, and store them through multiple types of storage components after obtaining high-density data; Synchronize high-density data, store it through multiple types of storage components, and connect to external business application interfaces.

11. The data layered fusion storage method according to claim 1, characterized in that: The accessing of native data and performing hierarchical storage includes: The storage components Hudi and HDFS are used to store the accessed native data in layers and grades.

12. The data layered fusion storage method according to claim 11, characterized in that: The layered and hierarchical storage includes: The structured data sub-tables are accessed and stored in the storage component Hudi, and the semi-structured data and unstructured data are accessed and stored in the storage component HDFS in the form of files.

13. The data layered fusion storage method according to claim 12, characterized in that: The structured data sub-tables are accessed and stored in the storage component Hudi, including: After the structured data is generated, it is stored in tables according to different power grid objects in the form of online message packets or binary files, and is stored in partitions after evaluation based on data service time and data volume; The semi-structured data is stored in the storage component HDFS in the form of files, including: Semi-structured data is organized and stored according to business categories, business subcategories, and business time.

14. The data layered fusion storage method according to claim 10, characterized in that: The preliminarily processing the multi-source data in the structured data in the original data layer into cooked data includes: Perform data cleansing, data instrumentation, and data aggregation on multi-source data.

15. The data layered fusion storage method according to claim 10, characterized in that: The synchronization includes: Integrate real-time data streams and historically stored batch data to obtain integrated data; The memory computing component Flink is used to realize integrated data stream and batch processing and complete synchronization.

16. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the steps of the data hierarchical fusion storage method as described in any one of claims 10-15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data hierarchical fusion storage method according to any one of claims 10 to 15 are implemented.

Images