Data storage method and device, equipment and storage medium

Abstract

The invention discloses a data storage method, device and equipment and a storage medium, and relates to the technical field of big data, the method is applied to a real-time data warehouse comprising a source pasting layer, a detail layer and a service layer, and the method comprises the steps of obtaining original data from the source pasting layer based on the detail layer; normalizing the original data based on the detail layer to obtain detail data corresponding to the original data; and writing the detail data into a column data management module of the service layer based on the detail layer so as to store the detail data based on the column data management module, and providing a data query interface based on the column data management module. According to the technical scheme, the updating efficiency of the original data and the flexibility and the real-time performance of data storage are improved through the real-time data warehouse designed in a layered mode, the multi-dimensional analysis response efficiency of the original data is improved through standardization processing of the original data, and in addition, the real-time performance of the data storage system is improved. And the column data management module based on the service layer provides a data query interface for the detail data, so that the practicability of data storage is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular, to a data storage method, apparatus, device and storage medium. Background Art

[0002] In order to cope with the real-time requirements brought by enterprise development, data warehouses must be able to achieve continuous data integration in order to process the latest business data. For enterprises that need decision support when transactions occur, real-time data warehouses have great potential.

[0003] In the existing technology, real-time data warehouses are usually built based on Lambda architecture, and the real-time data warehouses built have two links: offline and real-time. The real-time part is usually implemented by Flink+Kafka, and real-time incremental consumption is performed in the form of message queues. The computing engine of the offline part is usually implemented by Hive or Spark, and the historical stock data is analyzed using T+1 periodic scheduling, and the output is produced every morning, and the result data covering the previous day is updated.

[0004] The disadvantage of Lambda architecture is that the same requirement requires the development of two sets of identical codes, which increases resource usage and high operation and maintenance costs. Summary of the invention

[0005] The present invention provides a data storage method, device, equipment and storage medium to reduce the maintenance cost of a real-time data warehouse and improve flexibility and real-time performance.

[0006] In a first aspect, an embodiment of the present invention provides a data storage method, which is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer. The method includes:

[0007] Acquire raw data from the source layer based on the detail layer;

[0008] Normalizing the original data based on the detail layer to obtain detail data corresponding to the original data;

[0009] The detail data is written into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module, and provide a data query interface based on the column data management module.

[0010] The technical solution of an embodiment of the present invention provides a data storage method, which is applied to a real-time data warehouse including a source layer, a detail layer and a service layer. The method includes: obtaining original data from the source layer based on the detail layer; normalizing the original data based on the detail layer to obtain detailed data corresponding to the original data; writing the detailed data into a columnar data management module of the service layer based on the detail layer, so as to store the detailed data based on the columnar data management module, and providing a data query interface based on the columnar data management module. The above technical scheme can firstly obtain the original data written into the source layer based on the detail layer, and realize the data flow of the original data between the source layer and the detail layer. Secondly, the original data entering the detail layer can be processed based on the detail layer to obtain the detailed data corresponding to the original data, and realize the normalization of the original data. Then, the detailed data can be written into the columnar data management module of the service layer based on the detail layer, so that the service layer can realize the real-time storage of the detailed data based on the columnar data management module. Through the hierarchically designed real-time data warehouse, the updating efficiency of the original data and the flexibility and real-time performance of the data storage are improved. Through the normalization of the original data, the multi-dimensional analysis response efficiency of the original data is improved. In addition, the columnar data management module based on the service layer provides a data query interface for the detailed data, thereby improving the practicality of data storage.

[0011] Furthermore, the source layer includes a stream processing module. Accordingly, before obtaining the original data from the source layer based on the detail layer, it also includes:

[0012] Based on the source layer, the original data written into the real-time data warehouse is written into a corresponding topic in the streaming processing module.

[0013] Further, obtaining original data from the source layer based on the detail layer includes:

[0014] Based on the detail layer, the original data is subscribed to and obtained from the corresponding topic in the streaming processing module of the source layer.

[0015] Furthermore, the original data is normalized based on the detail layer to obtain the detail data corresponding to the original data, including:

[0016] Extracting, converting and loading the original data to obtain intermediate data;

[0017] After determining the dimensional data corresponding to the intermediate data, the intermediate data is associated with the corresponding dimensional data to obtain the detailed data corresponding to the original data.

[0018] Furthermore, the column data management module that writes the detail data into the service layer based on the detail layer includes:

[0019] Determine the local table of the detailed data in the column data management module according to the data identifier corresponding to the detailed data;

[0020] The detailed data is written into the corresponding local table.

[0021] Furthermore, when the detail data is written into the column data management module of the service layer based on the detail layer, it also includes:

[0022] The detail data is written into the first storage module of the detail layer.

[0023] In a second aspect, an embodiment of the present invention further provides a data storage device, which is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer. The device includes:

[0024] An acquisition module, used for acquiring original data from the source layer based on the detail layer;

[0025] A processing module, used for normalizing the original data based on the detail layer to obtain the detail data corresponding to the original data;

[0026] A storage module is used to write the detail data into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module and provide a data query interface based on the column data management module.

[0027] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0028] at least one processor; and a memory communicatively coupled to the at least one processor;

[0029] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data storage method as described in any one of the first aspects.

[0030] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data storage method as described in any one of the first aspects.

[0031] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the data storage method provided in the first aspect.

[0032] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data storage device, or may be packaged separately from the processor of the data storage device, which is not limited in this application.

[0033] The description of the second, third, fourth and fifth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0034] In this application, the name of the above data storage device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0035] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a data storage method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a real-time data warehouse in a data storage method provided by an embodiment of the present invention;

[0039] Figure 3 A flowchart of another data storage method provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the structure of a data storage device provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0043] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0044] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0045] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0046] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0049] For enterprises with high real-time requirements, traditional data warehouses cannot support any dynamics in structure and content when used for online analytical processing (OLAP). Therefore, real-time data warehouses are needed for data storage.

[0050] In the prior art, a real-time data warehouse built based on the Lambda architecture requires the development of two identical sets of codes, which increases resource usage and has high operation and maintenance costs. A real-time data warehouse can also be built based on the Kappa architecture. The real-time data warehouse built can remove the offline part and achieve the purpose of re-consuming and processing historical data by passing any desired offset. However, it is required to save the stock data in the message queue, and the main business logic is in the computing layer, which consumes more memory resources. A real-time data warehouse can also be built based on the real-time OLAP variant architecture. The real-time OLAP variant architecture is a further evolution of the Kappa architecture. The OLAP engine undertakes the aggregation analysis calculation, which reduces the aggregation processing pressure of the real-time computing part. However, the pressure of the computing part is transferred to the query layer, which has high requirements on the throughput and real-time ingestion performance of the query engine.

[0051] Therefore, the present application proposes a data storage method, which implements data storage based on a real-time data warehouse including a source layer, a detail layer and a service layer, and improves the multidimensional analysis response time and transaction update efficiency through a layered design.

[0052] The data storage method proposed in the present application will be described in detail below with reference to diagrams and embodiments.

[0053] Figure 1 A flowchart of a data storage method provided by an embodiment of the present invention is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer. It can be used to reduce the maintenance cost of the real-time data warehouse while improving flexibility and real-time performance. The method can be executed by a data storage device, such as Figure 1 As shown, the specific steps include:

[0054] Step 110: Obtain original data from the source layer based on the detail layer.

[0055] Figure 2 A schematic diagram of a real-time data warehouse in a data storage method provided by an embodiment of the present invention, such as Figure 2 As shown in the figure, the real-time data warehouse consists of the source layer, detail layer and service layer. The source layer is the ODS layer, the detail layer is the DWD layer, and the service layer is the DWS layer. The layered design of the real-time data warehouse still draws on the traditional data warehouse, taking flatness as the best strategy to minimize the delay in data transmission.

[0056] The original data may be the real-time business data of the current enterprise.

[0057] Specifically, after the raw data enters the source layer, the detail layer can obtain the raw data from the source layer. In practical applications, the raw data is stored in the source layer in the form of a topic, which is a mechanism used to organize and distinguish different business message flows. After the raw data is sent to a specific topic in the source layer in the form of a message, the detail layer can subscribe to the corresponding topic and receive the message for processing, thereby obtaining the raw data in the source layer.

[0058] In the embodiment of the present invention, the detail layer obtains the original data from the source layer, that is, realizes the data flow of the original data between the source layer and the detail layer.

[0059] Step 120: normalize the original data based on the detail layer to obtain detail data corresponding to the original data.

[0060] Specifically, after the detail layer obtains the original data from the source layer, it can normalize the original data. Specifically, the original data can be formatted first, and then the formatted original data can be dimensionally processed, that is, real-time dimension association operations can be performed. Usually, real-time business data (usually continuously flowing stream data, such as real-time order data, real-time monitoring data, etc.) can be associated with corresponding dimensional data to obtain detailed data corresponding to the original data.

[0061] In the embodiment of the present invention, the original data entering the detail layer is processed based on the detail layer to obtain the detail data corresponding to the original data, thereby achieving normalized processing of the original data.

[0062] Step 130: Write the detail data into the column-based data management module of the service layer based on the detail layer, so as to store the detail data based on the column-based data management module, and provide a data query interface based on the column-based data management module.

[0063] Specifically, after the detail layer normalizes the original data to obtain the detail data, the detail data can be written to the service layer, specifically, the local table of the columnar data management module in the service layer. Of course, the frequency of data writing can be controlled according to the two conditions of data batch size and batch interval to balance the pressure and real-time performance of the real-time database.

[0064] The columnar data management module can be ClickHouse. Clickhouse includes distributed tables and local tables. Distributed tables are logical tables, which can be understood as views in the database and are used for data query. The distributed table engine can route query requests to local tables for query, and then summarize and return them. Local tables are used to actually store data.

[0065] Based on this component, ClickHouse can write detailed data into local tables instead of distributed tables to avoid excessive amplitude of temporary data, and data storage is more efficient and flexible.

[0066] It should be noted that the column data management module can provide a data query interface to facilitate direct data query on detailed data at the service layer.

[0067] In an embodiment of the present invention, the detail data is written into the column-based data management module of the service layer based on the detail layer, so that the service layer can realize real-time storage of the detail data based on the column-based data management module, thereby improving the flexibility and real-time performance of data storage. In addition, the column-based data management module based on the service layer provides a data query interface for the detail data, thereby improving the practicality of data storage.

[0068] The data storage method provided by an embodiment of the present invention is applied to a real-time data warehouse including a source layer, a detail layer and a service layer. The method includes: obtaining original data from the source layer based on the detail layer; normalizing the original data based on the detail layer to obtain detailed data corresponding to the original data; writing the detailed data into a columnar data management module of the service layer based on the detail layer, so as to store the detailed data based on the columnar data management module, and providing a data query interface based on the columnar data management module. The above technical scheme can firstly obtain the original data written into the source layer based on the detail layer, and realize the data flow of the original data between the source layer and the detail layer. Secondly, the original data entering the detail layer can be processed based on the detail layer to obtain the detailed data corresponding to the original data, and realize the normalization of the original data. Then, the detailed data can be written into the columnar data management module of the service layer based on the detail layer, so that the service layer can realize the real-time storage of the detailed data based on the columnar data management module. Through the hierarchically designed real-time data warehouse, the updating efficiency of the original data and the flexibility and real-time performance of the data storage are improved. Through the normalization of the original data, the multi-dimensional analysis response efficiency of the original data is improved. In addition, the columnar data management module based on the service layer provides a data query interface for the detailed data, thereby improving the practicality of data storage.

[0069] Figure 3 This is a flowchart of another data storage method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. Figure 3 As shown, in this embodiment, the method may further include:

[0070] Step 310: Write the original data written into the real-time data warehouse into a corresponding topic in the streaming processing module based on the source layer.

[0071] like Figure 2 As shown, the source layer includes a stream processing module, and the stream processing module can be Kafka.

[0072] Specifically, the original data usually enters the real-time data warehouse through Flume, that is, it can enter the source layer of the real-time data warehouse. The source layer can write the original data entering the real-time data warehouse into the corresponding Topic in the streaming processing module.

[0073] In the embodiment of the present invention, the original data written into the real-time data warehouse is written into the corresponding Topic in the source layer based on the source layer.

[0074] Step 320: Obtain original data from the source layer based on the detail layer.

[0075] In one implementation, step 320 may specifically include:

[0076] Based on the detail layer, the original data is subscribed to and obtained from the corresponding topic in the streaming processing module of the source layer.

[0077] Specifically, the original data is written to the corresponding Topic in Kafka, and the detail layer can obtain the original data from the detail layer by subscribing to the corresponding Topic in Kafka.

[0078] In the embodiment of the present invention, the detail layer obtains the original data from the source layer by subscribing to the corresponding Topic in Kafka, that is, realizes the data flow of the original data between the source layer and the detail layer.

[0079] Step 330: normalize the original data based on the detail layer to obtain detail data corresponding to the original data.

[0080] like Figure 2 As shown, the real-time data warehouse also includes a dimension layer, namely, a DIM layer. The dimension layer is composed of a MySQL image library and is used to store all dimensional data.

[0081] In one implementation, step 330 may specifically include:

[0082] The original data is extracted, converted and loaded to obtain intermediate data; after determining the dimensional data corresponding to the intermediate data, the intermediate data is associated with the corresponding dimensional data to obtain the detailed data corresponding to the original data.

[0083] Specifically, after the detail layer obtains the original data from the source layer, it can normalize the original data. Specifically, first, the original data can be extracted, transformed, and loaded (Extract Transform Load, ETL) processed. Secondly, the original data processed by ETL can be dimensionally processed, that is, real-time dimension association operations can be performed. Usually, the dimension data corresponding to the real-time business data (usually the continuously flowing stream data, such as real-time order data, real-time monitoring data, etc.) can be obtained from the dimension layer, and then the real-time business data can be associated with the corresponding dimension data to obtain the detailed data corresponding to the original data.

[0084] When the real-time business data is real-time order data (including user ID and product ID, etc.), first, the user dimension data (user dimension table, including user ID and user details, etc.) and product dimension data (product dimension table, including product ID and product details, etc.) corresponding to the real-time order data can be obtained from the dimension layer. Secondly, the real-time order data can be associated with the user dimension data and the product dimension data to obtain complete order data including user details and product details, that is, the order detail data corresponding to the real-time order data can be obtained, which is convenient for real-time analysis of the real-time order data, such as real-time viewing of the order status of users in different regions and age groups.

[0085] In an embodiment of the present invention, the original data entering the detail layer is processed based on the detail layer to obtain the detail data corresponding to the original data, thereby realizing normalized processing of the original data. Moreover, the detail data including dimensional data can present and analyze the business situation more comprehensively and accurately in a real-time analysis scenario.

[0086] Step 340: Write the detail data into the column-based data management module of the service layer based on the detail layer, so as to store the detail data based on the column-based data management module, and provide a data query interface based on the column-based data management module.

[0087] In one implementation, the column data management module that writes the detail data into the service layer based on the detail layer includes:

[0088] Determine a local table of the detailed data in the column-based data management module according to a data identifier corresponding to the detailed data; and write the detailed data into the corresponding local table.

[0089] like Figure 2As shown, the detail layer includes a first storage module HDFS (Hive) and a second storage module Redis, and the service layer includes a third storage module Redis.

[0090] Among them, the columnar data management module is ClickHouse, and ClickHouse includes the BalancedClickhouseDataSource component.

[0091] Specifically, after the detail layer normalizes the original data to obtain the detail data, the detail data can be written to the service layer. Specifically, the detail data can be first written to the second storage module Redis of the detail layer, and then the detail data can be written to the column data management module of the service layer through the second storage module Redis of the detail layer. When writing the detail data to the column data management module of the service layer, the local table of the detail data in the column data management module can be first determined according to the data identifier of the detail data. The column data management module defines three data writing strategies: round-robin training, random partitioning, and field-hash (based on field consistent hashing, which can ensure that the same partition field is hashed to the same local table). The local table of the detail data in the column data management module can be determined based on any of the above three data writing strategies, and then the detail data can be written to the corresponding local table based on the BalancedClickhouseDataSource component.

[0092] In addition, the columnar data management module can periodically summarize detailed data through mechanisms such as ClickHouse materialized views to form corresponding reports and page heat maps.

[0093] In particular, detailed data is also open for query at this layer. Specifically, a data query interface can be provided based on the columnar data management module, so as to facilitate flexible ad-hoc queries such as funnel, retention, user path, etc. on detailed data directly at the service layer.

[0094] The BalancedClickhouseDataSource component ensures the load balancing of each ClickHouse in the service layer through random routing. Specifically, it can detect ClickHouse through periodic ping, block the currently inaccessible ClickHouse, and do not set up failover - that is, once you try to write to the inaccessible ClickHouse, it will cause data loss. Therefore, a retry mechanism is designed for ClickHouse, and the number of retries and intervals can be configured. When the retry opportunities are exhausted and the writing is still unsuccessful, the batch of data will be transferred to the configured path, and an alarm will be issued to require timely inspection and backfilling.

[0095] In actual applications, first, the storage engine related information of the local table is determined from the system table according to the detailed data. Secondly, the storage engine related information is parsed to extract the cluster name and the local table name. Then, based on the parsed cluster name, the corresponding shard node information is retrieved from the system table. The target shard is determined according to the shard node information and the host address corresponding to each shard. Then, based on the BalancedClickhouseDataSource component, the detailed data is written into the local table corresponding to the local table name in the target shard.

[0096] In another implementation manner, when the detail data is written into the column data management module of the service layer based on the detail layer, it further includes:

[0097] The detail data is written into the first storage module of the detail layer.

[0098] While the detail layer writes the detail data into the column data management module of the service layer through the second storage module Redis of the detail layer, the detail layer can also write the detail data into the first storage module HDFS (Hive) of the detail layer to realize the backup of the detail data.

[0099] In an embodiment of the present invention, the detail data is written into the column-based data management module of the service layer based on the detail layer, so that the service layer can realize real-time storage of the detail data based on the column-based data management module, thereby improving the flexibility and real-time performance of data storage. In addition, the column-based data management module based on the service layer provides a data query interface for the detail data, thereby improving the practicality of data storage.

[0100] The data storage method provided by an embodiment of the present invention includes: writing the original data written into the real-time data warehouse into the corresponding topic in the streaming processing module based on the source layer; obtaining the original data from the source layer based on the detail layer; normalizing the original data based on the detail layer to obtain the detailed data corresponding to the original data; writing the detailed data into the columnar data management module of the service layer based on the detail layer, so as to store the detailed data based on the columnar data management module, and providing a data query interface based on the columnar data management module. The above technical scheme can firstly write the original data written into the real-time data warehouse into the corresponding Topic in the source layer based on the source layer, and secondly obtain the original data from the source layer based on the corresponding Topic in Kafka subscribed by the detail layer, so as to realize the data flow of the original data between the source layer and the detail layer, and then process the original data entering the detail layer based on the detail layer to obtain the detailed data corresponding to the original data, so as to realize the normalized processing of the original data, and the detailed data containing the dimensional data can present and analyze the business situation more comprehensively and accurately in the real-time analysis scenario, and can also write the detailed data into the columnar data management module of the service layer based on the detailed layer, so that the service layer can realize the real-time storage of the detailed data based on the columnar data management module, and improve the update efficiency of the original data and the flexibility and real-time performance of the data storage through the hierarchically designed real-time data warehouse, and improve the multi-dimensional analysis response efficiency of the original data through the normalized processing of the original data, and provide a data query interface for the detailed data based on the service layer, so as to improve the practicality of data storage.

[0101] In addition, data flow is realized between each layer through Flink, which ensures that the upper and lower limits of data do not affect each other and improves data security. Data can be traced back, which is convenient for abnormal investigation. When the daily activity decreases, it is necessary to trace back and investigate which logical problems of the indicator caliber are caused (reported data difference, data flow Kafka is lost, and users have not reported). The calculation is fast, and all indicator calculations are completed within one cycle. It is necessary to calculate all indicators of all dimensions within five minutes. Support real-time streaming, distributed deployment, and simple operation and maintenance: Support Kafka data real-time streaming.

[0102] The embodiment of the present invention also provides a real-time data warehouse, such as Figure 2As shown, the real-time data warehouse includes a dimension layer, a source layer, a detail layer, and a service layer. The dimension layer is the DIM layer, which is composed of a MySQL image library and is used to store all dimension data. The source layer is the ODS layer, which includes a stream processing module. The stream processing module can be Kafka, and the original data can be directly entered into the corresponding topic Topic in Kafka by Flume. The detail layer includes a stream processing module, a first storage module, and a second storage module. The stream processing module can also be Kafka. The first storage module can be HDFS (Hive), and the second storage module can be Redis. The detail layer can subscribe to the original data from the corresponding topic Topic in Kafka, and process the original data to obtain standardized detail data. The detail data can also be written into the first storage module, and written into the service layer through the second storage module. The detail data written into the first storage module can realize data backup, and the detail data written into the service layer can realize data query and analysis. The service layer includes a third storage module and a columnar data management module. The third storage module can be Redis, and the columnar data management module can be ClickHouse. The third storage module is used to store business indicators such as business volume, concurrency, success rate, etc. The columnar data management module is used to periodically summarize detailed data to form reports and page heat maps, and the columnar data management module provides an interface for data query.

[0103] The real-time data warehouse provided by the embodiment of the present invention can execute the data storage method provided by any embodiment of the present invention, and has the corresponding beneficial effects of executing the data storage method.

[0104] Figure 4 This is a schematic diagram of the structure of a data storage device provided by an embodiment of the present invention. The device is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer, and can be used to reduce the maintenance cost of the real-time data warehouse while improving flexibility and real-time performance. The device can be implemented by software and / or hardware and is generally integrated in an electronic device, such as a computer device.

[0105] like Figure 4 As shown, the device comprises:

[0106] An acquisition module 410, configured to acquire original data from the source layer based on the detail layer;

[0107] A processing module 420, configured to perform normalization processing on the original data based on the detail layer to obtain detail data corresponding to the original data;

[0108] The storage module 430 is used to write the detail data into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module, and provide a data query interface based on the column data management module.

[0109] The data storage device provided in this embodiment is applied to a real-time data warehouse including a source layer, a detail layer and a service layer. The method includes: acquiring original data from the source layer based on the detail layer; normalizing the original data based on the detail layer to obtain detailed data corresponding to the original data; writing the detailed data into the column-based data management module of the service layer based on the detail layer, so as to store the detailed data based on the column-based data management module, and providing a data query interface based on the column-based data management module. The above technical scheme can firstly obtain the original data written into the source layer based on the detail layer, and realize the data flow of the original data between the source layer and the detail layer. Secondly, the original data entering the detail layer can be processed based on the detail layer to obtain the detailed data corresponding to the original data, and realize the normalization of the original data. Then, the detailed data can be written into the columnar data management module of the service layer based on the detail layer, so that the service layer can realize the real-time storage of the detailed data based on the columnar data management module. Through the hierarchically designed real-time data warehouse, the updating efficiency of the original data and the flexibility and real-time performance of the data storage are improved. Through the normalization of the original data, the multi-dimensional analysis response efficiency of the original data is improved. In addition, the columnar data management module based on the service layer provides a data query interface for the detailed data, thereby improving the practicality of data storage.

[0110] On the basis of the above embodiment, the source layer includes a stream processing module, and accordingly, the device further includes:

[0111] A writing module is used to write the original data written into the real-time data warehouse into the corresponding topic in the streaming processing module based on the source layer before obtaining the original data from the source layer based on the detail layer.

[0112] Based on the above embodiment, the acquisition module 410 is specifically used for:

[0113] Based on the detail layer, the original data is subscribed to and obtained from the corresponding topic in the streaming processing module of the source layer.

[0114] Based on the above embodiment, the processing module 420 is specifically used for:

[0115] The original data is extracted, converted and loaded to obtain intermediate data; after determining the dimensional data corresponding to the intermediate data, the intermediate data is associated with the corresponding dimensional data to obtain the detailed data corresponding to the original data.

[0116] Based on the above embodiment, the storage module 430 is specifically used for:

[0117] Determine the local table of the detail data in the column-based data management module according to the data identifier corresponding to the detail data; write the detail data into the corresponding local table to store the detail data based on the column-based data management module, and provide a data query interface based on the column-based data management module; write the detail data into the first storage module of the detail layer.

[0118] The data storage device provided in the embodiment of the present invention can execute the data storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data storage method.

[0119] It is worth noting that in the embodiment of the above-mentioned data storage device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0120] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary electronic device 5 suitable for implementing embodiments of the present invention is shown. Figure 5 The electronic device 5 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0121] like Figure 5 As shown, the electronic device 5 is in the form of a general purpose computing electronic device. The components of the electronic device 5 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0122] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0123] The electronic device 5 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 5, including volatile and non-volatile media, removable and non-removable media.

[0124] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 5 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0125] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0126] The electronic device 5 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 5, and / or any device that enables the electronic device 5 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 5 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 5 As shown, the network adapter 20 communicates with other modules of the electronic device 5 via the bus 18. It should be understood that although Figure 5Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 5, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0127] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, for example, implementing the data storage method provided by the embodiment of the present invention, which is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer. The method includes:

[0128] Acquire raw data from the source layer based on the detail layer;

[0129] Normalizing the original data based on the detail layer to obtain detail data corresponding to the original data;

[0130] The detail data is written into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module, and provide a data query interface based on the column data management module.

[0131] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data storage method provided by any embodiment of the present invention.

[0132] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a data storage method provided by an embodiment of the present invention is implemented, for example, and is applied to a real-time data warehouse including a source layer, a detail layer, and a service layer. The method includes:

[0133] Acquire raw data from the source layer based on the detail layer;

[0134] Normalizing the original data based on the detail layer to obtain detail data corresponding to the original data;

[0135] The detail data is written into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module, and provide a data query interface based on the column data management module.

[0136] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0139] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0141] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0142] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data storage method, characterized in that: Applied to a real-time data warehouse including a source layer, a detail layer and a service layer, the method comprises: Acquire raw data from the source layer based on the detail layer; Normalizing the original data based on the detail layer to obtain detail data corresponding to the original data; The detail data is written into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module, and provide a data query interface based on the column data management module.

2. The data storage method according to claim 1, characterized in that: The source layer includes a stream processing module, and accordingly, before obtaining raw data from the source layer based on the detail layer, it also includes: Based on the source layer, the original data written into the real-time data warehouse is written into a corresponding topic in the streaming processing module.

3. The data storage method according to claim 2, characterized in that: Acquiring original data from the source layer based on the detail layer includes: Based on the detail layer, the original data is subscribed to and obtained from the corresponding topic in the streaming processing module of the source layer.

4. The data storage method according to claim 1, characterized in that: The original data is normalized based on the detail layer to obtain detail data corresponding to the original data, including: Extracting, converting and loading the original data to obtain intermediate data; After determining the dimensional data corresponding to the intermediate data, the intermediate data is associated with the corresponding dimensional data to obtain the detailed data corresponding to the original data.

5. The data storage method according to claim 1, characterized in that: The column data management module that writes the detail data into the service layer based on the detail layer includes: Determine the local table of the detailed data in the column data management module according to the data identifier corresponding to the detailed data; The detailed data is written into the corresponding local table.

6. The data storage method according to claim 1, characterized in that: When the detail data is written into the column data management module of the service layer based on the detail layer, it also includes: The detail data is written into the first storage module of the detail layer.

7. A data storage device, characterized in that: Applicable to a real-time data warehouse including a source layer, a detail layer and a service layer, the device comprises: An acquisition module, used for acquiring original data from the source layer based on the detail layer; A processing module, used for normalizing the original data based on the detail layer to obtain the detail data corresponding to the original data; A storage module is used to write the detail data into the column data management module of the service layer based on the detail layer, so as to store the detail data based on the column data management module and provide a data query interface based on the column data management module.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data storage method as described in any one of claims 1 to 6.

9. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the data storage method according to any one of claims 1 to 6 when executed by a computer processor.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data storage method according to any one of claims 1 to 6.

Images