Data storage method, system, device and non-volatile storage medium

Abstract

The application discloses a data storage method, system and device and a nonvolatile storage medium. The method comprises the following steps: obtaining target incremental data; determining a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data into the distributed data storage table to obtain a preprocessing layer; performing aggregation calculation on the data in the preprocessing data layer to obtain a summary data layer; determining a statistical index corresponding to the data in the summary data layer, and importing the statistical index into a corresponding hot data storage system. The application solves the technical problem of poor data processing timeliness caused by the fact that the related art cannot process incremental data and provide related analysis results to users in time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data storage, in particular to a data storage method, system, device and non-volatile storage medium. BACKGROUND

[0002] The data storage method in the related art cannot process and provide incremental data and analysis results related to the incremental data to users in a timely manner when facing real-time incremental data, resulting in poor timeliness of data processing.

[0003] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0004] The embodiments of the present application provide a data storage method, system, device and non-volatile storage medium to at least solve the technical problem of poor timeliness of data processing caused by the inability to process and provide incremental data and related analysis results to users in a timely manner in the related art.

[0005] According to an aspect of an embodiment of the present application, a data storage method is provided, comprising: obtaining target incremental data; determining a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data in the distributed data storage table to obtain a preprocessing layer; performing aggregation calculation on the data in the preprocessing data layer to obtain a summary data layer; determining a statistical index corresponding to the data in the summary data layer, and importing the statistical index into a corresponding hot data storage system.

[0006] Optionally, the step of obtaining target incremental data comprises: deploying a distributed data processing cluster, wherein the distributed processing cluster includes distributed data processing units corresponding to different data types; determining a data collection tool corresponding to the data type of the target incremental data according to the data type of the target incremental data, and obtaining the target incremental data through the data collection tool; determining a corresponding distributed data processing unit according to the data type of the target incremental data, and storing the target incremental data in the distributed data processing unit.

[0007] Optionally, the data type of the target incremental data includes log data and business library change data, wherein in the case that the data type of the target incremental data is log data, the data collection tool is a Flume cluster, and in the case that the data type of the target incremental data is business library change data, the data collection tool is Flink CDC.

[0008] Optionally, the cluster type of the distributed data processing cluster includes a kafka cluster, and the unit type of the distributed data processing unit includes a kafka Topic.

[0009] Optionally, the step of performing aggregate calculation on the data in the pre-processing data layer to obtain the summary data layer comprises: performing data cleaning on the data stored in the pre-processing data layer to obtain an operation data layer; performing dimension degradation processing on the operation data layer, degrading the dimension table in the operation data layer to the fact table in the operation data layer to obtain a detailed data layer; performing wide table processing on the detailed data layer, and performing aggregate calculation on the data in the detailed data layer to obtain the summary data layer.

[0010] Optionally, after the step of importing the statistical indicators into the corresponding hot data storage system, the data storage method further comprises: uniformly publishing the database tables and the distributed data storage tables of the hot data storage system to the outside; after uniformly publishing the database tables and the distributed data storage tables of the hot data storage system to the outside, receiving data acquisition requests sent by each business subsystem, and providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request.

[0011] Optionally, the step of providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request comprises: determining the data acquisition authority of each business subsystem according to the data acquisition request; determining the obtainable data of each business subsystem according to the data acquisition authority; and providing each business subsystem with the corresponding obtainable data.

[0012] According to another aspect of the embodiments of the present application, a data storage system is further provided, which comprises a unified data collection layer, a unified data storage layer, a unified data calculation layer, and a unified data service layer, wherein the unified data collection layer is configured to collect target incremental data; the unified data storage layer is configured to determine a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and store the target incremental data in the distributed data storage table to obtain a pre-processing data layer; the unified data calculation layer is configured to perform aggregate calculation on the data in the pre-processing data layer to obtain a summary data layer, determine statistical indicators corresponding to the data in the summary data layer, and import the statistical indicators into a corresponding hot data storage system; and the unified data service layer is configured to uniformly publish the database tables and the distributed data storage tables of the hot data storage system to the outside, receive data acquisition requests sent by each business subsystem after uniformly publishing the database tables and the distributed data storage tables of the hot data storage system to the outside, and provide each business subsystem with data corresponding to the data acquisition request according to the data acquisition request.

[0013] According to a further aspect of the embodiments of the present application, a data storage apparatus is also provided, comprising: a collection module configured to acquire target incremental data; a storage module configured to determine a distributed data storage table corresponding to the target incremental data according to a data type of the target incremental data, and store the target incremental data into the distributed data storage table to obtain a pre-processing layer; a first processing module configured to perform aggregated calculation on data in the pre-processing data layer to obtain a summary data layer; and a second processing module configured to determine a statistical index corresponding to data in the summary data layer, and import the statistical index into a corresponding hot data storage system.

[0014] According to a further aspect of the embodiments of the present application, a non-volatile storage medium is also provided, which stores a program, wherein the program, when executed, controls a device in which the non-volatile storage medium is located to perform the data storage method.

[0015] According to a further aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory and a processor, wherein the processor is configured to execute a program stored in the memory, and the program, when executed, performs the data storage method.

[0016] In the embodiments of the present application, the target incremental data is acquired, the distributed data storage table corresponding to the target incremental data is determined according to the data type of the target incremental data, and the target incremental data is stored into the distributed data storage table to obtain the pre-processing layer; the aggregated calculation is performed on the data in the pre-processing data layer to obtain the summary data layer; the statistical index corresponding to the data in the summary data layer is determined, and the statistical index is imported into the corresponding hot data storage system, so that the incremental data is analyzed and processed according to the data type of the incremental data after the incremental data is acquired, the purpose of quickly processing the incremental data is achieved, the technical effect of improving the data processing timeliness is achieved, and the technical problem of poor data processing timeliness caused by the inability to timely process and provide the incremental data and the related analysis results to the user in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and the description thereof, and do not limit the present application. In the drawings:

[0018] Figure 1 FIG. 1 is a structural schematic diagram of a computer terminal according to an embodiment of the present application;

[0019] Figure 2 FIG. 2 is a flowchart of a data storage method according to an embodiment of the present application;

[0020] Figure 3is a structural schematic diagram of a data storage system according to an embodiment of the present application.

[0021] Figure 4 is a structural schematic diagram of a data storage device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] Iceberg: Apache Iceberg is an open table format for large analytical datasets, which uses a high-performance table format similar to SQL tables, mainly providing related operations of tables for computing engines such as Spark, Trino, PrestoDB, Flink, Hive and Impala, and Iceaberg can support PB-level large tables and support the management and performance characteristics of DBMS.

[0026] The first generation of data warehouse platform architecture is based on traditional database implementation, and the calculation and storage are coupled, the operation and expansion cost is high, and the non-structured data and PB level large data volume business scenarios are not supported. The second generation of data warehouse platform architecture is based on the double-layer architecture of data lake + data warehouse separation, which solves the problem of storage and calculation separation. However, due to multiple ETL, the data has low reliability, high delay, and increased storage cost. In order to solve this problem, the related solutions are provided in the embodiments of the present application, which are described in detail below.

[0027] According to the embodiments of the present application, a method embodiment of a data storage method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiment provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the data storage method is shown. As shown in Figure 1 The computer terminal 10 (or mobile device 10) can include one or more processors 102 (the processor 102 can include but is not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 104 for storing data, and a transmission module 106 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or less components than those shown in Figure 1 or have a different configuration from that shown in Figure 1 .

[0029] It should be noted that the one or more processors 102 and / or other data processing circuitry described above can be referred to herein generally as "data processing circuitry". The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuitry can be a single standalone processing module, or incorporated in whole or in part within any one of the other elements of the computer terminal 10 (or mobile device). As referred to in the embodiments of the present application, the data processing circuitry serves as a processor to control, for example, the selection of the variable resistance terminal path connected to the interface.

[0030] The memory 104 can be used to store software programs of application software and modules, such as the program instructions / data storage means corresponding to the data storage method of the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e. implements the data storage method of the application program described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory disposed remotely with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0031] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0032] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0033] Under the above operating environment, the embodiments of the present application provide a data storage method, as shown in Figure 2 The method includes the following steps:

[0034] In step S202, target incremental data is acquired;

[0035] In the technical solution provided in step S202, the step of obtaining the target incremental data comprises: deploying a distributed data processing cluster, wherein the distributed processing cluster comprises distributed data processing units corresponding to different data types; determining a data collection tool corresponding to the data type of the target incremental data according to the data type of the target incremental data, and obtaining the target incremental data through the data collection tool; determining a corresponding distributed data processing unit according to the data type of the target incremental data, and storing the target incremental data in the distributed data processing unit.

[0036] In some embodiments of the present application, the data type of the target incremental data comprises log data and business library change data, wherein, in the case that the data type of the target incremental data is log data, the data collection tool is a Flume cluster, and in the case that the data type of the target incremental data is business library change data, the data collection tool is Flink CDC.

[0037] As an optional implementation, the cluster type of the distributed data processing cluster comprises a kafka cluster, and the unit type of the distributed data processing unit comprises a kafka Topic.

[0038] Specifically, in order to obtain the incremental data, first, the kafka cluster required by the data bus needs to be deployed, and the kafka Topic corresponding to different source data is created respectively. Then, the incremental data can be synchronized to the Kafka topic corresponding to the message bus in real time through the Flume cluster monitoring log data, FlinkCDC monitoring business library change data and the like.

[0039] Step S204, determining a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data in the distributed data storage table to obtain a preprocessing layer.

[0040] In the technical solution provided in step S204, after the incremental data is collected, the incremental data corresponding to the Kakfa Topic in the message bus can be distributed according to the message, and is landed in different Iceberg tables in the distributed storage to form a preprocessing data layer.

[0041] Step S206, aggregating and calculating the data in the preprocessing data layer to obtain a summary data layer.

[0042] In the technical solution provided in step S206, the step of performing aggregate calculation on the data in the pre-processing data layer to obtain the summary data layer includes: performing data cleaning on the data stored in the pre-processing data layer to obtain an operation data layer; performing dimension degradation processing on the operation data layer to degrade the dimension table in the operation data layer to the fact table in the operation data layer to obtain a detailed data layer; performing wide table processing on the detailed data layer and performing aggregate calculation on the data in the detailed data layer to obtain the summary data layer.

[0043] Specifically, the step of performing data cleaning on the data stored in the pre-processing data layer to obtain the operation data layer includes writing an Apache Beam program to perform preliminary data cleaning on the pre-processing layer data to form an operation data layer (ODS) in an Iceberg table organization format, and store all basic data.

[0044] As an optional implementation, the specific implementation of the step of performing dimension degradation processing on the operation data layer to degrade the dimension table in the operation data layer to the fact table in the operation data layer to obtain the detailed data layer includes using a dimension degradation method based on the operation data layer, writing an Apache Beam program to degrade the dimension table to the fact table to form a detailed data layer (DWD) in an Iceberg table organization format, reducing the association between the fact table and the dimension table, and improving the usability of the detailed table.

[0045] In some embodiments of the present application, the specific steps of performing wide table processing on the detailed data layer and performing aggregate calculation on the data in the detailed data layer to obtain the summary data layer include: using a wide table method based on the detailed data layer, writing an Apache Beam program to perform preliminary aggregate calculation to form a public data indicator layer (DWS) in an Iceberg table organization format, i.e., the summary data layer.

[0046] Step S208: determining the statistical indicators corresponding to the data in the summary data layer, and importing the statistical indicators into the corresponding hot data storage system.

[0047] Specifically, the step of determining the statistical indicators and importing the statistical indicators into the corresponding hot data storage system includes: based on the data in the summary data layer, obtaining data product personalized statistical indicators through multi-dimensional analysis calculation, and importing the business indicators obtained by data aggregation into different hot data storage systems through writing an Apache Beam program, to accelerate the response speed of each business system in the upper layer.

[0048] In the technical solution provided in step S208, after the step of importing the statistical index into the corresponding hot data storage system, the data storage method further comprises: uniformly publishing the database table and the distributed data storage table of the hot data storage system to the outside; after uniformly publishing the database table and the distributed data storage table of the hot data storage system to the outside, receiving the data acquisition request sent by each business subsystem, and providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request.

[0049] It should be noted that, by uniformly publishing the database table in Iceaberg and each hot storage system, the data quality can be conveniently managed, so as to avoid falling into the data swamp.

[0050] As an optional implementation manner, the step of providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request comprises: determining the data acquisition authority of each business subsystem according to the data acquisition request; determining the acquirable data of each business subsystem according to the data acquisition authority; and providing each business subsystem with the corresponding acquirable data. In this way, each business subsystem can acquire corresponding data in different ranges from the data unified publishing system according to its own authority.

[0051] By acquiring the target incremental data; determining the distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data into the distributed data storage table to obtain a preprocessing layer; performing aggregation calculation on the data in the preprocessing data layer to obtain a summary data layer; determining the statistical index corresponding to the data in the summary data layer, and importing the statistical index into the corresponding hot data storage system, the incremental data is analyzed and processed according to the data type of the incremental data after the incremental data is acquired, so as to achieve the purpose of quickly processing the incremental data, thereby realizing the technical effect of improving the data processing timeliness, and further solving the technical problem of poor data processing timeliness caused by the inability to timely process and provide incremental data and related analysis results to users in the related art.

[0052] In the embodiments of the present application, a data storage system is provided. Figure 3 is a structural schematic diagram of the data storage system, like Figure 3As shown, the system includes a unified data collection layer 30, a unified data storage layer 32, a unified data calculation layer 34, and a unified data service layer 36. The unified data collection layer 30 is configured to collect target incremental data. The unified data storage layer 32 is configured to determine a distributed data storage table corresponding to the target incremental data according to a data type of the target incremental data, and store the target incremental data in the distributed data storage table to obtain a preprocessing layer. The unified data calculation layer 34 is configured to perform aggregation calculation on data in the preprocessing data layer to obtain an aggregated data layer. The unified data calculation layer 34 is further configured to determine a statistical index corresponding to data in the aggregated data layer, and import the statistical index into a corresponding hot data storage system. The unified data service layer 36 is configured to uniformly publish database tables of the hot data storage system and the distributed data storage table to the outside. After the database tables of the hot data storage system and the distributed data storage table are uniformly published to the outside, the unified data service layer 36 receives data acquisition requests sent by each business subsystem, and provides data corresponding to the data acquisition requests to each business subsystem according to the data acquisition requests.

[0053] In some embodiments of the present application, a data collection bus based on multi-source heterogeneous data is arranged in the unified data collection layer 30, which can realize unified data import into the lake, including real-time collection of log data and business data.

[0054] Specifically, the unified data collection layer 30 can acquire change data of multi-source heterogeneous data through real-time monitoring collection, and synchronize the collected change data to corresponding Kafka Topic according to different data source types, so as to realize unified synchronization of incremental change data to the message bus, and unified message processing and distribution. The different Kafka Topic data messages in the message bus can also be parsed and landed to the distributed storage preprocessing layer to obtain a database source table in an Iceberg organization form, and store the incremental change data.

[0055] In some embodiments of the present application, the unified data storage layer 32 is a data storage layer based on Iceberg to realize lake-warehouse integration. After all data is asynchronously imported into the lake, related ETL jobs of the data warehouse can perform data flow within the data lake, and Alluxio is used for data caching to accelerate data reading and writing.

[0056] Specifically, the unified data storage layer 32 can land the preprocessing layer data into an operation data layer (ODS), a detailed data layer (DWD), an aggregated data layer (DWS), and an application data layer (ADS) in an Iceberg table organization format after preliminary data cleaning, conversion, and aggregation, and synchronize the application data layer to the hot data storage system, so as to form a data warehouse and improve the reusability of the data lake table.

[0057] In addition, in the unified data storage layer 32, Alluxio is also introduced as a data cache layer. When a computing engine such as Spark or Flink accesses the file system, Alluxio acts as a distributed memory system to accelerate the read and write speed of data.

[0058] In some embodiments of the present application, the unified data computing layer 34 is a unified data computing layer constructed using Apache Beam. The distributed data processing task generated by the unified data computing layer 34 can be executed on a general distributed execution engine such as Flink or Spark, and the user can freely switch the execution engine and execution environment of the distributed data processing task.

[0059] Specifically, in the unified data computing layer 34, Apache Beam framework can be used for related data processing. SDK dependency is set according to the source data, data source is obtained by calling PCollection, Pipline is constructed for data aggregation and conversion, and Runner is specified to determine the computing engine for script publishing.

[0060] In addition, the unified data computing layer 34 can also publish the packaged script to a Flink cluster or a Spark cluster to execute and obtain a data processing result, thereby realizing the unification of the programming model and avoiding the complex management of multi-language scripts.

[0061] In some embodiments of the present application, the unified data service layer 36 is a unified data service layer constructed based on all library tables in the data warehouse. The data publishing can be standardized, thereby realizing the goal of data warehouse as a service.

[0062] Specifically, in the unified data service layer 36, Iceberg related library tables and hot data storage system related library tables formed by metadata, master data, business data and the like can be uniformly published, thereby realizing the unified publishing and management of data.

[0063] In addition, the unified data service layer 36 can also determine the corresponding visible data according to the permissions of each business subsystem, thereby enabling each business subsystem to obtain different ranges of corresponding data from the unified data publishing system according to its own permissions.

[0064] The lake-warehouse integrated data storage system provided in the embodiments of the present application greatly accelerates the enterprise data production and development cycle, thereby superimposing the values of the data warehouse and the data lake, overcoming the data swamp, reducing repeated construction, allowing the data in the lake to flow into the data warehouse and be directly called, and allowing the data in the data warehouse to be saved in the data lake for future data mining. With the lake-warehouse integrated data storage system, the hot data in the data warehouse and the historical data in the data lake can be quickly processed, providing stronger data support for market operation personnel and decision makers to make market decisions, responding to market demand changes more quickly, and acquiring business opportunities in time.

[0065] In addition, the lake-warehouse integrated data storage system provided in the embodiments of the present application can connect the data lake and the data warehouse from the infrastructure, provide real-time query and analysis capabilities, and make up for the deficiency of the Hadoop technology in real-time data processing capability; when storing data files, an open and standardized storage format is used, laying a foundation for the implementation of storage and computing separation, allowing the system to be extended to larger concurrent capabilities and data capacities, based on the scheme provided in the present application, the data mining related system can directly read the table in the data lake format, greatly accelerating the access efficiency of advanced analysis; and the lake-warehouse integrated architecture can also reduce the development cost and computing and storage overhead caused by data flow.

[0066] The embodiments of the present application provide a data storage device, Figure 4 is a structural schematic diagram of the data storage device, as Figure 4 shown, the device comprises: an acquisition module 40, configured to obtain target incremental data; a storage module 42, configured to determine a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and store the target incremental data into the distributed data storage table to obtain a preprocessing layer; a first processing module 44, configured to perform aggregation calculation on the data in the preprocessing data layer to obtain a summary data layer; and a second processing module 46, configured to determine a statistical index corresponding to the data in the summary data layer, and import the statistical index into a corresponding hot data storage system.

[0067] In some embodiments of the present application, the step of the acquisition module 40 obtaining the target incremental data comprises: deploying a distributed data processing cluster, wherein the distributed processing cluster comprises distributed data processing units corresponding to different data types; determining a data acquisition tool corresponding to the data type of the target incremental data, and obtaining the target incremental data through the data acquisition tool; and determining a corresponding distributed data processing unit according to the data type of the target incremental data, and storing the target incremental data into the distributed data processing unit.

[0068] In some embodiments of the present application, the data type of the target incremental data includes log data and business library change data, wherein, in the case that the data type of the target incremental data is log data, the data collection tool is a Flume cluster, and in the case that the data type of the target incremental data is business library change data, the data collection tool is Flink CDC.

[0069] In some embodiments of the present application, the cluster type of the distributed data processing cluster includes a kafka cluster, and the unit type of the distributed data processing unit includes a kafka Topic.

[0070] In some embodiments of the present application, the first processing module 44 performs aggregation calculation on the data in the pre-processing data layer to obtain the summary data layer, and the step includes: performing data cleaning on the data stored in the pre-processing data layer to obtain an operation data layer; performing dimension degradation processing on the operation data layer to degrade the dimension table in the operation data layer to the fact table in the operation data layer to obtain a detailed data layer; performing wide table processing on the detailed data layer, and performing aggregation calculation on the data in the detailed data layer to obtain the summary data layer.

[0071] In some embodiments of the present application, after the step of the second processing module 46 importing the statistical indicators into the corresponding hot data storage system, the second processing module 46 is further configured to: uniformly publish the database tables and the distributed data storage tables of the hot data storage system to the outside; after uniformly publishing the database tables and the distributed data storage tables of the hot data storage system to the outside, receiving data acquisition requests sent by each business subsystem, and providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request.

[0072] In some embodiments of the present application, the step of the second processing module 46 providing each business subsystem with data corresponding to the data acquisition request according to the data acquisition request includes: determining the data acquisition authority of each business subsystem according to the data acquisition request; determining the acquirable data of each business subsystem according to the data acquisition authority; and providing each business subsystem with the corresponding acquirable data.

[0073] It should be noted that each module in the above data storage device can be a program module (for example, a set of program instructions for implementing a certain specific function) or a hardware module. For the latter, it can be in the following form, but is not limited to this: the form of each module is a processor, or the functions of each module are implemented by a processor.

[0074] According to the embodiment of the present application, a nonvolatile storage medium is also provided, and the nonvolatile storage medium stores a program. When the program is executed, the device in which the nonvolatile storage medium is located performs the following data storage method: obtaining target incremental data; determining a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data into the distributed data storage table to obtain a preprocessing layer; performing aggregation calculation on the data in the preprocessing data layer to obtain an aggregated data layer; determining a statistical index corresponding to the data in the aggregated data layer, and importing the statistical index into a corresponding hot data storage system.

[0075] According to the embodiment of the present application, an electronic device is also provided, and the electronic device includes a memory and a processor. The processor is configured to execute a program stored in the memory. When the program is executed, the electronic device performs the following data storage method: obtaining target incremental data; determining a distributed data storage table corresponding to the target incremental data according to the data type of the target incremental data, and storing the target incremental data into the distributed data storage table to obtain a preprocessing layer; performing aggregation calculation on the data in the preprocessing data layer to obtain an aggregated data layer; determining a statistical index corresponding to the data in the aggregated data layer, and importing the statistical index into a corresponding hot data storage system.

[0076] In the above-described embodiments of the present application, the description of each embodiment focuses on different aspects. The parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0077] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0078] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0079] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0080] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part that essentially contributes to the related art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0081] The above is only the preferred embodiment of the present application. It should be pointed out that, for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which should be regarded as the protection scope of the present application.

Claims

1. A data storage method, characterized in that, include: Obtain the target incremental data; Based on the data type of the target incremental data, the distributed data storage table corresponding to the target incremental data is determined, and the target incremental data is stored in the distributed data storage table to obtain the preprocessing layer; The data in the preprocessing layer is aggregated and calculated to obtain the summary data layer; Determine the statistical indicators corresponding to the data in the summary data layer, and import the statistical indicators into the corresponding hot data storage system; The database tables of the hot data storage system and the distributed data storage tables are uniformly published to the outside world. The distributed data storage tables include metadata, master data, and business data. The system receives data acquisition requests from various business subsystems and provides data corresponding to the data acquisition requests to each business subsystem accordingly. Based on the data acquisition request, determine the data acquisition permissions of each business subsystem; Based on the data acquisition permissions, the data that can be acquired by each business subsystem is determined, wherein the data that can be acquired is the data within the range corresponding to the data acquisition permissions; Provide the corresponding available data to each of the business subsystems.

2. The data storage method according to claim 1, characterized in that, The steps for obtaining the target incremental data include: Deploy a distributed data processing cluster, wherein the distributed data processing cluster includes distributed data processing units corresponding to different data types; Based on the data type of the target incremental data, determine the data acquisition tool corresponding to the data type, and acquire the target incremental data through the data acquisition tool; Based on the data type of the target incremental data, the corresponding distributed data processing unit is determined, and the target incremental data is stored in the distributed data processing unit.

3. The data storage method according to claim 2, characterized in that, The data types of the target incremental data include log data and business database change data. When the data type of the target incremental data is log data, the data collection tool is a Flume cluster. When the data type of the target incremental data is business database change data, the data collection tool is Flink CDC.

4. The data storage method according to claim 2, characterized in that, The cluster type of the distributed data processing cluster includes a Kafka cluster, and the unit type of the distributed data processing unit includes a Kafka Topic.

5. The data storage method according to claim 1, characterized in that, The step of aggregating and calculating the data in the preprocessing layer to obtain the summary data layer includes: The data stored in the preprocessing layer is cleaned to obtain the operational data layer; The operational data layer is subjected to dimension degradation processing, which degenerates the dimension tables in the operational data layer into the fact tables in the operational data layer to obtain the detailed data layer. The detailed data layer is processed into a wide table, and the data in the detailed data layer is aggregated and calculated to obtain the summary data layer.

6. A data storage system, characterized in that, The data storage system includes a unified data acquisition layer, a unified data storage layer, a unified data computing layer, and a unified data service layer, wherein... The unified data acquisition layer is used to acquire target incremental data; The unified data storage layer is used to determine the distributed data storage table corresponding to the target incremental data based on the data type of the target incremental data, and store the target incremental data in the distributed data storage table to obtain the preprocessing layer; The unified data calculation layer is used to aggregate and calculate the data in the preprocessing layer to obtain the summary data layer; determine the statistical indicators corresponding to the data in the summary data layer, and import the statistical indicators into the corresponding hot data storage system; The unified data service layer is used to uniformly publish the database tables of the hot data storage system and the distributed data storage tables to the outside world. The distributed data storage tables include metadata, master data, and business data. After uniformly publishing the database tables of the hot data storage system and the distributed data storage tables, the layer receives data acquisition requests from various business subsystems and provides data corresponding to the data acquisition requests to each business subsystem. Based on the data acquisition requests, the layer determines the data acquisition permissions of each business subsystem; based on the data acquisition permissions, the layer determines the data that each business subsystem can acquire, wherein the acquireable data is data within the range corresponding to the data acquisition permissions; and the layer provides the corresponding acquireable data to each business subsystem.

7. A data storage device, characterized in that, include: The acquisition module is used to acquire target incremental data; The storage module is used to determine the distributed data storage table corresponding to the target incremental data based on the data type of the target incremental data, and store the target incremental data in the distributed data storage table to obtain the preprocessing layer; The first processing module is used to perform aggregation calculations on the data in the preprocessing layer to obtain the summary data layer. The second processing module is used to determine the statistical indicators corresponding to the data in the summary data layer and import the statistical indicators into the corresponding hot data storage system. The database tables of the hot data storage system and the distributed data storage tables are uniformly published externally. The distributed data storage tables include metadata, master data, and business data. Data acquisition requests are received from various business subsystems, and data corresponding to the data acquisition requests is provided to each business subsystem. Based on the data acquisition requests, the data acquisition permissions of each business subsystem are determined. Based on the data acquisition permissions, the obtainable data for each business subsystem is determined, where the obtainable data is the data range corresponding to the data acquisition permissions. The corresponding obtainable data is then provided to each business subsystem.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the data storage method according to any one of claims 1 to 5.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data storage method according to any one of claims 1 to 5.

Images