A method, apparatus and electronic device for real-time offline data storage

Abstract

This invention provides a method, apparatus, and electronic device for offline storage of real-time data, relating to the field of data processing technology. The method includes: acquiring real-time data to be stored; determining the data type of the real-time data based on attribute parameters of the real-time data to obtain a target data type; determining a target offline table corresponding to the target data type based on a pre-set correspondence between data types and offline tables; and storing the real-time data in the target offline table, thereby enabling more efficient offline storage of real-time data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a real-time data offline storage method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of big data application technology, real-time computing as the core technology of big data processing can perform instant calculation and analysis on real-time data, thereby dynamically processing the current application scenario. Therefore, the application scenarios of real-time computing are increasing. It can be understood that in the processing of real-time data by real-time computing, the real-time data inevitably has the need for offline writing, for example, offline backup, offline processing, etc. However, the offline writing scheme for real-time data in the prior art is relatively single and cannot meet the diversified needs of the increasing application scenarios for offline writing of real-time data. SUMMARY

[0003] The purpose of the embodiment of the present application is to provide a real-time data offline storage method, device and electronic equipment to realize more perfect offline storage of real-time data. The specific technical scheme is as follows:

[0004] In the first aspect of the present application, a real-time data offline storage method is first provided, which comprises:

[0005] obtaining real-time data to be stored;

[0006] determining the data type of the real-time data according to the attribute parameter of the real-time data, to obtain a target data type, wherein the attribute parameter of the real-time data includes at least one of the access frequency, data timeliness and access reflection time of the real-time data, and the data type includes cold data, hot data and warm data;

[0007] determining a target offline table corresponding to the target data type based on a pre-set correspondence between data type and offline table, wherein the correspondence includes that cold data corresponds to an offline table in a data warehouse, warm data corresponds to an offline table in a data lake, and hot data corresponds to an offline table in a query engine;

[0008] storing the real-time data into the target offline table.

[0009] In an embodiment of the present application, the target offline table includes a plurality of data partitions, each data partition corresponding to a corresponding time period; and the step of storing the real-time data into the target offline table comprises:

[0010] obtaining a current data writing mode and an effective writing time of the real-time data, wherein the effective writing time of the real-time data is a generation time of the real-time data and / or a receiving time of the real-time data;

[0011] determining a target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and a time period corresponding to each data partition in the target offline table;

[0012] storing the real-time data into the target data partition.

[0013] In an embodiment of the present application, the current data writing mode is a system time writing mode, and the effective writing time of the real-time data is the receiving time of the real-time data.

[0014] The determining of the target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and the time period corresponding to each data partition in the target offline table, comprises:

[0015] In the case that the current data writing mode is the system time writing mode, a time period in which the receiving time of the real-time data is located is determined to obtain a target time period; a data partition corresponding to the target time period in the target offline table is taken as the target data partition corresponding to the real-time data.

[0016] In an embodiment of the present application, the current data writing mode is a strict event time writing mode, and the effective writing time of the real-time data is the generation time of the real-time data.

[0017] The determining of the target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and the time period corresponding to each data partition in the target offline table, comprises:

[0018] In the case that the current data writing mode is the strict event time writing mode, a time period in which the generation time of the real-time data is located is determined to obtain a target time period; a data partition corresponding to the target time period in the target offline table is taken as the target data partition corresponding to the real-time data.

[0019] In an embodiment of the present application, the current data writing mode is a supplementary event time writing mode, and the effective writing time of the real-time data is the generation time of the real-time data and the receiving time of the real-time data.

[0020] The determining of the target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and the time period corresponding to each data partition in the target offline table, comprises:

[0021] In the case that the current data writing mode is a supplement event time writing mode, determine a time difference between the generation time of the real-time data and the current time;

[0022] In the case that the time difference is greater than a preset time threshold, determine a time period in which the receiving time of the real-time data is located, to obtain a target time period; and take the data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data;

[0023] In the case that the time difference is not greater than the preset time threshold, determine a time period in which the generation time of the real-time data is located, to obtain a target time period; and take the data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data.

[0024] In an embodiment of the present application, the method further comprises:

[0025] In the case that the target data partition corresponding to the effective writing time does not exist in each data partition of the target offline table, divide the target data partition corresponding to the effective writing time in the target offline table.

[0026] In an embodiment of the present application, the method further comprises:

[0027] Monitor the target data partition, and after storing the real-time data into the target data partition, mark the target data partition as storage completed and send a prompt information indicating storage completion.

[0028] In an embodiment of the present application, after storing the real-time data into the target data partition, the method further comprises:

[0029] Determine whether offline data exists in the target data partition;

[0030] If the offline data exists, determine whether the real-time data is consistent with the offline data, and if consistent, determine that the real-time data is written completely;

[0031] If the offline data does not exist in the target data partition or the real-time data is not consistent with the offline data, determine that the real-time data is not written completely and send a prompt information indicating incomplete writing.

[0032] In a second aspect of the present application, a real-time data offline storage device is further provided, which comprises:

[0033] A data acquisition module, configured to acquire real-time data to be stored;

[0034] The data type determination module is configured to determine a data type of the real-time data according to an attribute parameter of the real-time data, and obtain a target data type, wherein the attribute parameter of the real-time data comprises at least one of an access frequency, data timeliness and access reflection time of the real-time data, and the data type comprises cold data, hot data and warm data.

[0035] The offline table determination module is configured to determine a target offline table corresponding to the target data type based on a preset correspondence between data types and offline tables, wherein the correspondence comprises that cold data corresponds to an offline table in a data warehouse, warm data corresponds to an offline table in a data lake, and hot data corresponds to an offline table in a query engine.

[0036] The data storage module is configured to store the real-time data into the target offline table.

[0037] In an embodiment of the present application, the target offline table comprises a plurality of data partitions, and each data partition corresponds to a corresponding time period; and the data storage module comprises:

[0038] The data acquisition sub-module is configured to acquire a current data writing mode and an effective writing time of the real-time data, wherein the effective writing time of the real-time data is a generation time of the real-time data and / or a receiving time of the real-time data.

[0039] The data partition determination sub-module is configured to determine a target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and time periods corresponding to each data partition in the target offline table.

[0040] The data storage sub-module is configured to store the real-time data into the target data partition.

[0041] In an embodiment of the present application, the device further comprises:

[0042] The data partition division sub-module is configured to divide a target data partition corresponding to the effective writing time in the target offline table when the target data partition does not exist in each data partition of the target offline table.

[0043] In an embodiment of the present application, the device further comprises:

[0044] The data partition monitoring sub-module is configured to monitor the target data partition, mark the target data partition as storage completed after storing the real-time data into the target data partition, and send prompt information indicating storage completion.

[0045] In an embodiment of the present application, the device further comprises:

[0046] a data judgment submodule, configured to:

[0047] determine whether offline data exists in the target data partition;

[0048] if the offline data exists, determine whether the real-time data is consistent with the offline data, and if the real-time data is consistent with the offline data, determine that the real-time data is written completely;

[0049] if the offline data does not exist in the target data partition or the real-time data is not consistent with the offline data, determine that the real-time data is not written completely and send prompt information indicating that the real-time data is not written completely.

[0050] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a writer and a communication bus, wherein the processor, the communication interface and the writer are in communication with each other through the communication bus;

[0051] the writer is configured to store a computer program;

[0052] the processor is configured to execute the program stored in the writer, thereby implementing the method steps described above.

[0053] In a fourth aspect, a computer readable storage medium is provided, wherein the computer readable storage medium has a computer program written therein, and the computer program is executed by a processor to implement the method steps described above.

[0054] As can be seen from the above, the real-time data offline storage method provided by the embodiments of the present application comprises the following steps: obtaining real-time data to be stored, and determining a target data type of the real-time data. Based on a pre-set correspondence between the data type and an offline table, a target offline table corresponding to the target data type is determined, and then the real-time data is stored in the target offline table. As can be seen, the data is stored in the corresponding target offline table according to the respective data types, and the different types of data can be stored according to the types of the data, so that the storage requirements of different types of data can be met, the efficiency of the real-time data offline storage is improved, and the real-time data can be stored more perfectly. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0056] Figure 1 a flowchart of a real-time data offline storage method provided by the embodiments of the present application;

[0057] Figure 2A schematic diagram of real-time data corresponding to a target offline table according to a data type is provided for an embodiment of the present application.

[0058] Figure 3 A flowchart of a first possible implementation of step S14 is provided for an embodiment of the present application.

[0059] Figure 4 A flowchart of a first possible implementation of step S32 is provided for an embodiment of the present application.

[0060] Figure 5 A flowchart of a second possible implementation of step S32 is provided for an embodiment of the present application.

[0061] Figure 6 A flowchart of a third possible implementation of step S32 is provided for an embodiment of the present application.

[0062] Figure 7 A flowchart of a second possible implementation of step S14 is provided for an embodiment of the present application.

[0063] Figure 8a A flowchart of a possible implementation of step S33 is provided for an embodiment of the present application.

[0064] Figure 8b A flowchart of an offline storage example of real-time data is provided for an embodiment of the present application.

[0065] Figure 9 A structural schematic diagram of an offline storage device of real-time data is provided for an embodiment of the present application.

[0066] Figure 10 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0068] Since the offline writing scheme of real-time data in the prior art is relatively single, it cannot meet the diversified needs of real-time data offline writing in the increasing application scenarios. In order to solve this technical problem, the present application provides a real-time data offline storage method, device and electronic device.

[0069] In an embodiment of the present application, a real-time data offline storage method is provided, and the above-mentioned method comprises:

[0070] acquiring real-time data to be stored;

[0071] According to the attribute parameter of the real-time data, the data type of the real-time data is determined, and a target data type is obtained, wherein the attribute parameter of the real-time data includes at least one of access frequency, data timeliness and access reflection time of the real-time data, and the data type includes cold data, hot data and warm data.

[0072] Based on the correspondence relationship between the preset data type and the offline table, a target offline table corresponding to the target data type is determined, wherein the correspondence relationship includes that the cold data corresponds to the offline table in the data warehouse, the warm data corresponds to the offline table in the data lake, and the hot data corresponds to the offline table in the query engine.

[0073] The real-time data is stored into the target offline table.

[0074] As can be seen from the above, the real-time data offline storage method provided by the embodiment of the application acquires the real-time data to be stored, and determines the target data type of the real-time data, including cold data, hot data and warm data. Based on the correspondence relationship between the preset data type and the offline table, the target offline table corresponding to the target data type is determined, and then the real-time data is stored into the target offline table. It can be seen that the data is divided into cold data, hot data and warm data and stored into the corresponding target offline table. Different types of data can be stored according to the type of the data, so that the storage requirements of different types of data can be met, and different types of real-time data can be conveniently managed, the efficiency of real-time data offline storage is improved, and the real-time data can be more perfectly offline stored.

[0075] The real-time data offline storage method provided by the embodiment of the application will be described in detail below through specific embodiments.

[0076] The method of the embodiment of the application is applied to a smart terminal, and can be implemented through the smart terminal. In actual use, the smart terminal can be a device that has a real-time data offline storage function itself, or can be a separate device specially used for real-time data offline storage. Specifically, the smart terminal can be a computer, a server or the like.

[0077] Referring to Figure 1 , a flowchart of a real-time data offline storage method is provided, and the above method includes the following steps S11-S14.

[0078] Step S11: Acquire real-time data to be stored.

[0079] Real-time data is live data with time attribute. Offline storage of the real-time data to be stored is a process of collecting and storing live data within a certain time of occurrence of the live data.

[0080] Step S12: determining the data type of the real-time data according to the attribute parameter of the real-time data, to obtain a target data type.

[0081] The attribute parameter of the real-time data includes at least one of access frequency, data timeliness and access reflection time of the real-time data, and the data type includes cold data, hot data and warm data.

[0082] The real-time data with different data characteristics has different attribute parameters, and the real-time data with different data characteristics can be divided into different data types according to the attribute parameters of the real-time data. In the embodiment of the application, the attribute parameter includes at least one of the access frequency, data timeliness and access reflection time of the real-time data, so that the real-time data can be divided into different data types according to at least one of the access frequency, data timeliness and access reflection time of the real-time data.

[0083] In one example, the cold data can be data that meets at least one of the following three conditions: the lowest access frequency, the lowest data timeliness and the longest access reflection time. The hot data can be data that meets at least one of the following three conditions: the highest access frequency, the highest data timeliness and the shortest access reflection time. The access frequency, data timeliness and access reflection time of the warm data are between those of the cold data and the hot data.

[0084] Therefore, after obtaining the real-time data to be stored, the target data type of the real-time data can be determined.

[0085] Step S13: determining a target offline table corresponding to the target data type based on a pre-set correspondence between the data type and the offline table.

[0086] The correspondence includes that the cold data corresponds to an offline table in a data warehouse, the warm data corresponds to an offline table in a data lake, and the hot data corresponds to an offline table in a query engine.

[0087] The offline table can be a database for offline storage of real-time data. The correspondence between the data type and the offline table can be pre-set, and data with different data types can be stored in different offline tables.

[0088] In the embodiment of the application, when the target data type is cold data, the target offline table is an offline table in a data warehouse, and the data warehouse is an integrated database that can reflect historical changes of data. When the target data type is warm data, the target offline table is an offline table in a data lake, and the data lake is a data storage or system that stores data in a raw format without the need for prior structured processing of data. When the target data type is hot data, the target offline table is an offline table in a query engine, and the query engine can be an open source distributed query engine that integrates search, analysis and data storage, facilitating timely processing of hot data.

[0089] In one example, as shown in Figure 2 Figure 2 Different target offline tables corresponding to different data types of real-time data are shown. Real-time data is written by the device and stored in different target offline tables according to data types, wherein the target offline table of cold data can be an offline table in a data warehouse, the data warehouse can be Hive (a data warehouse tool), and the cold data meets any one of the following conditions: the lowest access frequency, the lowest data timeliness, and the longest access reflection time. The data warehouse has stable storage performance, and the data stored therein is neither easy to modify nor easy to lose, which can meet the storage demand of stable storage of cold data with low calling frequency. The target offline table of hot data can be an offline table in a query engine, and the query engine can be ES (ElasticSearch, real-time query engine). The query engine can quickly capture and process data, which can meet the storage demand of hot data with high calling frequency. The target offline table of warm data can be an offline table in a data lake, and the data lake is between the data warehouse and the query engine, and has balanced and moderate storage stability and data retrieval speed, which meets the storage demand of warm data with calling frequency between cold data and hot data.

[0090] After obtaining the real-time data to be stored, the target data type of the real-time data can be determined, and the offline table corresponding to the target data type can be determined.

[0091] Step S14: storing the real-time data in the target offline table.

[0092] After determining the target data type of the real-time data and the target offline table corresponding thereto, the real-time data can be stored in the target offline table.

[0093] As can be seen from the above, the real-time data offline storage method provided by the embodiment of the present application acquires real-time data to be stored and determines the target data type of the real-time data. Based on the pre-set correspondence between the data type and the offline table, the target offline table corresponding to the target data type is determined, and then the real-time data is stored in the target offline table. It can be seen that the data is stored in the corresponding target offline table according to the respective data types, and different types of data can be stored according to the types of data, so that the storage demand of different types of data can be met, and different types of real-time data can be managed conveniently, the efficiency of real-time data offline storage is improved, and real-time data can be more perfectly offline stored.

[0094] In one possible implementation, the target offline table includes a plurality of data partitions, and each data partition corresponds to a corresponding time period; see Figure 3 ​The step S14 stores the real-time data into the target offline table, including:

[0095] Step S31: obtaining a current data writing mode and a valid writing time of the real-time data.

[0096] The valid writing time of the real-time data is a generation time of the real-time data and / or a receiving time of the real-time data received by the offline local.

[0097] The data writing mode is an offline local preset, and is the most appropriate offline storage mode for the real-time data, which can be changed according to different application scenarios. In one example, the data writing mode can be related to the valid writing time of the real-time data, and in different data writing modes, the valid writing time of the real-time data can be different, and the actual writing time of the real-time data to the offline local can also be different. The current data writing mode is the data writing mode currently set by the offline local.

[0098] Step S32: determining a target data partition corresponding to the real-time data according to the current data writing mode, the valid writing time and the time period corresponding to each data partition in the target offline table.

[0099] Step S33: storing the real-time data into the target data partition.

[0100] In one embodiment of the present application, when the target data partition corresponding to the valid writing time does not exist in each data partition of the target offline table, the target data partition corresponding to the valid writing time is divided in the target offline table.

[0101] The target offline table includes a plurality of data partitions for storing real-time data. In one embodiment of the present application, each data partition in each offline table can be distinguished according to a time period, and each time period can correspond to a data partition. For example, the offline table is divided into a plurality of data partitions according to different dates, and the same data partition is divided into 24 data partitions according to 24 hours, and 24 hours in a day correspond to 24 data partitions.

[0102] When the real-time data to be stored is obtained, the actual writing time of the real-time data to the offline local is determined according to the current data writing mode and the valid writing time, and the data partition corresponding to the time period in which the time is located is determined as the target data partition.

[0103] If the target data partition corresponding to the valid writing time does not exist in each data partition of the target offline table, the target data partition corresponding to the valid writing time is divided in the target offline table.

[0104] After the target data partition is determined, the real-time data to be stored can be stored in the target data partition.

[0105] From the above, the real-time data offline storage method provided by the application determines the target data partition of the real-time data offline storage according to the offline local current data write mode and the effective write time of the real-time data, and multiple data write modes provide rich choices for different application scenarios, can meet different application scenarios for different needs of real-time data offline storage, compared with the direct storage mode of real-time data, this way of determining different effective write time of real-time data according to different needs, and then storing real-time data in the data partition more in line with the needs is more flexible, convenient for multi-form management of real-time data and subsequent retrieval and application of real-time data.

[0106] In a possible implementation manner, the current data write mode is a system time write mode, and the effective write time of the real-time data is a receiving time of the real-time data.

[0107] Referring to Figure 4 The step S32 determines the target data partition corresponding to the real-time data according to the current data write mode, based on the effective write time and the time period corresponding to each data partition in the target offline table, and includes the following steps.

[0108] The step S41: in the case that the current data write mode is the system time write mode, determining a time period in which a receiving time of the real-time data is located, obtaining a target time period; taking a data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data.

[0109] The system time write mode can refer to a mode of writing data according to system time, in the case that the offline local current data write mode is the system time write mode, taking a time period in which a receiving time of the real-time data is located as a target time period, then in the target offline table, taking a data partition corresponding to the target time period as a target data partition, and storing the real-time data into the target data partition.

[0110] For example, in the case that the offline local current data write mode is the system time write mode, the current time of the offline local is 10 o'clock, and the real-time data to be stored generated at 6 o'clock is received, taking a time period in which the receiving time 10 o'clock is located as a target time period, and storing the real-time data into a target data partition corresponding to the target time period.

[0111] From the above, the real-time data offline storage method provided by the application can strictly manage and store real-time data according to the receiving time of real-time data under the system time, meet the demand for real-time data storage in the application scenario of searching and calling real-time data according to the system time, and improve the storage efficiency of real-time data and the subsequent management efficiency under this condition.

[0112] In a possible implementation manner, the current data write mode is a strict event time write mode, and the valid write time of real-time data is the generation time of real-time data.

[0113] Referring to Figure 5 The step S32 determines the target data partition corresponding to the real-time data according to the current data write mode, based on the valid write time and the time period corresponding to each data partition in the target offline table, and includes the following steps.

[0114] The step S51: The strict event time write mode can refer to a mode of writing data strictly according to the time of an event itself. In the case where the current data write mode is the strict event time write mode, the time period in which the generation time of real-time data is determined is obtained, and the target time period is obtained. The data partition corresponding to the target time period in the target offline table is taken as the target data partition corresponding to the real-time data.

[0115] In the case where the current data write mode of the offline local is the strict event time write mode, the time period in which the real-time data is generated, i.e., the generation time, is taken as the target time period. Then, in the target offline table, the data partition corresponding to the target time period is taken as the target data partition, and the real-time data is stored in the target data partition.

[0116] For example, in the case where the current data write mode of the offline local is the strict event time write mode, the current time of the offline local is 10 o'clock, and the real-time data to be stored with a generation time of 6 o'clock is received. The real-time data is still stored in the target data partition corresponding to the target time period in which the generation time 6 o'clock is located.

[0117] From the above, the real-time data offline storage method provided by the application can set the valid writing time of the real-time data as the generation time of the real-time data, determine the corresponding data partition of the real-time data according to the generation time of the real-time data, that is, store the real-time data in the data partition corresponding to the time period in which the generation time of the real-time data is located as the target data partition, strictly manage and store the real-time data according to the time of event occurrence, meet the requirement of real-time data storage in the application scenario of searching and calling real-time data according to the event time, and improve the storage efficiency of real-time data and the subsequent management efficiency.

[0118] In a possible implementation manner, the current data writing mode is a supplementary event time writing mode, and the valid writing time of the real-time data is the generation time of the real-time data and the receiving time of the real-time data.

[0119] Referring to Figure 6 The step S32 determines the target data partition corresponding to the real-time data according to the current data writing mode, based on the valid writing time and the time period corresponding to each data partition in the target offline table, and includes the following steps.

[0120] The step S61 calculates the time difference between the generation time of the real-time data and the current time, in the case that the current data writing mode is a supplementary event time writing mode.

[0121] The step S62 determines the time period in which the receiving time of the real-time data is located, to obtain a target time period, in the case that the time difference is greater than a preset time threshold, and takes the data partition corresponding to the target time period in the target offline table as the target data partition corresponding to the real-time data.

[0122] The step S63 determines the time period in which the generation time of the real-time data is located, to obtain a target time period, in the case that the time difference is not greater than a preset time threshold, and takes the data partition corresponding to the target time period in the target offline table as the target data partition corresponding to the real-time data.

[0123] The preset time threshold can be preset, and can be one hour, five hours, etc., and is set according to requirements.

[0124] In the embodiment of the application, in the case that the current data writing mode of the offline local is a supplementary event time writing mode, the time difference between the generation time of the real-time data and the current system time of the offline local is first calculated.

[0125] When the time difference value is greater than the preset time threshold, the time period in which the receiving time of the real-time data is located is taken as a target time period, and then in the target offline table, the data partition corresponding to the target time period is taken as a target data partition, and the real-time data is stored into the target data partition. It can also be understood that when the time difference value is greater than the preset time threshold, the real-time data can be stored according to the system time writing mode.

[0126] For example, when the preset time threshold is one hour, at the current time of 10 o'clock offline locally, the real-time data to be stored generated at 6 o'clock is received, and since the time difference value is four hours, which is greater than the preset time threshold, the real-time data is stored into the target data partition corresponding to the target time period according to the receiving time of 10 o'clock.

[0127] When the time difference value is not greater than the preset time threshold, the time period in which the generation time of the real-time data is located is taken as a target time period, and then in the target offline table, the data partition corresponding to the target time period is taken as a target data partition, and the real-time data is stored into the target data partition. It can also be understood that when the time difference value is not greater than the preset time threshold, the real-time data can be stored according to the strict event time writing mode.

[0128] For example, when the preset time threshold is one hour, at the current time of 10 o'clock offline locally, the real-time data to be stored generated at 9 o'clock and 30 minutes is received, and since the time difference value is half an hour, which is not greater than the preset time threshold, the real-time data is stored into the target data partition corresponding to the target time period according to the generation time of 9 o'clock and 30 minutes.

[0129] From the above, the real-time data offline storage method provided by the application can determine the storage scheme of the real-time data according to the time difference between the generation time of the real-time data and the current offline local time, when the time difference is greater than the preset time threshold, the generation time of the real-time data is considered to be unable to serve as the effective write time, at this time, the receiving time of the real-time data is taken as the effective write time, and the real-time data is stored in the data partition corresponding to the time period in which the receiving time is located, when the time difference is not greater than the preset time threshold, the generation time of the real-time data is considered to still be able to serve as the effective write time for the real-time data, and the generation time of the real-time data can be taken as the effective write time, and the real-time data is stored in the data partition corresponding to the time period in which the generation time is located, this data write mode can meet the reasonable storage of the late real-time data, the real-time data can be supplemented within a certain time range, meanwhile, data storage disorder caused by excessively late real-time data is avoided, effective storage and processing of the real-time data in this case are realized, and the completeness and visibility of the real-time data offline storage are improved.

[0130] In an embodiment of the application, referring to Figure 7 , Figure 7 The flowchart of the second possible implementation mode of step S14 provided by the embodiment of the application is shown in the figure, and the above method further comprises:

[0131] Step S71: monitoring the target data partition, after the real-time data is stored in the target data partition, the target data partition is marked as storage completed and prompt information indicating storage completion is sent.

[0132] In the embodiment of the application, each offline table can be monitored in real time, when real-time data is written, the target data partition is monitored. After the real-time data is stored in the target data partition, the target data partition can be marked as storage completed. When the target data partition is marked as storage completed, prompt information indicating storage completion can be sent to prompt the user that the target data partition has been stored, for example, pop-up window prompt, ringing prompt, etc.

[0133] From the above, the real-time data offline storage method provided by the application can determine the storage scheme of the real-time data according to the time difference between the generation time of the real-time data and the current offline local time, when the time difference is greater than the preset time threshold, the generation time of the real-time data is considered to be unable to serve as the effective write time, at this time, the receiving time of the real-time data is taken as the effective write time, and the real-time data is stored in the data partition corresponding to the time period in which the receiving time is located, when the time difference is not greater than the preset time threshold, the generation time of the real-time data is considered to still be able to serve as the effective write time for the real-time data, and the generation time of the real-time data can be taken as the effective write time, and the real-time data is stored in the data partition corresponding to the time period in which the generation time is located, this data write mode can meet the reasonable storage of the late real-time data, the real-time data can be supplemented within a certain time range, meanwhile, data storage disorder caused by excessively late real-time data is avoided, effective storage and processing of the real-time data in this case are realized, and the completeness and visibility of the real-time data offline storage are improved.

[0134] In an embodiment of the application, referring to Figure 8a After the real-time data is stored in the target data partition in step S33, the above method further comprises:

[0135] Step S81: judging whether there is offline data in the target data partition.

[0136] Step S82: if there is, judging whether the real-time data is consistent with the offline data, if consistent, determining that the real-time data is written completely.

[0137] Step S83: if there is no offline data in the target data partition or the real-time data is inconsistent with the offline data, determining that the real-time data is not written completely and sending prompt information indicating that the writing is not complete.

[0138] After the real-time data is stored in the target data partition, that is, after the writing of the target data partition is completed, firstly, it is judged whether there is offline data in the target data partition, if there is, it is judged whether the total amount of data of the real-time data is consistent with the total amount of data of the offline data in the target data partition, if completely same, it can be determined that the real-time data is written completely. If there is no offline data in the target data partition or the total amount of data of the real-time data is inconsistent with the total amount of data of the offline data in the target data partition, it is determined that the real-time data is not written completely.

[0139] At this time, prompt information indicating that the writing is not complete can be sent, and the incomplete writing of the real-time data is alarmed to the user as an abnormal situation, so that the user can know the writing situation of the real-time data in time.

[0140] In one example, as shown in Figure 8b , Figure 8b a flow diagram of an example of offline storage of real-time data is shown, the real-time data is written through a writing module, and is stored in target data partitions of a data warehouse (for example, Hive), a data lake and a query engine (for example, ES) according to cold, warm and hot three types of data, and is made visible to users through a data visibility module, after the real-time data is stored in the target data partition, the target data partition is marked as storage completed, then a completion notification module is used to inform the user that the offline storage of the real-time data is completed, and finally a quality verification module is used to verify whether the real-time data is written completely according to the above steps.

[0141] As can be seen from the above, the offline storage method of real-time data provided by the present application can judge whether the real-time data is written completely and send prompt information indicating that the writing is not complete to the user in time to alarm, so as to ensure the integrity and security of the storage of the real-time data.

[0142] Referring to Figure 9 , the embodiment of the present application also provides a structural diagram of an offline storage device of real-time data, and the above device comprises:

[0143] A data acquisition module 901 is used to acquire real-time data to be stored.

[0144] The data type determination module 902 is configured to determine a data type of the real-time data according to an attribute parameter of the real-time data, to obtain a target data type, wherein the attribute parameter of the real-time data includes at least one of an access frequency, data timeliness and access reflection time of the real-time data, and the data type includes cold data, hot data and warm data.

[0145] The offline table determination module 903 is configured to determine a target offline table corresponding to the target data type based on a preset correspondence relationship between the data type and the offline table, wherein the correspondence relationship includes that the cold data corresponds to an offline table in a data warehouse, the warm data corresponds to an offline table in a data lake, and the hot data corresponds to an offline table in a query engine.

[0146] The data storage module 904 is configured to store the real-time data into the target offline table.

[0147] As can be seen from the above, the real-time data offline storage device provided by the embodiment of the present application can obtain the real-time data to be stored, and determine a target data type of the real-time data, including cold data, hot data and warm data. Based on a preset correspondence relationship between the data type and the offline table, a target offline table corresponding to the target data type is determined, and then the real-time data is stored into the target offline table. It can be seen that the data is divided into cold data, hot data and warm data and stored into the corresponding target offline table. Different types of data can be stored according to the type of the data, so that the storage requirements of different types of data can be met, and different types of real-time data can be conveniently managed, the efficiency of real-time data offline storage is improved, and the real-time data can be more perfectly offline stored.

[0148] In an embodiment of the present application, the target offline table includes a plurality of data partitions, and each data partition corresponds to a corresponding time period; the data storage module 904 includes:

[0149] The data acquisition sub-module is configured to acquire a current data writing mode and an effective writing time of the real-time data, wherein the effective writing time of the real-time data is a generation time of the real-time data and / or a receiving time of the real-time data.

[0150] The data partition determination sub-module is configured to determine a target data partition corresponding to the real-time data according to the current data writing mode, based on the effective writing time and the time period corresponding to each data partition in the target offline table.

[0151] The data storage sub-module is configured to store the real-time data into the target data partition.

[0152] From the above, the real-time data offline storage device provided by the application can meet different requirements of real-time data offline storage in different application scenarios, and can store real-time data in a data partition that is more in line with requirements, thereby facilitating management and subsequent application of the real-time data.

[0153] In an embodiment of the application, the device further comprises:

[0154] The data partition dividing sub-module is configured to divide the target data partition corresponding to the effective write time in the target offline table when the target data partition corresponding to the effective write time does not exist in each data partition of the target offline table.

[0155] In an embodiment of the application, the device further comprises:

[0156] The data partition monitoring sub-module is configured to monitor the target data partition, mark the target data partition as storage completed after storing the real-time data in the target data partition, and send prompt information indicating storage completion.

[0157] From the above, the real-time data offline storage device provided by the application can mark the target data partition as storage completed after monitoring storage of real-time data in the target data partition, thereby ensuring completeness of real-time data storage and improving applicability of the application by timely informing the user.

[0158] In an embodiment of the application, the device further comprises:

[0159] The data judging sub-module is configured to:

[0160] judge whether offline data exists in the target data partition;

[0161] if the offline data exists, judge whether the real-time data is consistent with the offline data, and if the real-time data is consistent with the offline data, determine that the real-time data is written completely;

[0162] if the offline data does not exist in the target data partition or the real-time data is not consistent with the offline data, determine that the real-time data is not written completely and send prompt information indicating that the real-time data is not written completely.

[0163] From the above, the real-time data offline storage device provided by the application can judge whether the real-time data is written completely by verification after storing the real-time data in the target data partition, and send prompt information indicating that the real-time data is not written completely to the user, thereby ensuring completeness and safety of real-time data storage.

[0164] The embodiment of the present application further provides an electronic device, such as Figure 10 As shown in the figure, the electronic device comprises a processor 1001, a communication interface 1002, a writer 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the writer 1003 complete communication with each other through the communication bus 1004,

[0165] The writer 1003 is used for storing a computer program.

[0166] The processor 1001 is used for executing the program stored in the writer 1003, so as to realize the real-time data offline storage method mentioned above.

[0167] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the terminal and other devices.

[0169] The writer can comprise a Random Access Memory (RAM) and can also comprise a non-volatile memory, for example at least one disk writer. Optionally, the writer can also be at least one writing device located away from the processor.

[0170] The processor mentioned above can be a general processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. and can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0171] In yet another embodiment of the present application, a computer readable storage medium is provided, having a computer program written therein, which, when executed by a processor, implements the real-time data offline storage method of any of the above embodiments.

[0172] In yet another embodiment of the present application, a computer program product containing instructions, which, when executed on a computer, causes the computer to perform the real-time data offline storage method of any of the above embodiments.

[0173] In the above embodiments, the implementation can be wholly or partially in software, hardware, firmware or any combination thereof. When implemented in software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions cause the computer to perform the processes or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be written in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data writing device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0174] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0175] Various embodiments are described in the specification with reference to a specific manner of describing the embodiments, and the same or similar parts among the various embodiments refer to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the device and electronic equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the part of the method embodiment.

[0176] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for storing real-time data offline, characterized in that, The method comprises: acquiring real-time data to be stored; determining a data type of the real-time data according to an attribute parameter of the real-time data, to obtain a target data type, wherein the attribute parameter of the real-time data comprises at least one of an access frequency, data timeliness and access reflection time of the real-time data, and the data type comprises cold data, hot data and warm data; determining a target offline table corresponding to the target data type based on a pre-set correspondence between data types and offline tables, wherein the offline table is a database for offline storage of real-time data, the correspondence comprises that cold data corresponds to an offline table in a data warehouse, warm data corresponds to an offline table in a data lake, and hot data corresponds to an offline table in a query engine, the data warehouse is an integrated database that can reflect historical changes of data, the query engine is an open source distributed query engine integrating search, analysis and data storage, and the data lake is a data storage library or system that stores data in a raw format, and the target offline table comprises a plurality of data partitions, each data partition corresponding to a corresponding time period; acquiring a current data write mode and an effective write time of the real-time data, wherein the data write mode comprises a system time write mode, a strict event time write mode and a supplementary event time write mode, and the effective write time of the real-time data is a generation time of the real-time data and / or a reception time of the real-time data corresponding to the data write mode; in a case where the current data write mode is the supplementary event time write mode, calculating a time difference between the generation time of the real-time data and a current time, in a case where the time difference is greater than a preset time threshold, determining a time period in which the reception time of the real-time data is located, to obtain a target time period, and taking a data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data, and in a case where the time difference is not greater than the preset time threshold, determining a time period in which the generation time of the real-time data is located, to obtain a target time period, and taking a data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data; in a case where the current data write mode is the system time write mode, the effective write time of the real-time data is the reception time of the real-time data, a time period in which the reception time of the real-time data is located is determined to obtain a target time period, and a data partition corresponding to the target time period in the target offline table is taken as a target data partition corresponding to the real-time data; in a case where the current data write mode is the strict event time write mode, the effective write time of the real-time data is the generation time of the real-time data, a time period in which the generation time of the real-time data is located is determined to obtain a target time period, and a data partition corresponding to the target time period in the target offline table is taken as a target data partition corresponding to the real-time data; storing the real-time data into the target data partition.

2. The method of claim 1, wherein, The method further comprises: When the target data partition corresponding to the effective write time instant does not exist in each data partition of the target offline table, the target data partition corresponding to the effective write time instant is divided in the target offline table.

3. The method of claim 1, wherein, The method further comprises: monitoring the target data partition, and after storing the real-time data into the target data partition, marking the target data partition as storage completion and issuing prompt information indicating storage completion.

4. The method of claim 1, wherein, After storing the real-time data into the target data partition, the method further comprises: determining whether offline data exists in the target data partition; if yes, determining whether the real-time data is consistent with the offline data, and if yes, determining that the real-time data is written completely; if the offline data does not exist in the target data partition or the real-time data is not consistent with the offline data, determining that the real-time data is not written completely and issuing prompt information indicating that the real-time data is not written completely.

5. A real-time data offline storage apparatus characterized by comprising: The device comprises: a data acquisition module configured to acquire real-time data to be stored; a data type determination module configured to determine a data type of the real-time data according to attribute parameters of the real-time data, to obtain a target data type, wherein the attribute parameters of the real-time data include at least one of access frequency, data timeliness and access reflection time of the real-time data, and the data type includes cold data, hot data and warm data; an offline table determination module configured to determine a target offline table corresponding to the target data type based on a pre-set correspondence between data types and offline tables, wherein the correspondence includes that cold data corresponds to an offline table in a data warehouse, warm data corresponds to an offline table in a data lake, and hot data corresponds to an offline table in a query engine, the data warehouse is an integrated database that can reflect historical changes of data, the query engine is an open source distributed query engine integrating search, analysis and data storage, the data lake is a data storage library or system that stores data in a raw format, and the target offline table includes a plurality of data partitions, each data partition corresponding to a corresponding time period; a data storage module comprising: a data acquisition sub-module configured to acquire a current data write mode and an effective write time instant of the real-time data, wherein the data write mode includes a system time write mode, a strict event time write mode and a supplementary event time write mode, and the effective write time instant of the real-time data is a generation time instant of the real-time data and / or a reception time instant of the real-time data corresponding to the data write mode; The data partition determination submodule is configured to, in a case where the current data writing mode is the event time writing mode, calculate a time difference between a generation time of the real-time data and a current time; in a case where the time difference is greater than a preset time threshold, determine a time period in which a receiving time of the real-time data is located, to obtain a target time period; and take a data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data; in a case where the time difference is not greater than the preset time threshold, determine a time period in which the generation time of the real-time data is located, to obtain a target time period; and take a data partition corresponding to the target time period in the target offline table as a target data partition corresponding to the real-time data. In a case where the current data writing mode is the system time writing mode, the effective writing time of the real-time data is the receiving time of the real-time data; a time period in which the receiving time of the real-time data is located is determined, to obtain a target time period; and a data partition corresponding to the target time period in the target offline table is taken as a target data partition corresponding to the real-time data. In a case where the current data writing mode is the strict event time writing mode, the effective writing time of the real-time data is the generation time of the real-time data; a time period in which the generation time of the real-time data is located is determined, to obtain a target time period; and a data partition corresponding to the target time period in the target offline table is taken as a target data partition corresponding to the real-time data. The data storage submodule is configured to store the real-time data into the target data partition.

6. The apparatus of claim 5, wherein, The device further comprises: The data partition division submodule is configured to, in a case where the target data partition corresponding to the effective writing time does not exist in each data partition of the target offline table, divide the target data partition corresponding to the effective writing time in the target offline table.

7. The apparatus of claim 5, wherein, The device further comprises: The data partition monitoring submodule is configured to monitor the target data partition, and after the real-time data is stored into the target data partition, mark the target data partition as storage completed and send prompt information indicating storage completion.

8. The apparatus of claim 5, wherein, The device further comprises: The data judgment submodule is configured to: judge whether offline data exists in the target data partition; if yes, judge whether the real-time data is consistent with the offline data, and if yes, determine that the real-time data is written completely; if no, or if the real-time data is not consistent with the offline data, determine that the real-time data is not written completely and send prompt information indicating incomplete writing.

9. An electronic device, comprising: The device comprises a processor, a communication interface, a writer and a communication bus, wherein the processor, the communication interface and the writer complete mutual communication through the communication bus; The writer is configured to store a computer program; The processor is configured to execute the program stored on the writer, to implement the method steps in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has the computer program written therein, and the computer program is executed by the processor to implement the method steps in any one of claims 1-4.

Images