Data delay processing method and device, electronic equipment, storage medium and product

Abstract

The invention provides a data delay processing method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: determining the event occurrence time of to-be-processed log data; based on the event occurrence time, storing a file directory of log data generated in a target time period in a temporary storage layer of a cluster; wherein the file directory is used for indicating a storage path of the log data in the cluster; and based on a preset rule and the file directory in the temporary storage layer, identifying a delay arrival log, and carrying out synchronization processing on the delay arrival log in each data layer of the offline data warehouse.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a data delay processing method, device, electronic device, storage medium and product. Background Art

[0002] As the scale of data becomes larger and larger, offline data warehouses are gradually being widely used to meet the needs of more efficient, accurate and comprehensive data display and utilization. The main function of offline data warehouses is to periodically clean, calculate and implement real-time data, provide data support for further data analysis, and assist business decision-making.

[0003] In the related technology, the data collection tool transmits the collected data to the HFDS cluster in real time, and then processes the data in the HFDS cluster through the offline data warehouse by means of scheduled calculation. From the generation of data to the transmission to the HDFS cluster, it will go through complex network IO and calculation processes, which may cause data to arrive at the HDFS cluster late. Since the calculation process of the offline data warehouse is scheduled calculation, delayed files may miss the calculation cycle and cannot participate in the calculation process, further resulting in data loss, affecting the accuracy and integrity of application layer data. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a data delay processing method, device, electronic device, storage medium and product.

[0005] In a first aspect, an embodiment of the present disclosure provides a data delay processing method, including:

[0006] Determine the event occurrence time of the log data to be processed;

[0007] Based on the time when the event occurred, a file directory of the log data generated within the target time period is stored in the temporary storage layer of the cluster; wherein the file directory is used to indicate the storage path of the log data in the cluster;

[0008] Based on preset rules and the file directory in the temporary storage layer, the delayed arrival log is identified, and the delayed arrival log is synchronously processed in each data layer of the offline data warehouse.

[0009] In an optional implementation manner, determining the event occurrence time of the log data to be processed includes:

[0010] After the log data is collected from various data sources, the timestamp in the log data is extracted through an interceptor to obtain the time when the event occurred.

[0011] In an optional implementation, based on the event occurrence time, a file directory of log data generated within a target time period is stored in a temporary storage layer of the cluster, including:

[0012] Determine a storage path of the log data in the cluster based on the event occurrence time;

[0013] Record the storage path in the cluster file directory;

[0014] At a preset time interval, the file directory of the log data generated within the target time period in the cluster file directory is synchronized to the data table of the temporary storage layer.

[0015] In an optional implementation, based on preset rules and the file directory in the temporary storage layer, identifying the delayed arrival log includes:

[0016] The delayed arrival log is identified based on the data volume of the log data corresponding to the file directory in the temporary storage layer or the event occurrence time of the log data.

[0017] In an optional implementation, identifying the delayed arrival log based on the amount of log data corresponding to the file directory in the temporary storage layer includes:

[0018] Determine a first data volume of the corresponding log data based on the file directory in the temporary storage layer, and determine a second data volume of the log data in the original data layer of the offline data warehouse;

[0019] When the first data amount and the second data amount are inconsistent, it is determined that the delayed arrival log exists in the temporary storage layer.

[0020] In an optional implementation, identifying the delayed arrival log based on the event occurrence time of the log data corresponding to the file directory in the temporary storage layer includes:

[0021] Identifying a target file directory in the file directory of the temporary storage layer; wherein the target file directory is a file directory of log data generated before the target time period;

[0022] When the target file directory is identified, the log corresponding to the target file directory is determined to be the delayed arrival log.

[0023] In a second aspect, an embodiment of the present disclosure provides a data delay processing device, including:

[0024] A determination unit, used to determine the event occurrence time of the log data to be processed;

[0025] A storage unit, used to store a file directory of log data generated within a target time period in a temporary storage layer of the cluster based on the time when the event occurred; wherein the file directory is used to indicate a storage path of the log data in the cluster;

[0026] The identification and synchronization unit is used to identify the delayed arrival log based on preset rules and the file directory in the temporary storage layer, and synchronize the delayed arrival log in each data layer of the offline data warehouse.

[0027] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0028] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0029] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0030] In an embodiment of the present disclosure, first, determine the event occurrence time of the log data to be processed; based on the event occurrence time, store the file directory of the log data generated within the target time period in the temporary storage layer of the cluster; wherein the file directory is used to indicate the storage path of the log data in the cluster; based on preset rules and the file directory in the temporary storage layer, identify the delayed arrival logs, and synchronize the delayed arrival logs in each data layer of the offline data warehouse.

[0031] In the above implementation mode, by determining the event occurrence time of the log data to be processed, and then storing the file directory of the log data generated within the target time period in the temporary storage layer of the cluster according to the event occurrence time, the delayed arrival log can be effectively identified according to the event occurrence time, thereby avoiding the situation where data is missing in the offline data warehouse due to data delay, effectively ensuring the timeliness of data output, and eliminating the need to regularly update and calculate historical data within a certain range, saving storage and computing resources.

[0032] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0034] Figure 1 A flow chart of a data delay processing method provided by an embodiment of the present disclosure is shown;

[0035] Figure 2 A schematic diagram showing a processing process of an offline data warehouse in the prior art is shown;

[0036] Figure 3 A schematic diagram showing a processing process of an offline data warehouse provided by an embodiment of the present disclosure is shown;

[0037] Figure 4 A schematic diagram of synchronization of a file directory in an HDFS cluster and a temporary storage layer provided by an embodiment of the present disclosure is shown;

[0038] Figure 5 A schematic diagram showing another offline data warehouse processing process provided by an embodiment of the present disclosure is shown;

[0039] Figure 6 A flowchart of processing delayed arrival logs within a target time period in a temporary storage layer and an original data layer provided by an embodiment of the present disclosure is shown;

[0040] Figure 7 A flowchart of processing a delayed arrival log before a target time period in a temporary storage layer and an original data layer provided by an embodiment of the present disclosure is shown;

[0041] Figure 8 A schematic diagram showing another offline data warehouse processing process provided by an embodiment of the present disclosure is shown;

[0042] Fig. 9 A schematic diagram of a data delay processing device provided by an embodiment of the present disclosure is shown;

[0043] Fig.10 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0046] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0047] Research has found that as the scale of data becomes larger and larger, offline data warehouses are gradually being widely used to meet the needs of more efficient, accurate and comprehensive data display and utilization. The main function of offline data warehouses is to periodically clean, calculate and implement real-time data, provide data support for further data analysis, and assist business decision-making.

[0048] In the related technology, the data collection tool transmits the collected data to the HFDS cluster in real time, and then processes the data in the HFDS cluster through the offline data warehouse by means of scheduled calculation. From the generation of data to the transmission to the HDFS cluster, it will go through complex network IO and calculation processes, which may cause data to arrive at the HDFS cluster late. Since the calculation process of the offline data warehouse is scheduled calculation, delayed files may miss the calculation cycle and cannot participate in the calculation process, further resulting in data loss, affecting the accuracy and integrity of application layer data.

[0049] Based on the above research, the present disclosure provides a data delay processing method, device, electronic device, storage medium and product. In the embodiment of the present disclosure, first, determine the event occurrence time of the log data to be processed; based on the event occurrence time, store the file directory of the log data generated in the target time period in the temporary storage layer of the cluster; wherein the file directory is used to indicate the storage path of the log data in the cluster; based on the preset rules and the file directory in the temporary storage layer, identify the delayed arrival log, and synchronize the delayed arrival log in each data layer of the offline data warehouse.

[0050] In the above implementation mode, by determining the event occurrence time of the log data to be processed, and then storing the file directory of the log data generated within the target time period in the temporary storage layer of the cluster according to the event occurrence time, the delayed arrival log can be effectively identified according to the event occurrence time, thereby avoiding the situation where data is missing in the offline data warehouse due to data delay, effectively ensuring the timeliness of data output, and eliminating the need to regularly update and calculate historical data within a certain range, saving storage and computing resources.

[0051] To facilitate understanding of this embodiment, a data delay processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the data delay processing method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities. In some possible implementations, the data delay processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0052] See also Figure 1 FIG. 1 is a flowchart of a data delay processing method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S103, wherein:

[0053] S101: Determine the event occurrence time of the log data to be processed.

[0054] like Figure 2 The following is a method for synchronizing user behavior logs and a solution for offline data warehouses. Figure 2 As shown, first, the data collection tool Flume is used to collect data stored in various data sources (for example, databases, logs, message queues, etc.) and store them in the HDFS cluster.

[0055] Based on the data of the cluster on HDFS, the original data layer ODS (Operation Data Store) is constructed to store the original data; then the original data of the original data layer ODS is cleaned and completed and stored in the detailed data layer DWD (DataWarehouse Detail); then the detailed data layer DWD layer data is periodically cleaned and summarized to generate a light summary layer, which is stored in the summary data layer DWS (Data Warehouse Summary); finally, according to business needs, the wide table data is periodically associated and calculated to generate business indicator data, which is stored in the data application layer ADS (Application Data Service). Based on the data of the ADS layer, data applications such as OLAP analysis, BI analysis, report display, and data mining are performed.

[0056] Here, an interceptor can be configured in the data collection tool Flume, so that the data can be placed in the corresponding file directory according to the event occurrence time when entering the offline data warehouse. Therefore, the event occurrence time of the log data to be processed can be determined by the Flume tool configured with the interceptor.

[0057] S102: Based on the event occurrence time, a file directory of the log data generated within the target time period is stored in the temporary storage layer of the cluster; wherein the file directory is used to indicate a storage path of the log data in the cluster.

[0058] In the embodiment of the present disclosure, a temporary storage layer (Stage) can be added with a preset time range N. The temporary storage layer stores log data within the past N days; wherein N days can be understood as the target time period described above. In addition, the target time period can also be set to any time period such as N hours, N months, etc. There is no specific limitation here, which is subject to what can be implemented.

[0059] Here, the event occurrence time of the log data can be determined, and then the log data is stored in the HDFS cluster according to the event occurrence time, and the file directory of the log data (that is, the HDFS file directory) is recorded in the HDFS cluster; wherein the file directory is used to indicate the storage path of the log data in the HDFS cluster. Afterwards, the file directory of the log data generated in the target time period in the HDFS cluster can be synchronized to the temporary storage layer of the cluster according to the preset synchronization cycle.

[0060] S103: Based on preset rules and the file directory in the temporary storage layer, identify the delayed arrival log, and synchronize the delayed arrival log in each data layer of the offline data warehouse.

[0061] In the embodiment of the present disclosure, log data that is delayed in arriving at the HDFS cluster, that is, delayed arrival logs, may be identified according to preset rules and the file directory stored in the temporary storage layer.

[0062] Here, the preset rule is associated with the data volume of the log data corresponding to the file directory in the temporary storage layer and / or the event occurrence time of the log data.

[0063] After the delayed arrival log is identified, the delayed arrival log can be synchronized at each data layer of the offline data warehouse. Here, the delayed arrival log can be synchronized to the first data layer of the offline data warehouse, for example, the original data layer ODS; then, starting from the original data layer ODS of the offline data warehouse, the synchronization is performed in the DWD layer, DWS layer, and ADS layer in turn.

[0064] Here, the matching log synchronization processing mode can be determined according to the data processing tasks corresponding to the offline data warehouse. The log synchronization processing modes include enhanced processing mode and full processing mode.

[0065] For example, if the matching log synchronization processing method is the full processing method, then after the delayed arrival log is synchronized to the first data layer of the offline data warehouse, all the data in the ODS layer can be cleaned and completed and stored in the DWD layer. Then, the DWD layer, DWS layer and ADS layer perform corresponding processing on the full data in turn.

[0066] For example, if the matching log synchronization processing method is incremental processing, then after synchronizing the delayed arrival log to the first data layer of the offline data warehouse, the delayed arrival log in the ODS layer can be cleaned and completed and stored in the DWD layer. Then, the DWD layer, DWS layer and ADS layer perform corresponding processing on the delayed arrival log in turn.

[0067] From the above description, we can see that Figure 2 As shown in the figure, in the offline data warehouse, the process from data generation and collection to transmission to the HFDS cluster is real-time transmission, and the calculation from the OSD layer to the ADS layer is scheduled calculation. From data generation to transmission to the HDFS cluster, it will go through complex network IO and calculation processes. Due to network jitter, machine downtime, program errors and other reasons, data may be delayed in reaching the HDFS cluster. Since the calculation process from the OSD layer to the ADS layer of the offline data warehouse is scheduled calculation, delayed files may miss the calculation cycle and cannot participate in the calculation process, further resulting in data loss, affecting the accuracy and integrity of application layer data.

[0068] There is no method in the related technology that can automatically identify data delays and perform incremental calculations on delayed data to supplement the offline data warehouse. Most of them postpone the calculation time to allow the delayed data to participate in the scheduled calculation as much as possible; or regularly recalculate and update a certain range of historical data to make the data as complete as possible. However, the delayed calculation time cannot be accurately grasped; and regularly updating a certain range of historical data will occupy a large amount of computing resources and storage resources, causing certain pressure on the cluster server.

[0069] Based on this, in the embodiments of the present disclosure, by determining the event occurrence time of the log data to be processed, and then storing the file directory of the log data generated within the target time period in the temporary storage layer of the cluster according to the event occurrence time, the delayed arrival logs can be effectively identified according to the event occurrence time, thereby avoiding the situation where data is missing in the offline data warehouse due to data delays, effectively ensuring the timeliness of data output, and eliminating the need to regularly update and calculate historical data within a certain range, thereby saving storage and computing resources.

[0070] The above steps will be described in detail below in conjunction with specific implementation methods.

[0071] In an optional implementation, the above step S101 determines the event occurrence time of the log data to be processed, and specifically includes the following steps:

[0072] After the log data is collected from various data sources, the timestamp in the log data is extracted through an interceptor to obtain the time when the event occurred.

[0073] In the embodiment of the present disclosure, when using the data collection tool FLume to consume the data in KafKa (data source) and putting it into the HDFS cluster, an interceptor can be added to the data collection tool to extract the event timestamp in the log data as the Header value of the Event (i.e., the time when the event occurred), so that the log data can fall into the corresponding data partition directory (i.e., the file directory) according to the event timestamp.

[0074] Flume uses the architecture of kafka source + file channel + hdfs sink to collect data. The key steps are as follows:

[0075] (1) Since the directory path where the log data falls into the HDFS cluster is determined by the Event Head, an interceptor can be added to the Kafka source. For each log data, the event timestamp in the log data can be extracted and added to the Event Head.

[0076] (2) Set up HDFS sink, the output path is as follows:

[0077] a1.sinks.pull_l1_sink.hdfs.path= / warehouse / stg / topic_log / day=%Y-%m-

[0078] %d / hour=%H

[0079] In this way, each piece of log data can be uploaded to the HDFS cluster after being processed by the interceptor. The "%Y-%m-%d" and "%H" in the above path will be converted into the date and hour corresponding to the event timestamp of the log data. The log data will fall into the file directory corresponding to the corresponding date / hour according to the time when the event occurred.

[0080] In the embodiment of the present disclosure, after determining the event occurrence time of the log data to be processed, the file directory of the log data generated within the target time period can be stored in the temporary storage layer of the cluster based on the event occurrence time, which specifically includes the following steps:

[0081] Step S11: determining a storage path of the log data in the cluster based on the event occurrence time;

[0082] Step S12: Record the storage path in the cluster file directory;

[0083] Step S13: Synchronize the file directory of the log data generated within the target time period in the cluster file directory to the data table of the temporary storage layer at a preset time interval.

[0084] In the embodiments of the present disclosure, Figure 3 As shown, a temporary storage layer called Stage can be established in the offline data warehouse, and then the file directory of the log data generated within the target time period N can be stored in the temporary storage layer. N can be set to any value that meets the requirements, such as 7 days, 30 days, etc., and the specific value of N is not specifically limited here. For example, the file directory of the log data generated within N days can be recorded in the temporary storage layer.

[0085] At the same time, as a temporary cache layer, the Stage needs to regularly delete data before the target time period (for example, data from N days ago) to prevent the data from occupying too much storage space. That is to say, if there is no data delay, the HDFS file directory corresponding to the Stage layer only has date / hour directories within the target time period (for example, N days).

[0086] Here, after determining the storage path of the log data in the cluster in the manner described in the above implementation, the log data can be stored in the corresponding file according to the storage path, and the storage path can be recorded in the cluster file directory. The storage path is used to indicate the event occurrence time of the log data.

[0087] After the log data falls into the HDFS cluster, the file directory of the log data generated in the target time period in the cluster file directory can be synchronized to the data table of the temporary storage layer, that is, the Hive external table, in response to the data synchronization operation. Here, a Hive external table is pre-set for the temporary storage layer, through which the storage path of the log data generated in the target time period in the cluster can be recorded.

[0088] In the disclosed embodiment, a data synchronization task may be pre-set, wherein the data synchronization task is triggered to execute at a certain interval, that is, the data synchronization task is triggered to execute at a preset time interval. After the data synchronization task is triggered to execute, the file directory of the log data generated in the target time period in the cluster file directory may be synchronized to the Hive external table of the temporary storage layer. For example, Figure 4 As shown, you can record the storage path of log data in the cluster file directory, and then synchronize the corresponding file directory to the temporary storage Hive external table by executing the data synchronization task regularly. For example, you can synchronize the file directory of log data generated between October 1, 2023 and October 7, 2023 with the Hive external table. Among them, October 1, 2023 to October 7 is the target time period described above. Figure 4 As shown in the figure, each "date / hour" in the temporarily stored Hive external table can be understood as a partition.

[0089] Here, it should be noted that if after synchronizing the file directory of log data generated between October 1 and October 7, 2023 in the cluster file directory to the Hive external table, at least one file directory of log data generated on October 2, 2023 is newly stored in the cluster file directory, then when the next data synchronization task is triggered, the file directory of the log data generated on October 2, 2023 can be synchronized to the partition generated in the Hive external table of the temporary data layer. If the log data of September 30, 2023 arrives, a new partition can be generated for the log data in the cluster, and the log data can be stored in the new partition.

[0090] In specific implementation, after triggering the data synchronization task, it can be determined whether the file directory (for example, date / hour directory) corresponding to the target time period exists in the HDFS file directory; if it is determined to exist, but there is no partition corresponding to the target time period in the Hive external table, then the mount is performed to increase the correspondence between the data partition and the data file, that is, the file directory corresponding to the target time period in the HDFS file directory is synchronized to the data table of the temporary storage layer. At this time, the corresponding HDFS file data content (that is, the corresponding log data) can be queried by querying the Hive external table.

[0091] After that, data warehouse modeling can be performed, such as Figure 5 As shown in the figure, the data of the Stage layer is synchronized to the ODS layer in turn to store the original data for a long time; then, after the data of the ODS layer is cleaned and completed, a fine-grained detail layer fact table is constructed and stored in the DWD layer; then, the DWD layer data is periodically cleaned and summarized to generate a light summary layer, which is stored in the DWS layer; finally, according to business needs, the wide table data is periodically associated and calculated to generate business indicator data, which is stored in the ADS layer.

[0092] In an optional implementation, the above steps identify delayed arrival logs based on preset rules and the file directory in the temporary storage layer, specifically including the following steps:

[0093] The delayed arrival log is identified based on the data volume of the log data corresponding to the file directory in the temporary storage layer or the event occurrence time of the log data.

[0094] In the embodiment of the present disclosure, since the Stage layer continuously receives real-time data, and the ODS layer performs scheduled calculations, if the data arrives at the Stage layer with a delay and the delay period does not exceed N days, the amount of data at the Stage layer is greater than that at the ODS layer. At the same time, in the technical solution of the present disclosure, the Stage layer only stores data within N days (target time period), and the HDFS file directory before N days will be deleted regularly. If the data delay time exceeds N days, the delayed data will fall into the newly generated partition, and only the delayed data will be stored.

[0095] Based on this, delayed arrival logs can be identified based on the data volume of the log data corresponding to the file directory in the temporary storage layer or the event occurrence time of the log data.

[0096] In the embodiment of the present disclosure, the above steps identify the delayed arrival log based on the data volume of the log data corresponding to the file directory in the temporary storage layer, and specifically include the following steps:

[0097] Step S21: determining a first data volume of corresponding log data based on the file directory in the temporary storage layer, and determining a second data volume of log data in the original data layer of the offline data warehouse;

[0098] Step S22: when the first data volume and the second data volume are inconsistent, determining that the delayed arrival log exists in the temporary storage layer.

[0099] In the disclosed embodiment, the data volume (i.e., the first data volume and the second data volume) of the log data corresponding to the Stage layer and the ODS layer at the date / hour granularity within the past N days (target time period) can be calculated regularly. If it is determined that the first data volume and the second data volume are inconsistent, it means that there is a data delay within N days in the Stage layer.

[0100] like Figure 6 As shown in the figure, assuming that the log data in the target time period arrives late, the file directory of the log data can be synchronized to the temporary storage layer by executing the data synchronization task; and fall into the cluster file directory generated in the temporary storage layer (i.e. Figure 6 ), where if the file directory of the delayed log data misses the scheduling cycle, the data synchronized by the ODS layer will not include the delayed log data.

[0101] For example, the time range (target time period) of the temporary storage layer is set to 7 days. When the data volume of the Stage layer and the ODS layer is inconsistent from 7:00 on October 2, 2023 to 6:00 on October 1, 2023, it means that the data of 6:00 on October 1, 2023 has been delayed to 7:00 on October 2, 2023. At this time, if the data before 7:00 on October 2, 2023 has been synchronized from the ODS layer to the ADS layer before the data of 6:00 on October 1, 2023 arrives, then there is data missing in the day=2023-10-01 / hour=06 partition from the ODS layer to the ADS layer.

[0102] At this time, the delayed arrival log may be determined based on the different log data between the Stage layer and the ODS layer.

[0103] In the embodiment of the present disclosure, the above steps identify the delayed arrival log based on the event occurrence time of the log data corresponding to the file directory in the temporary storage layer, and specifically include the following steps:

[0104] Step S31: identifying a target file directory in the file directory of the temporary storage layer; wherein the target file directory is a file directory of log data generated before the target time period;

[0105] Step S32: when the target file directory is identified, determining that the log corresponding to the target file directory is the delayed arrival log.

[0106] In the disclosed embodiment, it is regularly determined whether the HDFS file directory corresponding to the Stage layer has a date / hour directory N days ago (i.e., the target file directory), that is, a file directory of log data generated before the target time period. If the target file directory exists, it means that the log data N days (target time period) before has been delayed.

[0107] like Figure 7 As shown, assuming that the log data before the target time period arrives late, the file directory of the log data can be synchronized to the temporary storage layer by executing the data synchronization task; and fall into the newly generated cluster file directory in the temporary storage layer. If the file directory of the delayed log data misses the scheduling cycle, the data synchronized by the ODS layer will not include the delayed log data.

[0108] For example, suppose that the time range of temporary storage is set to 7 days. At 7:00 on October 8, 2023, a new file directory at 6:00 on October 1, 2023 was identified in the Stage temporary storage layer, which means that the data at 6:00 on October 1, 2023 was delayed at 7:00 on October 8, 2023. At this time, if the data before October 7, 2023 has been synchronized from the ODS layer to the ADS layer before the data at 6:00 on October 1, 2023 arrives, then there is data missing in the day=2023-10-01 / hour=06 partition from the ODS layer to the ADS layer. At this time, it can be determined that the log data corresponding to the target file directory is a delayed arrival log.

[0109] According to the above-described method, all delayed data can be identified. At this time, an alarm notification can be sent to the user, and the program can be triggered to perform incremental synchronization and calculation of the delayed data; wherein, the alarm notification can be sent to the user by email, SMS or phone call.

[0110] like Figure 8 As shown, in the embodiment of the present disclosure, after the delayed arrival log is identified, the partition where the delayed arrival log is located can be identified, and an automated program can be triggered to recalculate and update the historical data partition from the ODS layer to the ADS layer, thereby realizing incremental calculation of the delayed arrival log.

[0111] In the disclosed embodiment, the above-mentioned preset rules can capture the "date / hour" partition where the delayed arrival log is located, so as to call the automated program to recalculate and update all historical partition data of each data layer (ODS, DWD, DWS, ADS) of the offline data warehouse under the "date / hour" partition, synchronize incremental data and write it to the offline data warehouse to ensure the integrity and accuracy of data at each layer.

[0112] In specific implementation, starting from the original data layer ODS, the log data under the "date / hour" partition can be processed in sequence, and according to the processing results of the original data layer ODS, the log data under the "date / hour" partition in DWD, DWS, and ADS can be processed in sequence.

[0113] In addition, the log data under the "date / hour" partition can be supplemented to all historical partition data of ODS, DWD, DWS, and ADS in sequence, and then, starting from ODS, the supplemented log data can be recalculated in sequence.

[0114] From the above description of the implementation process, it can be seen that after configuring data delay arrival monitoring and automated program calculation, the offline data warehouse can automatically identify data delays, capture the partition where the delayed data is located, and self-inspire incremental data synchronization and calculation to ensure data accuracy and 100% integrity.

[0115] Combine the following Figure 3 The above process is introduced.

[0116] like Figure 3 As shown, the technical solution disclosed in the present invention adds an interceptor in the data collection tool Flume, through which the event occurrence time of the log data can be extracted and mapped into the corresponding cluster file directory, wherein the event occurrence time can make the log data fall into the corresponding "date / hour" directory according to the event occurrence time when entering the offline data warehouse.

[0117] like Figure 3 As shown, the technical solution disclosed in the present invention further adds a temporary storage layer (Stage) to realize the creation of Hive external tables and dynamic mounting of partitions. The temporary storage layer stores data within the past N days.

[0118] like Figure 3 As shown, delayed data monitoring is added to identify delayed arrival data and trigger SMS, phone, and email alarms. Among them, the monitoring logic of the delayed data monitoring can identify all delayed arrival logs and issue alarms.

[0119] like Figure 3As shown, the incremental calculation and synchronization link is also added to realize the incremental calculation and synchronization of the delayed arrival log, which is synchronized to each data layer of the offline data warehouse in turn. Among them, by performing incremental calculation and synchronization on the delayed arrival data, the integrity of the data can be guaranteed, and there is no need to consume too much storage and calculation, which can effectively solve the above problems. In addition, the technical solution disclosed in the present invention can effectively identify data with too long a delay time (for example, it can identify indefinite data delay arrival), which can effectively ensure the accuracy and 100% integrity of the data.

[0120] It can be seen from the above description that the technical solution disclosed in this disclosure specifically realizes the following functions:

[0121] 1) When Kafka transmits data to HDFS, add an interceptor to Flume to extract the event occurrence time in the log data and add it to the Head of the Event, so that the log data falls into the corresponding HDFS date / hour directory according to the event occurrence time;

[0122] 2) Add the Stage layer as a temporary storage layer to create data tables and dynamically mount partitions. Store data within the target time period and regularly delete historical data before the target time period;

[0123] 3) Identify delayed arrival logs: By comparing whether there is a difference in the data volume between the Stage layer and the ODS layer at the date / hour granularity, and determining whether there is a date / hour directory file before the target time period in the HDFS path corresponding to the Stage layer, these two rules can be used to identify all delayed arrival log data and capture the date / hour partition where the delayed arrival log data is located, and inform the R&D personnel through email, SMS, or phone calls;

[0124] 4) After identifying the delayed log data, the date / hour partition where the delayed log data is located can be captured, and the automated program can be called to recalculate and update all historical partition data of each layer of the offline data warehouse (ODS, DWD, DWS, ADS) under the date / hour partition, synchronize and write incremental data, and ensure the integrity and accuracy of data at each layer of the data warehouse.

[0125] Through the above points, the disclosed technical solution can effectively identify and process all delayed log data, achieve data integrity, and thus avoid data loss due to data delay. The disclosed technical solution does not need to postpone the task calculation time, and can effectively ensure the timeliness of data output. After the disclosed technical solution identifies the delayed log data, it only needs to perform automated incremental storage and calculation on the delayed log data, without the need to regularly update and calculate a certain range of historical data, saving storage and computing resources.

[0126] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0127] Based on the same inventive concept, a data delay processing device corresponding to the data delay processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data delay processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0128] Reference Fig. 9 , which is a schematic diagram of a data delay processing device provided by an embodiment of the present disclosure, the device comprises: a determination unit 10, a storage unit 20 and an identification synchronization unit 30; wherein,

[0129] A determination unit 10, used to determine the event occurrence time of the log data to be processed;

[0130] A storage unit 20, configured to store a file directory of log data generated within a target time period in a temporary storage layer of the cluster based on the time at which the event occurs; wherein the file directory is used to indicate a storage path of the log data in the cluster;

[0131] The identification and synchronization unit 30 is used to identify delayed arrival logs based on preset rules and the file directory in the temporary storage layer, and synchronize the delayed arrival logs in each data layer of the offline data warehouse.

[0132] In the above implementation mode, by determining the event occurrence time of the log data to be processed, and then storing the file directory of the log data generated within the target time period in the temporary storage layer of the cluster according to the event occurrence time, the delayed arrival log can be effectively identified according to the event occurrence time, thereby avoiding the situation where data is missing in the offline data warehouse due to data delay, effectively ensuring the timeliness of data output, and eliminating the need to regularly update and calculate historical data within a certain range, saving storage and computing resources.

[0133] In a possible implementation manner, the determining unit is further configured to:

[0134] After the log data is collected from various data sources, the timestamp in the log data is extracted through an interceptor to obtain the time when the event occurred.

[0135] In a possible implementation manner, the storage unit is further used for:

[0136] Determine a storage path of the log data in the cluster based on the event occurrence time;

[0137] Record the storage path in the cluster file directory;

[0138] At a preset time interval, the file directory of the log data generated within the target time period in the cluster file directory is synchronized to the data table of the temporary storage layer.

[0139] In a possible implementation manner, the identification synchronization unit is further used for:

[0140] The delayed arrival log is identified based on the data volume of the log data corresponding to the file directory in the temporary storage layer or the event occurrence time of the log data.

[0141] In a possible implementation manner, the identification synchronization unit is further used for:

[0142] Determine a first data volume of the corresponding log data based on the file directory in the temporary storage layer, and determine a second data volume of the log data in the original data layer of the offline data warehouse;

[0143] When the first data amount and the second data amount are inconsistent, it is determined that the delayed arrival log exists in the temporary storage layer.

[0144] In a possible implementation manner, the identification synchronization unit is further used for:

[0145] Identifying a target file directory in the file directory of the temporary storage layer; wherein the target file directory is a file directory of log data generated before the target time period;

[0146] When the target file directory is identified, the log corresponding to the target file directory is determined to be the delayed arrival log.

[0147] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0148] Corresponds to Figure 1 The data delay processing method in the present disclosure also provides an electronic device 1000, such as Fig.10 FIG. 1 is a schematic diagram of the structure of an electronic device 1000 provided in an embodiment of the present disclosure, including:

[0149] Processor 101, memory 102, and bus 103; memory 102 is used to store execution instructions, including internal memory 1021 and external memory 1022; the internal memory 1021 is also called internal memory, which is used to temporarily store the operation data in the processor 101 and the data exchanged with the external memory 1022 such as a hard disk. The processor 101 exchanges data with the external memory 1022 through the internal memory 1021. When the electronic device 1000 is running, the processor 101 communicates with the memory 102 through the bus 103, so that the processor 101 executes the following instructions:

[0150] Determine the event occurrence time of the log data to be processed;

[0151] Based on the time when the event occurred, a file directory of the log data generated within the target time period is stored in the temporary storage layer of the cluster; wherein the file directory is used to indicate the storage path of the log data in the cluster;

[0152] Based on preset rules and the file directory in the temporary storage layer, the delayed arrival log is identified, and the delayed arrival log is synchronously processed in each data layer of the offline data warehouse.

[0153] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data delay processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0154] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the data delay processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0155] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0156] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0160] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data delay processing method, characterized in that: include: Determine the event occurrence time of the log data to be processed; Based on the time when the event occurred, a file directory of the log data generated within the target time period is stored in the temporary storage layer of the cluster; wherein the file directory is used to indicate the storage path of the log data in the cluster; Based on preset rules and the file directory in the temporary storage layer, the delayed arrival log is identified, and the delayed arrival log is synchronously processed in each data layer of the offline data warehouse.

2. The method according to claim 1, characterized in that Determining the event occurrence time of the log data to be processed includes: After the log data is collected from various data sources, the timestamp in the log data is extracted through an interceptor to obtain the time when the event occurred.

3. The method according to claim 1, characterized in that Based on the event occurrence time, the file directory of the log data generated within the target time period is stored in the temporary storage layer of the cluster, including: Determine a storage path of the log data in the cluster based on the event occurrence time; Record the storage path in the cluster file directory; At a preset time interval, the file directory of the log data generated within the target time period in the cluster file directory is synchronized to the data table of the temporary storage layer.

4. The method according to claim 1, characterized in that: The identifying the delayed arrival log based on the preset rules and the file directory in the temporary storage layer includes: The delayed arrival log is identified based on the data volume of the log data corresponding to the file directory in the temporary storage layer or the event occurrence time of the log data.

5. The method according to claim 4, characterized in that Based on the data volume of the log data corresponding to the file directory in the temporary storage layer, identifying the delayed arrival log includes: Determine a first data volume of the corresponding log data based on the file directory in the temporary storage layer, and determine a second data volume of the log data in the original data layer of the offline data warehouse; When the first data amount and the second data amount are inconsistent, it is determined that the delayed arrival log exists in the temporary storage layer.

6. The method according to claim 4, characterized in that Based on the event occurrence time of the log data corresponding to the file directory in the temporary storage layer, identifying the delayed arrival log includes: Identifying a target file directory in the file directory of the temporary storage layer; wherein the target file directory is a file directory of log data generated before the target time period; When the target file directory is identified, the log corresponding to the target file directory is determined to be the delayed arrival log.

7. A data delay processing device, characterized in that: include: A determination unit, used to determine the event occurrence time of the log data to be processed; A storage unit, used to store a file directory of log data generated within a target time period in a temporary storage layer of the cluster based on the time when the event occurred; wherein the file directory is used to indicate a storage path of the log data in the cluster; The identification and synchronization unit is used to identify the delayed arrival log based on preset rules and the file directory in the temporary storage layer, and synchronize the delayed arrival log in each data layer of the offline data warehouse.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data delay processing method as described in any one of claims 1 to 6 are performed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data delay processing method according to any one of claims 1 to 6 are executed.

10. A computer program product, characterized in that The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the data delay processing method according to any one of claims 1 to 6.

Images