A data analysis method, device, equipment and storage medium

Abstract

The application provides a data analysis method, device, equipment and storage medium, which can be applied to the field of big data or the field of finance. The method comprises the following steps: obtaining a mapping relationship table of a processing table and a data source table, determining each target data source table in a batch data date corresponding to a target processing table according to the mapping relationship table, obtaining operation time points of each processing step and arrival time points of arriving at the step corresponding to each target data source table respectively, determining whether the target processing table meets the timeliness requirement of a downstream system according to the processing completion time points of the target processing table corresponding to each target data source table, and if not, determining the target processing step that needs to be optimized according to the obtained time points. The application can analyze whether the processing timeliness of the target processing table meets the timeliness requirement, and can analyze the target processing step that needs to be optimized, so that the target processing step can be optimized in the subsequent process to ensure that the downstream system receives the required processing table in time.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data analysis method, apparatus, device and storage medium. Background Art

[0002] The upstream system can send the data it generates to the big data platform in the form of batch data files. After the batch data files arrive at the big data platform, the big data platform can load the batch data files into the database table to form a data source table, and then perform batch data processing on the data source table. After the processing is completed, it is provided to the downstream system for use.

[0003] However, the timeliness of data processing on the big data platform may not meet the timeliness requirements of the downstream system, which will result in the downstream system not receiving the required data in a timely manner. Therefore, there is an urgent need for a method to analyze the timeliness of data processing. Summary of the Invention

[0004] In view of this, the present application provides a data analysis method, apparatus, device, and storage medium for determining whether the processing timeliness of a target processing table meets the timeliness requirements of a downstream system, and if it does not meet the requirements, determining the target processing steps that need to be optimized. The technical solution is as follows:

[0005] A data analysis method, comprising:

[0006] Obtain a mapping relationship table between a processing table and a data source table. The processing table refers to a table obtained by processing at least one corresponding data source table. The data source table has a data date, which is the date when the data in the data source table is generated.

[0007] Determine each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table, wherein the operation time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is the processing table that needs to be analyzed for timeliness, and the arrival step is the step where the target data source table arrives at the big data platform;

[0008] According to the processing completion time point of the target processing table corresponding to each target data source table, determine whether the target processing table meets the timeliness requirements of the downstream system;

[0009] If not, the target processing step that needs to be optimized in each processing step is determined according to the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table.

[0010] Optionally, determining whether the target processing table meets the timeliness requirement of the downstream system according to the processing completion time point of the target processing table corresponding to each target data source table includes:

[0011] Determine the relative processing completion time of the target processing tables corresponding to the target data source tables according to the processing completion time points of the target processing tables corresponding to the target data source tables and the data dates of the target data source tables;

[0012] If the timeliness requirement is a deterministic requirement, then determine whether the relative processing completion time of the target processing table corresponding to each target data source table meets the time point requirement in the deterministic requirement. If so, determine that the target processing table meets the deterministic requirement; otherwise, determine that the target processing table does not meet the deterministic requirement;

[0013] If the timeliness requirement is a probabilistic requirement, then the number of relative processing completion times of the target processing tables corresponding to each target data source table that meet the time point requirements in the probabilistic requirement is counted. If the counted number is less than or equal to the number threshold in the probabilistic requirement, then it is determined that the target processing table meets the probabilistic requirement; otherwise, it is determined that the target processing table does not meet the probabilistic requirement.

[0014] Optionally, the operation time point includes a start time point and a completion time point;

[0015] According to the operation time points of each processing step and the arrival time points of the arrival steps corresponding to each target data source table, the target processing steps that need to be optimized in each processing step are determined, including:

[0016] Determine the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time point and completion time point of each processing step corresponding to each target data source table, and the data date of each target data source table;

[0017] Determine the relative arrival time of the arrival steps corresponding to each target data source table according to the arrival time points of the arrival steps corresponding to each target data source table and the data dates respectively possessed by each target data source table;

[0018] According to the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table, the target processing step that needs to be optimized in each processing step is determined.

[0019] Optionally, determining the relative start time and relative completion time of each processing step corresponding to each target data source table according to the start time point and completion time point of each processing step corresponding to each target data source table, and the data date of each target data source table, includes:

[0020] Select a preset time point in the data date of each target data source table;

[0021] For each target data source table, calculate the difference between the start time of each processing step corresponding to the target data source table and the preset time point in the data date of the target data source table to obtain the relative start time of each processing step corresponding to the target data source table;

[0022] For each target data source table, calculate the difference between the completion time point of each processing step corresponding to the target data source table and the preset time point in the data date of the target data source table to obtain the relative completion time of each processing step corresponding to the target data source table;

[0023] According to the arrival time points of the arrival steps corresponding to the target data source tables and the data dates of the target data source tables, the relative arrival times of the arrival steps corresponding to the target data source tables are determined, including:

[0024] For each target data source table, calculate the arrival time point of the arrival step corresponding to the target data source table, and the difference between it and the preset time point in the data date of the target data source table to obtain the relative arrival time of the arrival step corresponding to the target data source table.

[0025] Optionally, determining a target processing step that needs to be optimized among the processing steps based on the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of each arrival step corresponding to each target data source table, includes:

[0026] Calculate the statistical start time corresponding to each processing step according to the relative start time of each processing step corresponding to each target data source table;

[0027] Calculate the statistical completion time corresponding to each processing step based on the relative completion time of each processing step corresponding to each target data source table;

[0028] Calculate the statistical arrival time of the arrival step based on the relative arrival time of the arrival steps corresponding to each target data source table;

[0029] According to the statistical start time and statistical completion time corresponding to each processing step, and the statistical arrival time of the arrival step, the target processing step that needs to be optimized in each processing step is determined.

[0030] Optionally, the statistical start time includes: earliest relative start time, latest relative start time and average relative start time;

[0031] Statistical completion time includes: earliest relative completion time, latest relative completion time and average relative completion time;

[0032] Statistical arrival time includes: earliest relative arrival time, latest relative arrival time and average relative arrival time.

[0033] Optionally, each processing step includes a loading step and a batch data processing step.

[0034] A data analysis device, comprising:

[0035] A mapping relationship table acquisition module is used to obtain a mapping relationship table between a processing table and a data source table. A processing table refers to a table obtained by processing at least one corresponding data source table. The data source table has a data date, which is the date when the data in the data source table is generated.

[0036] A time point acquisition module is used to determine each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, and obtain the operation time point of each processing step corresponding to each target data source table, as well as the arrival time point of each arrival step corresponding to each target data source table. The operation time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is the processing table that needs to be analyzed for timeliness, and the arrival step is the step where the target data source table arrives at the big data platform.

[0037] A timeliness requirement judgment module is used to determine whether the target processing table meets the timeliness requirement of the downstream system based on the processing completion time point of the target processing table corresponding to each target data source table;

[0038] The module for determining the steps to be optimized is used to determine the target processing steps that need to be optimized in each processing step according to the operation time points of each processing step and the arrival time points of the arrival steps corresponding to each target data source table if the conditions are not met.

[0039] A data analysis device comprising a memory and a processor;

[0040] Memory, used to store programs;

[0041] The processor is used to execute the program to implement each step of any of the above data analysis methods.

[0042] A readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the computer program implements the steps of any of the above-mentioned data analysis methods.

[0043] Through the above technical solution, it can be known that the data analysis method provided by the present application first obtains the mapping relationship table between the processing table and the data source table, and then determines each target data source table within the batch data date corresponding to the target processing table according to the mapping relationship table, obtains the operation time point of each processing step corresponding to each target data source table, and the arrival time point of the arrival step corresponding to each target data source table, and then determines whether the target processing table meets the timeliness requirements of the downstream system according to the processing completion time point of the target processing table corresponding to each target data source table. If not, the target processing step that needs to be optimized in each processing step is determined according to the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table. It can be seen from this that the present application can determine whether the processing timeliness of the target processing table meets the timeliness requirements of the downstream system through data analysis. If it does not meet the requirements, it can further analyze the target processing steps that need to be optimized, so that the target processing steps can be optimized to ensure that the downstream system can receive the required processing table in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0045] Figure 1 A flowchart of a data analysis method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of a data analysis device provided in an embodiment of the present application;

[0047] Figure 3 A hardware structure block diagram of a data analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] The present application provides a data analysis method, apparatus, device and storage medium. The data analysis method, apparatus, device and storage medium provided by the present application are described in detail through the following embodiments.

[0050] It should be noted that the data analysis method, apparatus, device, and storage medium provided by the present invention can be used in the field of big data or finance. The above is only an example and does not limit the application fields of the data analysis method, apparatus, device, and storage medium provided by the present invention.

[0051] It should also be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0052] See also Figure 1 , which shows a flow chart of a data analysis method provided in an embodiment of the present application, and the data analysis method may include:

[0053] Step S101: Obtain a mapping relationship table between a processing table and a data source table.

[0054] Here, the mapping relationship table refers to a table composed of multiple mapping relationships, any mapping relationship refers to the correspondence between a processing table and at least one data source table, and the processing table refers to a table obtained by processing the corresponding at least one data source table.

[0055] The above data source table has a data date, which is the date on which the data in the data source table is generated. The data in the data source table is data generated by the upstream system.

[0056] Specifically, the data in the data source table is data generated by the upstream system (for example, this data can be transfer data, loan data, etc.). Those skilled in the art will understand that the upstream system generates data daily. In the embodiments of this application, the data generated daily by the upstream system is stored in the data source table. To distinguish the data source tables generated on each day, the data source table can be labeled with a data date when the data is stored in the data source table each day. For example, the data source table containing data generated on July 1st has a data date of July 1st.

[0057] Optionally, an embodiment of the present application can generate a mapping relationship table between the processing table and the data source table based on the processing logic. For example, the processing logic processes field 1 of data source table a and field 2 of data source table 2 to generate processing table c, and processes fields 3 to 5 of data source table d to generate processing table e. The generated mapping relationship table includes two mapping relationships, namely the mapping relationship between processing table c and data source tables a and b, and the mapping relationship between processing table e and data source table d.

[0058] It is worth noting that the processing process in the embodiment of the present application occurs daily. If the processing logic does not change, the mapping relationship contained in the mapping relationship table remains unchanged. However, when data processing is performed based on the mapping relationship, the data objects processed on each day are all newly generated data. For example, if the processing logic is to process field 1 of data source table a and field 2 of data source table 2 from July 1 to July 7 to generate processing table c, then from July 1 to July 7, data processing will be performed every day based on the mapping relationship between processing table c and data source tables a and b. However, the processing task on July 1 is based on field 1 of data source table a and field 2 of data source table b on July 1, generating processing table c on July 1. Similarly, the processing task on July 2 is based on field 1 of data source table a and field 2 of data source table b on July 2, generating processing table c on July 2, and so on.

[0059] Step S102: Determine each target data source table within the batch data date corresponding to the target processing table according to the mapping relationship table, obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table.

[0060] The target processing table is the processing table that requires timeliness analysis. For example, on June 30th, the downstream system indicates that processing table C, generated by processing field 1 of data source table a and field 2 of data source table 2 on August 1st, needs to be used at 2:00 PM on August 1st. In this case, the target processing table is processing table C. In order to process the newly generated fields 1 and 2 on August 1st into processing table C before 2:00 PM on August 1st and push it to the downstream system in a timely manner, the steps provided in this application can be followed in July.

[0061] The batch data date includes multiple data dates, and the batch data date refers to the data date on which the timeliness analysis of the target processing table is required.

[0062] The above-mentioned target data source table includes at least one data source table, wherein, if the target processing table in the mapping relationship table corresponds to one data source table (that is, the target processing table is processed by one data source table), then the target data source table includes one data source table; if the target processing table in the mapping relationship table corresponds to multiple data source tables (that is, the target processing table is processed by multiple data source tables), then the target data source table includes multiple data source tables.

[0063] Take this step as an example. Assume that the mapping relationship table includes: the mapping relationship between processing table c and data source tables a and b, and the mapping relationship between processing table e and data source table d. The batch data date is from July 1 to July 7, and the target processing table is processing table c.

[0064] The target data source tables from July 1 to July 7 corresponding to the target processing table include: the target data source table on July 1 (i.e., data source tables a and b on July 1), the target data source table on July 2 (i.e., data source tables a and b on July 2), the target data source table on July 3 (i.e., data source tables a and b on July 3), the target data source table on July 4 (i.e., data source tables a and b on July 4), the target data source table on July 5 (i.e., data source tables a and b on July 5), the target data source table on July 6 (i.e., data source tables a and b on July 6), and the target data source table on July 7 (i.e., data source tables a and b on July 7).

[0065] This step can obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of the arrival step corresponding to each target data source table, wherein the operation time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step; the arrival step is the step in which the target data source table reaches the big data platform. More specifically, the arrival step refers to the step in which each data source table contained in the target data source table reaches the big data platform.

[0066] In this embodiment of the present application, the aforementioned processing steps may include: a loading step and a batch data processing step; optionally, the aforementioned operation time points include a start time point and a completion time point. Thus, the operation time points of the loading step include the start time point of loading and the completion time point of loading, and the operation time points of the batch data processing step include the start time point of processing the target processing table and the completion time point of processing the target processing table.

[0067] Here, the loading step refers to the step in which the big data platform loads each data source table contained in the target data source table after each table arrives at the big data platform. In actual applications, the big data platform may load each data source table at different times. Therefore, the loading step refers to each data source table contained in the target data source table. That is, each data source table contained in the target data source table corresponds to a loading step.

[0068] The above-mentioned batch data processing step refers to the step in which the big data platform performs batch data processing based on the fields in all data source tables after loading all data source tables contained in the target data source table to obtain the target processed table. Therefore, the batch data processing step is for the target processing table, that is, the target processing table corresponds to one batch data processing step.

[0069] Still taking the previous example, the processing completion time points of the batch data processing steps obtained in this step (that is, the processing completion time points of the target processing table) include 7, namely: the processing completion time point of the target processing table corresponding to the target data source table on July 1, the processing completion time point of the target processing table corresponding to the target data source table on July 2, the processing completion time point of the target processing table corresponding to the target data source table on July 3, the processing completion time point of the target processing table corresponding to the target data source table on July 4, the processing completion time point of the target processing table corresponding to the target data source table on July 5, the processing completion time point of the target processing table corresponding to the target data source table on July 6, and the processing completion time point corresponding to the target processing table of the target data source table on July 7; the same applies to the start processing time point, which will not be repeated here.

[0070] The arrival time points obtained in this step include 14, namely: the time point when data source table a arrives at the big data platform on July 1, the time point when data source table b arrives at the big data platform on July 1, the time point when data source table a arrives at the big data platform on July 2, the time point when data source table b arrives at the big data platform on July 2, the time point when data source table a arrives at the big data platform on July 3, the time point when data source table b arrives at the big data platform on July 3, the time point when data source table a arrives at the big data platform on July 4, the time point when data source table b arrives at the big data platform on July 4, the time point when data source table a arrives at the big data platform on July 5, the time point when data source table b arrives at the big data platform on July 5, the time point when data source table a arrives at the big data platform on July 6, the time point when data source table b arrives at the big data platform on July 6, the time point when data source table a arrives at the big data platform on July 7, and the time point when data source table b arrives at the big data platform on July 7.

[0071] The loading start time points obtained in this step include 14, namely: the time point when the big data platform starts loading data source table a on July 1, the time point when the big data platform starts loading data source table b on July 1, the time point when the big data platform starts loading data source table a on July 2, the time point when the big data platform starts loading data source table b on July 2, the time point when the big data platform starts loading data source table a on July 3, the time point when the big data platform starts loading data source table b on July 3, the time point when the big data platform starts loading data source table a on July 4, the time point when the big data platform starts loading data source table b on July 4, the time point when the big data platform starts loading data source table a on July 5, the time point when the big data platform starts loading data source table b on July 5, the time point when the big data platform starts loading data source table a on July 6, the time point when the big data platform starts loading data source table b on July 6, the time point when the big data platform starts loading data source table a on July 7, and the time point when the big data platform starts loading data source table b on July 7; the same applies to the loading completion time points, which will not be repeated here.

[0072] In an optional embodiment, during the daily processing of each data source table including but not limited to the target data source table, the operation time point of each processing step and the arrival time point of each data source table at the big data platform can be recorded.

[0073] Taking the target data source table on July 1 as an example, the arrival time point of the target data source table on July 1 at the big data platform can be recorded, and the operation time point of each processing step can be recorded during the process of the big data platform processing the target data source table on July 1.

[0074] Then, this step can obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table.

[0075] Step S103: Determine whether the target processing table meets the timeliness requirement of the downstream system according to the processing completion time point of the target processing table corresponding to each target data source table.

[0076] Specifically, after the target data source table is processed to obtain the target processing table, the target processing table can be sent to the downstream system. Therefore, based on the processing completion time point of the target processing table corresponding to the target data source table, it can be determined whether the processing timeliness of the target processing table meets the timeliness requirements of the downstream system.

[0077] Step S104: If not satisfied, determine the target processing step that needs to be optimized among the processing steps according to the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table.

[0078] Specifically, this step can analyze the time consumption of the entire link of the target processing table at each time point to analyze the time-consuming processing steps, which are the target processing steps that need to be optimized.

[0079] It is worth noting that for the big data platform, the time when the target data source table arrives at the big data platform (arrival time point) cannot be controlled and optimized, but the process of processing the target data source table is executed on the big data platform, so the big data platform can optimize each processing step.

[0080] The data analysis method provided by the present application first obtains a mapping relationship table between the processing table and the data source table, and then determines each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, obtains the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table, and then determines whether the target processing table meets the timeliness requirements of the downstream system based on the processing completion time point of the target processing table corresponding to each target data source table. If not, the target processing step that needs to be optimized in each processing step is determined based on the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table. It can be seen from this that the present application can determine whether the processing timeliness of the target processing table meets the timeliness requirements of the downstream system through data analysis. If it does not meet the requirements, it can further analyze the target processing steps that need to be optimized, so that the target processing steps can be optimized to ensure that the downstream system can receive the required processing table in time.

[0081] In some embodiments of the present application, the process of the aforementioned "step S103, determining whether the target processing table meets the timeliness requirements of the downstream system according to the processing completion time points of the target processing tables corresponding to the target data source tables" is introduced.

[0082] In practical applications, the timeliness requirements of downstream systems can be either deterministic or probabilistic. For example, a deterministic requirement might be: Processing list c must be available before 2:00 PM on the day the data is generated. A probabilistic requirement might be: For data generated between August 1st and August 10th, processing list c must be available before 2:00 PM on the day the data is generated. It may occasionally be available later than 2:00 PM, but the probability cannot exceed 10%.

[0083] In the embodiments of the present application, different judgment methods may be used for different types of timeliness requirements, which are introduced below respectively.

[0084] First, for the deterministic requirement, the judgment process is as follows:

[0085] As introduced in the above paragraphs, the deterministic requirement includes a certain time point. The initial idea of ​​the inventor of this case is: compare the processing completion time points of the target processing tables corresponding to each target data source table within the batch data date with the time points in the deterministic requirement. If each processing completion time point meets the time point in the deterministic requirement, then it is determined that the target processing table meets the deterministic requirement. If any processing completion time point meets the time point in the uncertainty requirement, then it is determined that the target processing table does not meet the deterministic requirement.

[0086] However, when applying this solution, the inventors of this case found that the entire processing process may not be completed on the same day. For example, in most scenarios, the data generated from 0:00 to 24:00 on August 1st will not be sent to the big data platform until 0:00 on August 2nd, so it may take until 1:00 p.m. on August 2nd to complete the processing and obtain the processing table; if only the processing completion time points are compared, since 1:00 p.m. is before 2:00 p.m., the judgment result will be that the target processing table meets the timeliness requirements, but in fact the downstream system obtains the target processing table after 23 hours of timeout, which obviously does not meet the determinism requirement.

[0087] In order to accurately determine whether the target processing table meets the deterministic requirements, in a feasible embodiment, the relative processing completion time of the target processing table corresponding to each target data source table can be determined based on the processing completion time point of the target processing table corresponding to each target data source table and the data date of each target data source table. Here, the relative processing completion time refers to the time delay generated by the target data source table compared to the target data source table.

[0088] To facilitate calculation, preferably, a preset time point in the data date of each target data source table, such as zero o'clock, can be selected, and then for each target data source table, the difference between the processing completion time point of the target processing table corresponding to the target data source table and the preset time point in the data date of the target data source table is calculated, and the difference is determined as the relative processing completion time corresponding to the target data source table.

[0089] Taking the target data source table of July 1 (data date) as an example, if the target data source table is processed, the processing completion time is 6:00 am on July 2, and the preset time point is selected as 0:00, then the relative processing completion time is 30:00.

[0090] In an embodiment of the present application, if the timeliness requirement is a deterministic requirement, it is determined whether the relative processing completion time of the target processing table corresponding to each target data source table meets the time point requirement in the deterministic requirement. If so, it is determined that the target processing table meets the deterministic requirement; otherwise, it is determined that the target processing table does not meet the deterministic requirement.

[0091] For example, the seven target processing tables from July 1 to July 7 correspond to relative processing completion times of 32:00, 33:00, 34:00, 32.5:00, 33.5:00, 35:00, and 34:00, respectively. The time point in the deterministic requirement requires that the target processing table must be obtained before 12:00 a.m. the day after the data is generated. Since the above seven relative processing completion times are all less than 36:00, it is determined that the target processing table meets the deterministic requirement.

[0092] For another example, the seven target processing tables from July 1 to July 7 correspond to relative processing completion times of 33:00, 34:00, 35:00, 37:00, 35:00, 36.5:00, and 30:00, respectively. The time point in the deterministic requirement requires that the target processing table must be obtained before 12:00 noon the next day after the data is generated. Since two of the above seven relative processing completion times (i.e., 37:00 and 36.5:00) are greater than 36:00, it is determined that the target processing table does not meet the deterministic requirement.

[0093] Second, for probabilistic requirements, the judgment process is as follows:

[0094] Optionally, if the timeliness requirement is a probabilistic requirement, the number of relative processing completion times of the target processing tables corresponding to each target data source table that meet the time point requirements in the probabilistic requirement is counted. If the counted number is less than or equal to the number threshold in the probabilistic requirement, it is determined that the target processing table meets the probabilistic requirement; otherwise, it is determined that the target processing table does not meet the probabilistic requirement.

[0095] For example, the seven target processing tables from July 1st to July 7th correspond to relative processing completion times of 32:00, 33:00, 34:00, 32.5:00, 33.5:00, 35:00, and 34:00, respectively. The probabilistic requirement is: the target processing table must be obtained before 12:00 AM on the day after the data is generated. It may occasionally be later than 12:00 AM, but the probability does not exceed 15% (for this example, this translates to approximately 1 occurrence). However, the seven relative processing completion times shown in this example are all less than 36:00, so it is determined that the target processing table meets the probabilistic requirement.

[0096] For another example, the seven target processing tables from July 1st to July 7th correspond to relative processing completion times of 32:00, 33:00, 34:00, 32.5:00, 33.5:00, 37:00, and 38:00, respectively. The probabilistic requirement is: the target processing table must be obtained before 12:00 AM on the day after the data is generated. It may occasionally be later than 12:00 AM, but the probability does not exceed 15% (for this example, this translates to approximately 1 occurrence). However, of the seven relative processing completion times shown in this example, two exceed 36:00. Since 2 is greater than 1 (corresponding to 15%), it is determined that the target processing table does not meet the probabilistic requirement.

[0097] In summary, the embodiments of the present application provide different judgment processes for different types of timeliness requirements, and the judgment process takes into account the actual application situation, so that the ideas provided by the present application can be applied to actual scenarios, and the accuracy of the judgment results is relatively high.

[0098] In other embodiments of the present application, taking the above-mentioned operation time points including the start time point and the completion time point as an example, the process of the aforementioned "step S105, determining the target processing steps that need to be optimized in each processing step according to the operation time points of each processing step and the arrival time points of the arrival steps corresponding to each target data source table" is introduced.

[0099] In this embodiment, the process of step S105 includes: determining the relative start time and relative completion time of each processing step corresponding to each target data source table according to the start time point and completion time point of each processing step corresponding to each target data source table, and the data date respectively possessed by each target data source table; determining the relative arrival time of the arrival step corresponding to each target data source table according to the arrival time point of the arrival step corresponding to each target data source table, and the data date respectively possessed by each target data source table; determining the target processing step that needs to be optimized in each processing step according to the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table.

[0100] Similar to the previous embodiment, the embodiment of the present application also needs to analyze the actual interval time between each processing step based on the relative start time, relative completion time and relative arrival time, so as to determine the target processing step that needs to be optimized through analysis.

[0101] Optionally, the above process of "determining the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time point and completion time point of each processing step corresponding to each target data source table, and the data date respectively possessed by each target data source table" may include: selecting a preset time point in the data date respectively possessed by each target data source table; for each target data source table, calculating the difference between the start time point of each processing step corresponding to the target data source table and the preset time point in the data date possessed by the target data source table, to obtain the relative start time of each processing step corresponding to the target data source table; for each target data source table, calculating the difference between the completion time point of each processing step corresponding to the target data source table and the preset time point in the data date possessed by the target data source table, to obtain the relative completion time of each processing step corresponding to the target data source table.

[0102] Correspondingly, the above-mentioned process of "determining the relative arrival time of the arrival step corresponding to each target data source table based on the arrival time point of the arrival step corresponding to each target data source table and the data date respectively possessed by each target data source table" may include: for each target data source table, calculating the difference between the arrival time point of the arrival step corresponding to the target data source table and the preset time point in the data date possessed by the target data source table, to obtain the relative arrival time of the arrival step corresponding to the target data source table.

[0103] Optionally, for ease of calculation, the above-mentioned preset time point can be 0 o'clock. Of course, the preset time point can also be other, and this application does not make specific limitations on this.

[0104] For example, let's assume that the target data source table for July 1st includes data source table a, and the preset time point is 0:00. Assume that the arrival time corresponding to data source table a on July 1st is 8:00 AM on July 2nd, the loading time starts at 12:00 AM on July 2nd, and the loading completion time is 6:00 PM on July 2nd. The processing time corresponding to data source table a on July 1st is 8:00 PM on July 2nd, and the processing completion time is 1:00 AM on July 3rd.

[0105] Then, the relative arrival time of the arrival step corresponding to data source table a is 32 hours, the relative start time of the loading step corresponding to data source table a is 36 hours, and the relative completion time is 42 hours, and the relative start time of the batch data processing step corresponding to data source table a is 44 hours, and the relative completion time is 48 hours.

[0106] In an embodiment of the present application, the above-mentioned calculation method can be used to calculate the relative start time and relative completion time of each processing step corresponding to all target data source tables within the batch data date, as well as the relative arrival time of the arrival step, and then determine the target processing step through analysis.

[0107] The embodiment of the present application can calculate the relative start time, relative completion time and relative arrival time (collectively referred to as relative time) when the date of each processing step and arrival step is inconsistent with the generation date of the data in the target data source table. The relative time can better reflect the actual time delay between each step, thereby being able to more accurately analyze which processing steps need further optimization.

[0108] The embodiment of the present application can implement the process of obtaining the target processing steps through analysis in a variety of ways. The present application provides but is not limited to the following two implementation methods.

[0109] In an optional implementation, the embodiment of the present application can perform statistics on the relative start time, relative completion time and relative arrival time respectively during the analysis process. Specifically, the statistical start time corresponding to each processing step can be calculated based on the relative start time of each processing step corresponding to each target data source table; the statistical completion time corresponding to each processing step can be calculated based on the relative completion time of each processing step corresponding to each target data source table; the statistical arrival time of the arrival step can be calculated based on the relative arrival time of the arrival step corresponding to each target data source table; and the target processing step that needs to be optimized in each processing step can be determined based on the statistical start time and statistical completion time corresponding to each processing step, as well as the statistical arrival time of the arrival step.

[0110] Optionally, the above-mentioned statistical start time may include: the earliest relative start time, the latest relative start time and the average relative start time; the above-mentioned statistical completion time may include: the earliest relative completion time, the latest relative completion time and the average relative completion time; the above-mentioned statistical arrival time may include: the earliest relative arrival time, the latest relative arrival time and the average relative arrival time.

[0111] Taking the relative arrival times of the arrival steps corresponding to the target data source tables from July 1 to July 7 as an example, which are 34:00, 36:00, 34:00, 35:00, 36:00, 33:00 and 35:00 respectively, the earliest relative arrival time corresponding to the arrival step is 33:00, the latest relative arrival time is 36:00, and the average relative arrival time is approximately 34.71 hours.

[0112] For example, the relative start times of the loading steps corresponding to the target data source tables from July 1 to July 7 are 37:00, 38:00, 39:00, 37:00, 38:00, 40:00, and 44:00, respectively. The earliest relative start time for the loading step is 37:00, the latest is 44:00, and the average relative start time is approximately 39:00.

[0113] For example, the relative completion times of the loading steps corresponding to the target data source table from July 1 to July 7 are 40:00, 40:00, 41:00, 40:00, 42:00, 44:00, and 49:00, respectively. The earliest relative completion time of the loading step is 40:00, the latest relative completion time is 49:00, and the average relative completion time is approximately 42.29 hours.

[0114] The statistical start time and statistical completion time corresponding to the batch data processing step can refer to the calculation process of the loading step, which will not be repeated here.

[0115] After analysis, the latest relative arrival time in each target data source table from July 1 to July 7 is 36:00, but the earliest relative start time corresponding to the loading step is 37:00, and the latest relative start time is 40:00. It can be seen that there is a relatively large delay in the process of starting to load each target data source table. Therefore, if the target processing table does not meet the timeliness requirements, the start time point of the loading step can be advanced as much as possible, that is, by optimizing the start time point of the loading step, to ensure that the subsequent target processing table meets the timeliness requirements.

[0116] At the same time, the average relative start time of the loading step is about 39 hours, and the average relative completion time is about 42.29 hours, with a difference of about 3.29 hours. Under normal circumstances, the time spent loading each target data source table does not exceed 1 hour. Therefore, the actual time spent on the loading step is relatively long. The processing time of the loading step can be shortened as much as possible by optimizing the processing flow of the loading step to ensure that the subsequent target processing table meets the timeliness requirements.

[0117] It should be noted that the above-mentioned statistical start time, statistical completion time and statistical arrival time are only examples and are not intended to limit this application.

[0118] It should also be noted that the process of "determining the target processing steps that need to be optimized in each processing step based on the statistical start time and statistical completion time corresponding to each processing step, as well as the statistical arrival time of the arrival step" requires specific analysis of specific circumstances. The analysis process in the above example is only an optional implementation method of this application and cannot be used as a limitation of this application.

[0119] The embodiment of the present application counts the relative start time, relative completion time, and relative arrival time within a batch data date, and performs analysis based on the statistical results. Compared with analyzing the relative start time, relative completion time, and relative arrival time within a single data date, the analysis result is more accurate, that is, the target processing steps determined are more accurate.

[0120] In another optional implementation, the embodiment of the present application can also calculate the relative interval time of each processing step corresponding to each target data source table during the analysis process, and then compare the relative interval time with a preset interval threshold. If the number of dates on which the relative interval time of a processing step is greater than the preset interval threshold reaches a preset percentage threshold, the processing step is determined as the target processing step.

[0121] For example, the relative start times of the loading steps corresponding to the target data source tables from July 1 to July 7 (a total of 7 days) are 37:00, 38:00, 39:00, 37:00, 38:00, 40:00 and 44:00, and the relative completion times are 40:00, 40:00, 41:00, 40:00, 42:00, 44:00 and 49:00, respectively. The relative interval times of the loading steps corresponding to the target data source tables from July 1 to July 7 are 3 hours, 2 hours, 2 hours, 3 hours, 4 hours, 4 hours and 5 hours, respectively. If the maximum duration of the loading steps corresponding to the target data source tables does not exceed 3 hours, then it can be analyzed that the processing time of the loading steps corresponding to the target data source tables from July 5 to July 7 (a total of 3 days) exceeds the preset 3-hour interval threshold. Since 3 / 7 is greater than the preset percentage threshold of 10%, the loading step is determined as the target processing step that needs to be optimized, and the processing flow of the loading step can be optimized subsequently.

[0122] It should be noted that the above-mentioned interval threshold and percentage threshold need to be set according to actual conditions and are not limited in this application.

[0123] In summary, the data analysis method provided in the embodiments of the present application can automatically record the time points of each stage of data processing, and can perform timeliness analysis on demand. The analysis results (including the analysis results for determining whether the target processing table meets the timeliness requirements, and the analysis results for determining the target processing steps) are more accurate.

[0124] An embodiment of the present application further provides a data analysis device. The data analysis device provided in the embodiment of the present application is described below. The data analysis device described below and the data analysis method described above can be referenced to each other.

[0125] See also Figure 2 , shows a schematic diagram of the structure of the data analysis device provided in an embodiment of the present application, such as Figure 2 As shown, the data analysis device may include: a mapping relationship table acquisition module 201, a time point acquisition module 202, a timeliness requirement judgment module 203 and a step to be optimized determination module 204.

[0126] The mapping relationship table acquisition module 201 is used to obtain a mapping relationship table between a processing table and a data source table. The processing table is a table obtained by processing at least one corresponding data source table. The data source table has a data date, which is the date when the data in the data source table is generated.

[0127] The time point acquisition module 202 is used to determine each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, and obtain the operation time point of each processing step corresponding to each target data source table, as well as the arrival time point of the arrival step corresponding to each target data source table. The operation time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is a processing table that requires timeliness analysis, and the arrival step is the step where the target data source table arrives at the big data platform.

[0128] The timeliness requirement judgment module 203 is used to determine whether the target processing table meets the timeliness requirement of the downstream system according to the processing completion time point of the target processing table corresponding to each target data source table.

[0129] The module 204 for determining the step to be optimized is used to determine the target processing step that needs to be optimized among the processing steps according to the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table if the conditions are not met.

[0130] In one possible implementation, the timeliness requirement determination module may determine whether the target processing table meets the timeliness requirement of the downstream system based on the processing completion time points of the target processing tables corresponding to the target data source tables, including:

[0131] Determine the relative processing completion time of the target processing tables corresponding to the target data source tables according to the processing completion time points of the target processing tables corresponding to the target data source tables and the data dates of the target data source tables;

[0132] If the timeliness requirement is a deterministic requirement, then determine whether the relative processing completion time of the target processing table corresponding to each target data source table meets the time point requirement in the deterministic requirement. If so, determine that the target processing table meets the deterministic requirement; otherwise, determine that the target processing table does not meet the deterministic requirement;

[0133] If the timeliness requirement is a probabilistic requirement, then the number of relative processing completion times of the target processing tables corresponding to each target data source table that meet the time point requirements in the probabilistic requirement is counted. If the counted number is less than or equal to the number threshold in the probabilistic requirement, then it is determined that the target processing table meets the probabilistic requirement; otherwise, it is determined that the target processing table does not meet the probabilistic requirement.

[0134] In a possible implementation, the operation time point includes a start time point and a completion time point.

[0135] The process of the above-mentioned step determination module determining the target processing step to be optimized among the processing steps according to the operation time points of the processing steps and the arrival time points of the arrival steps corresponding to the target data source tables may include:

[0136] Determine the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time point and completion time point of each processing step corresponding to each target data source table, and the data date of each target data source table;

[0137] Determine the relative arrival time of the arrival steps corresponding to each target data source table according to the arrival time points of the arrival steps corresponding to each target data source table and the data dates respectively possessed by each target data source table;

[0138] According to the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table, the target processing step that needs to be optimized in each processing step is determined.

[0139] In one possible implementation, the to-be-optimized step determination module may determine the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time and completion time of each processing step corresponding to each target data source table and the data date of each target data source table, including:

[0140] Select a preset time point in the data date of each target data source table;

[0141] For each target data source table, calculate the difference between the start time of each processing step corresponding to the target data source table and the preset time point in the data date of the target data source table to obtain the relative start time of each processing step corresponding to the target data source table;

[0142] For each target data source table, calculate the completion time point of each processing step corresponding to the target data source table, and the difference between it and the preset time point in the data date of the target data source table to obtain the relative completion time of each processing step corresponding to the target data source table.

[0143] Accordingly, the process of the above-mentioned module for determining the steps to be optimized determining the relative arrival time of the arrival steps corresponding to each target data source table according to the arrival time point of each arrival step corresponding to each target data source table and the data date of each target data source table may include:

[0144] For each target data source table, calculate the arrival time point of the arrival step corresponding to the target data source table, and the difference between it and the preset time point in the data date of the target data source table to obtain the relative arrival time of the arrival step corresponding to the target data source table.

[0145] In one possible implementation, the process of the to-be-optimized step determination module determining the target processing step to be optimized among the processing steps based on the relative start time and the relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table, may include:

[0146] Calculate the statistical start time corresponding to each processing step according to the relative start time of each processing step corresponding to each target data source table;

[0147] Calculate the statistical completion time corresponding to each processing step based on the relative completion time of each processing step corresponding to each target data source table;

[0148] Calculate the statistical arrival time of the arrival step based on the relative arrival time of the arrival steps corresponding to each target data source table;

[0149] According to the statistical start time and statistical completion time corresponding to each processing step, and the statistical arrival time of the arrival step, the target processing step that needs to be optimized in each processing step is determined.

[0150] In one possible implementation, the statistical start time includes: the earliest relative start time, the latest relative start time and the average relative start time; the statistical completion time includes: the earliest relative completion time, the latest relative completion time and the average relative completion time; the statistical arrival time includes: the earliest relative arrival time, the latest relative arrival time and the average relative arrival time.

[0151] In a possible implementation, the above-mentioned processing steps include a loading step and a batch data processing step.

[0152] The present application embodiment also provides a data analysis device. Optionally, Figure 3 The hardware structure diagram of the data analysis device is shown in FIG. Figure 3 , the hardware structure of the data analysis device may include: at least one processor 301, at least one communication interface 302, at least one memory 303 and at least one communication bus 304;

[0153] In the embodiment of the present application, the number of the processor 301, the communication interface 302, the memory 303, and the communication bus 304 is at least one, and the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0154] The processor 301 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0155] The memory 303 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0156] The memory 303 stores a program, and the processor 301 can call the program stored in the memory 303, and the program is used to:

[0157] Obtain a mapping relationship table between a processing table and a data source table. The processing table refers to a table obtained by processing at least one corresponding data source table. The data source table has a data date, which is the date when the data in the data source table is generated.

[0158] Determine each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table, wherein the operation time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is the processing table that needs to be analyzed for timeliness, and the arrival step is the step where the target data source table arrives at the big data platform;

[0159] According to the processing completion time point of the target processing table corresponding to each target data source table, determine whether the target processing table meets the timeliness requirements of the downstream system;

[0160] If not, the target processing step that needs to be optimized in each processing step is determined according to the operation time point of each processing step and the arrival time point of the arrival step corresponding to each target data source table.

[0161] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0162] An embodiment of the present application further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data analysis method described above is implemented.

[0163] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0164] Finally, it should be noted that, in this document, relational terms such as and and the like are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0165] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0166] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data analysis method, characterized in that: include: Obtaining a mapping relationship table between a processing table and a data source table, wherein the processing table is a table obtained by processing at least one corresponding data source table, and the data source table has a data date, which is a generation date of the data in the data source table; Determine each target data source table within the batch data date corresponding to the target processing table according to the mapping relationship table, obtain the operation time point of each processing step corresponding to each target data source table, and the arrival time point of each arrival step corresponding to each target data source table, wherein the operation time point of each processing step includes the start time point and completion time point of each processing step, and the completion time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is the processing table that needs to be analyzed for timeliness, and the arrival step is the step in which the target data source table arrives at the big data platform; Determining whether the target processing table meets the timeliness requirement of the downstream system according to the processing completion time points of the target processing tables corresponding to the target data source tables; If not, determining the target processing step that needs to be optimized among the processing steps according to the operation time points of the processing steps and the arrival time points of the arrival steps respectively corresponding to the target data source tables; The determining, based on the processing completion time points of the target processing tables corresponding to the target data source tables, whether the target processing tables meet the timeliness requirements of the downstream system includes: Determine the relative processing completion time of the target processing table corresponding to each target data source table according to the processing completion time point of the target processing table corresponding to each target data source table and the data date of each target data source table; If the timeliness requirement is a deterministic requirement, determining whether the relative processing completion times of the target processing tables corresponding to the respective target data source tables all meet the time point requirements in the deterministic requirement; if so, determining that the target processing table meets the deterministic requirement; otherwise, determining that the target processing table does not meet the deterministic requirement; If the timeliness requirement is a probabilistic requirement, then the number of relative processing completion times of the target processing table corresponding to each target data source table that meets the time point requirement in the probabilistic requirement is counted; if the counted number is less than or equal to the number threshold in the probabilistic requirement, then it is determined that the target processing table meets the probabilistic requirement; otherwise, it is determined that the target processing table does not meet the probabilistic requirement; Determining the target processing step that needs to be optimized among the processing steps according to the operation time points of the processing steps and the arrival time points of the arrival steps respectively corresponding to the target data source tables includes: Determine the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time point and completion time point of each processing step corresponding to each target data source table and the data date respectively contained in each target data source table; Determine the relative arrival time of the arrival steps corresponding to the target data source tables according to the arrival time points of the arrival steps corresponding to the target data source tables and the data dates respectively possessed by the target data source tables; According to the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table, the target processing step that needs to be optimized among the processing steps is determined.

2. The data analysis method according to claim 1, characterized in that Determining the relative start time and relative completion time of each processing step corresponding to each target data source table according to the start time point and completion time point of each processing step corresponding to each target data source table, and the data date respectively contained in each target data source table, includes: Selecting a preset time point in the data date of each target data source table; For each target data source table, calculating the difference between the start time point of each processing step corresponding to the target data source table and the preset time point in the data date of the target data source table, to obtain the relative start time of each processing step corresponding to the target data source table; For each target data source table, calculate the difference between the completion time point of each processing step corresponding to the target data source table and the preset time point in the data date of the target data source table to obtain the relative completion time of each processing step corresponding to the target data source table; The determining, based on the arrival time points of the arrival steps respectively corresponding to the target data source tables and the data dates respectively possessed by the target data source tables, the relative arrival times of the arrival steps respectively corresponding to the target data source tables includes: For each of the target data source tables, calculate the arrival time point of the arrival step corresponding to the target data source table, and the difference between the arrival time point and the preset time point in the data date of the target data source table to obtain the relative arrival time of the arrival step corresponding to the target data source table.

3. The data analysis method according to claim 1, wherein: Determining the target processing step that needs to be optimized among the processing steps according to the relative start time and the relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table, includes: Calculate the statistical start time corresponding to each processing step according to the relative start time of each processing step corresponding to each target data source table; Calculate the statistical completion time corresponding to each processing step according to the relative completion time of each processing step corresponding to each target data source table; Calculate the statistical arrival time of the arrival step according to the relative arrival time of the arrival steps corresponding to each target data source table; A target processing step that needs to be optimized among the processing steps is determined according to the statistical start time and the statistical completion time respectively corresponding to the processing steps, and the statistical arrival time of the arrival step.

4. The data analysis method according to claim 3, characterized in that: The statistical start time includes: the earliest relative start time, the latest relative start time and the average relative start time; The statistical completion time includes: the earliest relative completion time, the latest relative completion time and the average relative completion time; The statistical arrival time includes: the earliest relative arrival time, the latest relative arrival time and the average relative arrival time.

5. The data analysis method according to any one of claims 1 to 4, characterized in that: The processing steps include a loading step and a batch data processing step.

6. A data analysis device, characterized in that: include: a mapping relationship table acquisition module, configured to acquire a mapping relationship table between a processing table and a data source table, wherein the processing table is a table obtained by processing at least one corresponding data source table, and the data source table has a data date, which is the date on which the data in the data source table is generated; A time point acquisition module is used to determine each target data source table within the batch data date corresponding to the target processing table based on the mapping relationship table, and obtain the operation time point of each processing step corresponding to each target data source table, as well as the arrival time point of the arrival step corresponding to each target data source table, wherein the operation time point of each processing step includes the start time point and completion time point of each processing step, and the completion time point of each processing step includes the processing completion time point of the target processing table obtained by the batch data processing step. The target processing table is the processing table that needs to be analyzed for timeliness, and the arrival step is the step in which the target data source table arrives at the big data platform; a timeliness requirement judgment module, configured to determine the relative processing completion time of the target processing table corresponding to each target data source table based on the processing completion time points of the target processing tables corresponding to each target data source table and the data dates respectively possessed by each target data source table; if the timeliness requirement is a deterministic requirement, then determining whether the relative processing completion times of the target processing tables corresponding to each target data source table all meet the time point requirements in the deterministic requirement; if so, determining that the target processing table meets the deterministic requirement; otherwise, determining that the target processing table does not meet the deterministic requirement; if the timeliness requirement is a probabilistic requirement, then counting the number of relative processing completion times of the target processing tables corresponding to each target data source table that meet the time point requirements in the probabilistic requirement; if the counted number is less than or equal to a number threshold in the probabilistic requirement, then determining that the target processing table meets the probabilistic requirement; otherwise, determining that the target processing table does not meet the probabilistic requirement; The module for determining the steps to be optimized is used to determine, if the conditions are not met, the relative start time and relative completion time of each processing step corresponding to each target data source table based on the start time point and completion time point of each processing step corresponding to each target data source table, and the data date respectively possessed by each target data source table; determine the relative arrival time of the arrival step corresponding to each target data source table based on the arrival time point of the arrival step corresponding to each target data source table, and the data date respectively possessed by each target data source table; determine the target processing step that needs to be optimized among the processing steps based on the relative start time and relative completion time of each processing step corresponding to each target data source table, and the relative arrival time of the arrival step corresponding to each target data source table.

7. A data analysis device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the data analysis method according to any one of claims 1 to 5.

8. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the data analysis method according to any one of claims 1 to 5 is implemented.

Images