A data processing method and device for data warehouse

Abstract

The embodiment of the present application discloses a data processing method and device for a data warehouse, wherein the method includes: reading the job operation data generated within a work cycle in the data warehouse; performing lineage analysis on the business to be identified based on the job operation data to obtain the job lineage link of the business to be identified; determining the target start time of the head job and the target end time of the tail job through the job execution sequence of the job lineage link; obtaining the overall job processing time of the job lineage link based on the target start time and the target end time; when the overall job processing time is greater than the preset overall time threshold corresponding to the business to be identified, determining that the business to be identified is a time-abnormal business. The method obtains the overall job time of the related jobs of the business to be identified that needs to be completed by performing lineage analysis on the job operation data in the data warehouse, and then can determine that the business whose overall job time is greater than the preset time threshold is a time-abnormal business.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method and device for a data warehouse. Background Art

[0002] In the era of big data, all banking systems generate a large amount of data. Banks usually need data from multiple systems when processing business, and cannot let each system build its own data system. Therefore, it is necessary to establish a data warehouse to centrally manage the huge amount of data in each system. Because data warehouses need to process a large number of businesses during operation, and processing a business usually requires multiple workflows, which are also called jobs. Therefore, data warehouses generate a large number of jobs during operation. These jobs have complex relationships and different running times. When the job runs for too long, the corresponding business processing time will also be too long, which will affect the customer experience and even lead to business failure.

[0003] Currently, banks mainly process data warehouses manually. Data warehouse operation data is huge and complex. Faced with huge operation data, data warehouse personnel find it difficult to handle jobs with long operation times, that is, it is difficult to handle businesses that take too long. Businesses that take too long will affect customer experience and may even lead to business failures, affecting the normal operation of banking business. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device for a data warehouse, which can process time-consuming business.

[0005] In view of this, the first aspect of the present application provides a data processing method for a data warehouse, the method comprising:

[0006] Reading the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job;

[0007] Performing lineage analysis on the business to be identified according to the job running data to obtain a job lineage link of the business to be identified; the job lineage link includes a job execution sequence of jobs in the business to be identified;

[0008] Obtaining the target start time of the head job and the target end time of the tail job of the job lineage link through the job execution sequence;

[0009] Calculate the overall operation processing time of the operation lineage link according to the target start time and the target end time;

[0010] If the overall operation processing time is greater than a preset overall time threshold corresponding to the to-be-identified business, it is determined that the to-be-identified business is a time-abnormal business.

[0011] Optionally, obtaining the target start time of the head job and the target end time of the tail job of the job lineage link through the job execution sequence includes:

[0012] Obtaining the start time and end time of each job in the job lineage link through the job execution sequence;

[0013] After determining that the service to be identified is a time-abnormal service, the method further includes:

[0014] Calculate the processing time of each job according to the start time and end time of each job;

[0015] sorting the processing priorities of the jobs based on the processing time of the jobs;

[0016] Process the job processing time of the corresponding job lineage link according to the priority.

[0017] Optionally, the method further includes:

[0018] The sum of the job processing time of all jobs in the job lineage link is determined as the absolute overall job processing time of the job lineage link;

[0019] Correspondingly, when the overall operation processing time is less than the preset overall time threshold, if the absolute overall operation processing time is greater than the preset absolute overall time threshold, the business to be identified is determined to be a time-abnormal business.

[0020] Optionally, the preset overall duration threshold is obtained in the following manner:

[0021] A correspondence between the job lineage link level and the overall duration threshold is preset; the job lineage link level represents the total number of jobs in the job lineage link;

[0022] The preset overall duration threshold is determined according to the job lineage link level corresponding to the business to be identified and the corresponding relationship.

[0023] Optionally, performing lineage analysis on the to-be-identified business according to the job running data to obtain a job lineage link of the to-be-identified business includes:

[0024] A lineage analysis is performed on the business to be identified according to the job running data to obtain a job lineage link of the business to be identified, and a table lineage link or a field lineage link of the business to be identified.

[0025] Optionally, the method further includes:

[0026] According to the business query information inputted on the front-end page, a business data lineage link corresponding to the business query information is obtained; the business lineage link includes the job lineage link, the table lineage link or the field lineage link;

[0027] The corresponding business data lineage link is sent to the front-end page so that the front-end page can display it.

[0028] Optionally, the method further includes:

[0029] Processing a business lineage link, wherein the business lineage link includes the job lineage link, the table lineage link or the field lineage link;

[0030] If the business bloodline link generates abnormal information, process the abnormal information; the abnormal information includes a closed loop of bloodline link information, missing bloodline link information or broken bloodline link information; the closed loop of the bloodline link includes that the node information in the bloodline link cannot be sent from the current node; the missing bloodline link information includes that the upstream node information required by the node in the bloodline link is missing; the broken bloodline link information includes that the generation of information in the bloodline link is interrupted.

[0031] A second aspect of the present application provides a data processing device for a data warehouse, the device comprising:

[0032] A data reading module is used to read the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job;

[0033] A data lineage analysis module, configured to perform lineage analysis on the business to be identified based on the job running data, and obtain a job lineage link of the business to be identified; the job lineage link includes a job execution sequence of jobs in the business to be identified;

[0034] A job time determination unit, used to obtain the target start time of the head job and the target end time of the tail job of the job lineage link according to the job execution sequence;

[0035] An overall job processing time calculation unit, used to calculate the overall job processing time of the job lineage link according to the target start time and the target end time;

[0036] The overall operation processing time determination unit is used to determine that the business to be identified is a time-abnormal business if the overall operation processing time is greater than a preset overall time threshold corresponding to the business to be identified.

[0037] Optionally, the data lineage analysis module is used to perform lineage analysis on the business to be identified according to the job running data, obtain the job lineage link of the business to be identified, and obtain the table lineage link or field lineage link of the business to be identified, and the device also includes:

[0038] A lineage link display module is used to obtain the business data lineage link corresponding to the business query information input into the front-end page; the business lineage link includes the job lineage link, the table lineage link or the field lineage link; and the corresponding business data lineage link is sent to the front-end page so that the front-end page can display it.

[0039] Optionally, the data lineage analysis module is used to perform lineage analysis on the business to be identified according to the job running data, obtain the job lineage link of the business to be identified, and obtain the table lineage link or field lineage link of the business to be identified, and the device also includes:

[0040] An exception analysis module is used to process business lineage links, wherein the business lineage links include the job lineage links, the table lineage links or the field lineage links; if the business lineage links generate abnormal information, the abnormal information is processed; the abnormal information includes a lineage link information loop, a lineage link information missing or a lineage link information break, wherein the lineage link loop includes that the node information in the lineage link cannot be sent from the current node, the lineage link information missing includes that the upstream node information required by the node in the lineage link is missing, and the lineage link information break includes that the information generation in the lineage link is interrupted.

[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0042] An embodiment of the present application provides a data analysis method for a data warehouse, the method comprising: first, reading job running data generated within a work cycle in the data warehouse, wherein the job running data includes the start time and end time of the job; then, performing a lineage analysis on the business to be identified based on the job running data to obtain the job lineage link of the business to be identified, the lineage analysis can organize the job running data with complex relationships, and organize all jobs required to complete the business to be identified as job lineage links; then, through the job execution sequence of the job lineage link of the business to be identified, determine the target start time of the head job and the target end time of the tail job; finally, according to the target start time and target end time, obtain the overall job processing time, the overall job time can reflect the total time required to complete the business to be identified, and when the overall job processing time is greater than the preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business. The above method performs lineage analysis on the job running data in the data warehouse to obtain the overall job duration of all jobs corresponding to the identified business that needs to be completed, and determines that the overall job duration greater than the preset duration threshold is a time-abnormal business, thereby accurately processing time-consuming businesses, improving customer experience, reducing business failure rate, and facilitating the normal operation of banking business. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A flowchart of a data processing method for a data warehouse provided in an embodiment of the present application;

[0045] Figure 2 A network diagram of the logical relationship between the service data bloodline links provided in the embodiment of the present application;

[0046] Figure 3 The graph database provided in the embodiment of the present application displays a relationship diagram of business data lineage links;

[0047] Figure 4 A flowchart of another data processing method for a data warehouse provided in another embodiment of the present application;

[0048] Figure 5 A schematic diagram of a data processing device for a data warehouse provided in accordance with another embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] The bank's data warehouse needs to process a large number of businesses during operation, which in turn generates a large number of jobs. These jobs have complex relationships, and it is difficult for data warehouse personnel to sort out all the jobs corresponding to the businesses. In other words, it is difficult to determine the time required to complete all the jobs, and thus it is difficult to determine whether the time-consuming business is an abnormal business.

[0052] In order to solve the problems existing in the above-mentioned prior art, an embodiment of the present application provides a data processing method for a data warehouse, which can obtain the time required for all operations corresponding to the business, and then determine that the business that takes a long time is an abnormal business.

[0053] Specifically, in the data processing method for a data warehouse provided in an embodiment of the present application, the job running data generated within a working cycle in the data warehouse is first read, wherein the job running data includes the start time and end time of the job; then, a lineage analysis is performed on the business to be identified based on the job running data to obtain the job lineage link of the business to be identified. The lineage analysis can organize the job running data with complex relationships, and organize all the jobs required to complete the business to be identified as a job lineage link; then, the target start time of the head job and the target end time of the tail job are determined through the job execution sequence of the job lineage link of the business to be identified; finally, the overall job processing time is obtained based on the target start time and the target end time. The overall job processing time can reflect the total time required to complete the business to be identified. When the overall job processing time is greater than the preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business. The above method performs lineage analysis on the job running data in the data warehouse to obtain the overall job duration of all jobs corresponding to the identified business that needs to be completed, and determines that the overall job duration greater than the preset duration threshold is a time-abnormal business, thereby accurately processing time-consuming businesses, improving customer experience, reducing business failure rate, and facilitating the normal operation of banking business.

[0054] The data processing method for a data warehouse provided by the present application is described in detail below through a method embodiment.

[0055] With respect to the data warehouse described above, the present application embodiment provides a data processing method to analyze and process the data in the data warehouse. Figure 1 As shown, the data processing method for a data warehouse provided in an embodiment of the present application includes the following steps:

[0056] S101. Read the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job.

[0057] Specifically, the data warehouse stores the job running data generated in the work cycle, and the job running data can be read from the data warehouse.

[0058] It should be noted that, first of all, the working cycle can be determined based on the job running data actually generated by the data warehouse, or it can be determined based on the time period of the required job running data. There is no limitation here. For example, one day can be selected as a working cycle, and then the job running data can be read through the log file in the data warehouse.

[0059] Then, the job running data can be the data generated when the job is running and stored in the data warehouse. In addition to the start time and end time of the job, the job running data also includes data information generated when the job is running, such as the table name corresponding to the job. The table name corresponding to the job refers to the table name required for the job running and the table name generated by the job.

[0060] S102 . Perform lineage analysis on the business to be identified based on the job running data to obtain a job lineage link of the business to be identified; the job lineage link includes the job execution sequence of the jobs in the business to be identified.

[0061] Specifically, since the job running data includes the upstream and downstream relationships between jobs, that is, the processing logic between jobs, for example, job 1 can process the call data A to obtain data B, and job 2 can process the call data B to obtain data C. It can be seen that there is a relationship between job 1 and job 2. Job 1 is the upstream job of job 2, and job 2 is the downstream job of job 1. Therefore, based on the job running data, a lineage analysis can be performed on the business to be identified to obtain the job lineage link of the business to be identified. The job lineage link can reflect all jobs corresponding to the business to be identified and the execution order between these jobs.

[0062] Through lineage analysis, the operation data of complex relationships can be sorted out, and all the jobs required to complete the business to be identified and the execution order between these jobs can be sorted out as the job lineage link.

[0063] It should be noted that, first of all, data lineage analysis is a concept in data governance. It is to find the connection between related data in the process of data tracing. It is a logical concept.

[0064] Then, the lineage link refers to the link that generates data, which can be roughly understood as a data generation process. This generation process refers to which data the data depends on, how it is generated, and how the data it depends on is generated. To put it bluntly, it is how the data comes from and what processes and stages it goes through.

[0065] Optionally, a lineage analysis is performed on the business to be identified based on the job running data to obtain a job lineage link of the business to be identified, as well as a table lineage link or a field lineage link of the business to be identified.

[0066] Specifically, since the job running data includes the table name corresponding to the job, that is, the table name required for the job running and the table name generated by the job, the job running data not only reflects the processing logic between jobs, but also reflects the calling relationship between jobs and tables. For example, job 3 can process the calling table A to obtain table B, which shows that there is an association between table A and table B. Based on the job running data, the calling relationship between jobs and tables is analyzed to obtain the table lineage link of the business to be identified.

[0067] Since the job running data reflects the calling relationship between the job and the table, and further, it also reflects the calling relationship between the job and the fields in the table. The field refers to the specific information in the table. Therefore, based on the job running data, the calling relationship between the job and the fields in the table is analyzed to obtain the field lineage link of the business to be identified.

[0068] The obtained logical relationship network diagram between the job lineage link, table lineage link and field lineage link of the business to be identified can be shown as follows: Figure 2 shown.

[0069] Obtaining job lineage links, table lineage links, and field lineage links through job running data helps data warehouse personnel sort out the relationships between jobs, tables, and fields, and helps data warehouse personnel manage the data warehouse.

[0070] Optionally, after performing lineage analysis on the business to be identified based on the job operation data and obtaining the job lineage link of the business to be identified, as well as the table lineage link or field lineage link of the business to be identified, the embodiment of the present application can import various categories of business data lineage links and related data information into a graph database, wherein the various categories of business data include the above-mentioned job lineage links, table lineage links and field lineage links, and the graph database includes a graph formed according to the lineage link relationships of various categories of business data, and the display results can be seen Figure 3 .

[0071] The graph database can also store business data categories as nodes, store the blood relationship between business data blood links as lines, and store the relevant business data information corresponding to the nodes as attributes. In other words, the graph database can form a business blood relationship network diagram to display the business data blood links.

[0072] By importing the bloodline links of various categories of business data into the graph database, it is possible to store the bloodline links of various categories of business data in the form of a relationship network graph, which is conducive to the subsequent visualization of the bloodline links of the business data.

[0073] Optionally, after performing a lineage analysis on the business to be identified based on the job running data and obtaining the job lineage link of the business to be identified, as well as the table lineage link or field lineage link of the business to be identified, the embodiment of the present application can obtain the business data lineage link corresponding to the business query information based on the business query information input on the front-end page, where the business lineage link includes the job lineage link, the table lineage link or the field lineage link, and send the corresponding business data lineage link to the front-end page so that the front-end page can display it.

[0074] Specifically, the business data lineage link corresponding to the business query information can be obtained based on the business query information and the business data lineage link input on the front-end page, and then the business data lineage link related information can be output to the front-end page. In this way, data warehouse processing personnel can obtain the business data lineage link related to the business query information by inputting the business query information. The business data lineage link can reflect the logical relationship between businesses, which is conducive to data warehouse processing personnel to quickly find related data.

[0075] Optionally, after the embodiment of the present application performs a lineage analysis on the business to be identified based on the job operation data, obtains the job lineage link of the business to be identified, and obtains the table lineage link or field lineage link of the business to be identified, the business lineage link can be processed, and the business lineage link includes a job lineage link, a table lineage link or a field lineage link. If the business lineage link generates abnormal information, the abnormal information is processed; wherein the abnormal information includes a closed loop of lineage link information, missing lineage link information or a broken lineage link information, the closed loop of the lineage link includes that the node information in the lineage link cannot be sent from the current node, the missing lineage link information includes that the upstream node information required by the node in the lineage link is missing, and the broken lineage link information includes that the generation of information in the lineage link is interrupted. In this way, the data warehouse processing personnel can process the abnormal information generated during the operation of the data warehouse through the business data lineage link, and then analyze the cause of the abnormal information, the scope of impact, etc., which is beneficial for the data warehouse processing personnel to manage the data warehouse.

[0076] S103. Obtain the target start time of the head job and the target end time of the tail job in the job lineage link through the job execution sequence.

[0077] Specifically, the job lineage chain can reflect all jobs corresponding to the business to be identified and the execution order between these jobs. Through the execution order of the jobs, the head job and the tail job of the job lineage chain can be determined, and the target start time of the head job and the target end time of the tail job of the job lineage chain can also be determined.

[0078] It should be noted that the target start time of the head job refers to when the head job starts running, which can be the earliest arrival time of the head files in the head job. The target end time of the tail job refers to when the tail job is completed, which can be the end time of the tail job batch.

[0079] S104. Calculate the overall operation processing time of the operation lineage link according to the target start time and the target end time.

[0080] Specifically, the target start time is the target start time of the head job in the job lineage chain, which can indicate when the job lineage chain starts. The target end time is the target end time of the tail job in the job lineage chain, which can indicate when the job lineage chain ends. Through the target start time and target end time, we can determine when the job lineage chain starts and ends. By calculation, we can get the overall job processing time of the job lineage chain.

[0081] Because the job lineage link corresponding to the business to be identified can reflect all the jobs required for the business to be identified and the execution order of the jobs, the overall job duration required to implement the job lineage link can reflect the total duration required to complete the business to be identified.

[0082] S105: If the overall operation processing time is greater than a preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business.

[0083] Specifically, the overall operation duration can be compared with the preset overall duration threshold. Since the overall operation duration can reflect the total time required to complete the business to be identified, when the overall operation duration is greater than the preset overall duration threshold, it can be determined that the business to be identified is a time-abnormal business.

[0084] The preset overall duration threshold may represent a preset maximum value of the duration required for the service to be identified.

[0085] Optionally, the preset overall time threshold may be set according to the business type of the business to be identified. The time required for different types of business may be different, and the preset overall time threshold may be set according to the business type. For example, for ordinary transfer business, the preset overall time threshold may be 5 minutes, and for cross-border transfer business, the preset overall time threshold may be 7 working days.

[0086] Optionally, the correspondence between the job lineage link level and the overall duration threshold can be pre-set, where the job lineage link level represents the total number of jobs in the job lineage link; and then the preset overall duration threshold is determined based on the job lineage link level corresponding to the business to be identified and the above correspondence.

[0087] Specifically, the job lineage link level can represent the total number of jobs required to complete the lineage link. Generally speaking, the overall job duration is longer when the job lineage link level is higher. Therefore, the correspondence between the job lineage link level and the overall duration threshold is pre-set, and the preset overall duration threshold of the job lineage link is determined according to the job lineage link level corresponding to the required business to be identified.

[0088] It is relatively simple to determine the preset overall duration threshold according to the job lineage link level. The preset overall duration threshold can be determined by determining the job lineage link level of the business to be identified. There is no need to perform other analysis on the business to be identified, and the determined preset overall duration threshold can reflect a preset maximum value of the duration required for the business to be identified.

[0089] An embodiment of the present application provides a data processing method for a data warehouse, which first reads the job running data generated within a working cycle in the data warehouse, wherein the job running data includes the start time and end time of the job; then, a lineage analysis is performed on the business to be identified based on the job running data to obtain the job lineage link of the business to be identified. The lineage analysis can organize the job running data with complex relationships, and organize all the jobs required to complete the business to be identified as the job lineage link; then, the target start time of the head job and the target end time of the tail job are determined through the job execution sequence of the job lineage link of the business to be identified; finally, the overall job processing time is obtained based on the target start time and the target end time. The overall job time can reflect the total time required to complete the business to be identified. When the overall job processing time is greater than the preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business. The above method performs lineage analysis on the job running data in the data warehouse to obtain the overall job duration of all jobs corresponding to the identified business that needs to be completed, and determines that the overall job duration greater than the preset duration threshold is a time-abnormal business, thereby accurately processing time-consuming businesses, improving customer experience, reducing business failure rate, and facilitating the normal operation of banking business.

[0090] Another embodiment of the present application provides another data processing method for a data warehouse, such as Figure 4 Said, comprising the following steps:

[0091] S401, reading the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job.

[0092] It should be noted that the specific implementation of step S401 may refer to step S101 in the above method embodiment, and will not be described in detail here.

[0093] S402 , performing lineage analysis on the business to be identified based on the job running data to obtain a job lineage link of the business to be identified; the job lineage link includes the job execution sequence of the jobs in the business to be identified.

[0094] It should be noted that the specific implementation of step S402 may refer to step S102 in the above method embodiment, and will not be described in detail here.

[0095] S403. Obtain the start time and end time of each job in the job lineage chain through the job execution sequence.

[0096] Specifically, S403 is a possible specific implementation method of S103 in the embodiment of the present application. The job lineage link can reflect all jobs corresponding to the business to be identified and the execution order of these jobs. The start time and end time of each job in the job lineage link can be determined according to the job lineage link. The start time of the job can be the earliest arrival time of the job file, and the end time of the job can be the end time of the job batch running. By determining the start time and end time of each job, the start time and end time of the head job are naturally determined.

[0097] S404. Calculate the overall operation processing time of the operation bloodline link according to the start time of the head operation and the end time of the tail operation.

[0098] Specifically, S403 obtains the start time and end time of each job in the job lineage chain, among which the overall job processing time of the job lineage chain can be obtained according to the start time of the head job and the end time of the tail job.

[0099] S405: If the overall operation processing time is greater than the preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business.

[0100] It should be noted that the specific implementation of step S405 may refer to step S105 in the above method embodiment, which will not be described in detail here.

[0101] S406. Calculate the processing time of each job based on the start time and end time of each job; sort the jobs based on the processing time; and process the processing time of the corresponding job lineage link according to the priority.

[0102] Specifically, we can first calculate the processing time of each job based on the start time and end time of each job. The start time of the job can be the earliest arrival time of the job files, and the end time of the job can be the end time of the job batch.

[0103] The jobs can then be prioritized according to their processing time, and jobs with higher priorities can be processed first to reduce the time required for their running.

[0104] Finally, the job processing time of the job lineage chain is processed according to the priority. By prioritizing the jobs according to the job processing time, jobs with longer processing time can be processed first. This can give priority to jobs with longer processing time in the job lineage chain with longer overall job processing time, thereby effectively processing the job lineage chain with longer overall job processing time.

[0105] Optionally, the sum of the job processing times of all jobs in the job lineage link can be determined as the absolute overall job processing time of the job lineage link. Accordingly, when the overall job processing time is less than the preset overall time threshold, if the absolute overall job processing time is greater than the preset absolute overall time threshold, the business to be identified is determined to be a time-abnormal business.

[0106] Specifically, the absolute overall job processing time can reflect the absolute processing time required to complete the job lineage link excluding the waiting time. The time required to realize the job lineage link includes not only the processing time of each job, but also the waiting time. The waiting time may include the time waiting for the files required to run the job to arrive. When the overall job processing time is less than the preset overall time threshold, a situation may also occur: although the absolute processing time is longer, due to the short waiting time, the final overall job processing time is less than the preset overall time threshold. When the overall job processing time is less than the preset overall time threshold, more accurate identification of time-abnormal business can be achieved by determining the business to be identified whose absolute overall job processing time is greater than the preset absolute overall time threshold as a time-abnormal business.

[0107] Another embodiment of the present application provides a data processing device for a data warehouse, such as Figure 5 As shown, the device comprises:

[0108] The data reading unit 501 is used to read the job running data generated in the work cycle stored in the data warehouse, and the job running data includes the start time and the end time of the job.

[0109] The data lineage analysis unit 502 is used to perform lineage analysis on the business to be identified based on the job running data to obtain the job lineage link of the business to be identified; the job lineage link includes the job execution order of the jobs in the business to be identified.

[0110] The job time determination unit 503 is used to obtain the target start time of the head job and the target end time of the tail job in the job lineage link according to the job execution sequence.

[0111] The overall job processing time calculation unit 504 is used to calculate the overall job processing time of the job lineage link according to the target start time and the target end time.

[0112] The overall operation processing time determination unit 505 is used to determine that the business to be identified is a time-abnormal business if the overall operation processing time is greater than a preset overall time threshold corresponding to the business to be identified.

[0113] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, the job time determination unit is used to obtain the start time and end time of each job in the job lineage link according to the job execution order. After the overall job processing time determination unit determines that the business to be identified is a time-abnormal business, the data processing device for the data warehouse also includes:

[0114] The job processing time calculation unit is used to calculate the job processing time of each job according to the start time and end time of each job.

[0115] The job processing priority sorting unit is used to sort the processing priorities of the jobs based on the job processing time.

[0116] The job processing time processing unit is used to process the job processing time of the corresponding job lineage link according to the priority.

[0117] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, the data processing device for a data warehouse further includes:

[0118] The absolute overall job processing time calculation unit is used to determine the sum of the job processing times of all jobs in the job lineage link as the absolute overall job processing time of the job lineage link.

[0119] The absolute overall operation processing time judgment unit is used to determine that the business to be identified is a time-abnormal business when the overall operation processing time is less than a preset overall time threshold and if the absolute overall operation processing time is greater than the preset absolute overall time threshold.

[0120] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, the data processing device for a data warehouse further includes:

[0121] The preset overall duration threshold determination unit is used to pre-set the correspondence between the job lineage link level and the overall duration threshold; the job lineage link level represents the total number of jobs in the job lineage link; the preset overall duration threshold is determined based on the job lineage link level and the corresponding relationship corresponding to the business to be identified.

[0122] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, a data lineage analysis unit is used to perform lineage analysis on the business to be identified based on job running data to obtain the job lineage link of the business to be identified, and obtain the table lineage link or field lineage link of the business to be identified.

[0123] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, a data lineage analysis unit is used to perform lineage analysis on a to-be-identified business according to job running data, obtain a job lineage link of the to-be-identified business, and obtain a table lineage link or a field lineage link of the to-be-identified business. The data processing device for a data warehouse further includes:

[0124] The lineage link display unit is used to obtain the business data lineage link corresponding to the business query information according to the business query information input by the front-end page; the business lineage link includes the job lineage link, the table lineage link or the field lineage link; and the corresponding business data lineage link is sent to the front-end page for display on the front-end page.

[0125] Optionally, in a data processing device for a data warehouse provided in another embodiment of the present application, a data lineage analysis unit is used to perform lineage analysis on a to-be-identified business according to job running data, obtain a job lineage link of the to-be-identified business, and obtain a table lineage link or a field lineage link of the to-be-identified business. The data processing device for a data warehouse further includes:

[0126] The abnormality analysis unit is used to process the business lineage link, which includes the job lineage link, table lineage link or field lineage link; if the business lineage link generates abnormal information, the abnormal information is processed; the abnormal information includes the lineage link information loop, the lineage link information loss or the lineage link information break. The lineage link loop includes that the node information in the lineage link cannot be sent from the current node, the lineage link information loss includes the upstream node information required by the node in the lineage link is missing, and the lineage link information break includes the interruption of information generation in the lineage link.

[0127] It should be noted that the specific working processes of the various modules provided in the above embodiments of the present application can refer to the corresponding implementation methods in the above method embodiments, and will not be repeated here.

[0128] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method for a data warehouse, characterized in that: The method comprises: Reading the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job; Performing lineage analysis on the business to be identified according to the job running data to obtain a job lineage link of the business to be identified; the job lineage link includes a job execution sequence of jobs in the business to be identified; Obtaining the target start time of the head job and the target end time of the tail job of the job lineage link through the job execution sequence; Calculate the overall operation processing time of the operation lineage link according to the target start time and the target end time; If the overall operation processing time is greater than the preset overall time threshold corresponding to the business to be identified, the business to be identified is determined to be a time-abnormal business; wherein the preset overall time threshold is obtained in the following manner: presetting the correspondence between the operation lineage link level and the overall time threshold; the operation lineage link level represents the total number of operations in the operation lineage link; determining the preset overall time threshold according to the operation lineage link level corresponding to the business to be identified and the corresponding relationship; The sum of the job processing time and the waiting time of all jobs in the job lineage link is determined as the absolute overall job processing time of the job lineage link, and the waiting time includes the time for waiting for the files required to run the job to arrive; Correspondingly, when the overall operation processing time is less than the preset overall time threshold, if the absolute overall operation processing time is greater than the preset absolute overall time threshold, the business to be identified is determined to be a time-abnormal business.

2. The method according to claim 1, characterized in that The step of obtaining the target start time of the head job and the target end time of the tail job of the job lineage link through the job execution sequence includes: Obtaining the start time and end time of each job in the job lineage link through the job execution sequence; After determining that the service to be identified is a time-abnormal service, the method further includes: Calculate the processing time of each job according to the start time and end time of each job; sorting the processing priorities of the jobs based on the processing time of the jobs; Process the job processing time of the corresponding job lineage link according to the priority.

3. The method according to claim 1, characterized in that The performing lineage analysis on the to-be-identified business according to the job running data to obtain the job lineage link of the to-be-identified business includes: A lineage analysis is performed on the business to be identified according to the job running data to obtain a job lineage link of the business to be identified, and a table lineage link or a field lineage link of the business to be identified.

4. The method according to claim 3, characterized in that The method further comprises: According to the business query information inputted on the front-end page, a business data lineage link corresponding to the business query information is obtained; the business lineage link includes the job lineage link, the table lineage link or the field lineage link; The corresponding business data lineage link is sent to the front-end page so that the front-end page can display it.

5. The method according to claim 3, characterized in that: The method further comprises: Processing a business lineage link, wherein the business lineage link includes the job lineage link, the table lineage link or the field lineage link; If the business bloodline link generates abnormal information, process the abnormal information; the abnormal information includes a closed loop of bloodline link information, missing bloodline link information or a broken bloodline link information; the closed loop of the bloodline link includes that the node information in the bloodline link cannot be sent from the current node; the missing bloodline link information includes that the upstream node information required by the node in the bloodline link is missing; the broken bloodline link information includes that the generation of information in the bloodline link is interrupted.

6. A data processing device for a data warehouse, characterized in that: The device comprises: A data reading unit, used to read the job running data generated in the work cycle stored in the data warehouse; the job running data includes the start time and the end time of the job; A data lineage analysis unit, configured to perform lineage analysis on the business to be identified based on the job running data, and obtain a job lineage link of the business to be identified; the job lineage link includes a job execution sequence of jobs in the business to be identified; A job time determination unit, used to obtain the target start time of the head job and the target end time of the tail job of the job lineage link according to the job execution sequence; An overall job processing time calculation unit, used to calculate the overall job processing time of the job lineage link according to the target start time and the target end time; An overall operation processing time determination unit, configured to determine that the business to be identified is a time-abnormal business if the overall operation processing time is greater than a preset overall time threshold corresponding to the business to be identified; A preset overall duration threshold determination unit is used to preset a correspondence between the job lineage link level and the overall duration threshold; the job lineage link level represents the total number of jobs in the job lineage link; the preset overall duration threshold is determined according to the job lineage link level corresponding to the business to be identified and the correspondence; An absolute overall job processing time calculation unit, used to determine the sum of the job processing time and waiting time of all jobs in the job lineage link as the absolute overall job processing time of the job lineage link, wherein the waiting time includes the time for waiting for the files required to run the job to arrive; The absolute overall operation processing time judgment unit is used to determine that the business to be identified is a time-abnormal business when the overall operation processing time is less than the preset overall time threshold and if the absolute overall operation processing time is greater than the preset absolute overall time threshold.

7. The device according to claim 6, characterized in that The data lineage analysis unit is used to perform lineage analysis on the business to be identified according to the job running data, obtain the job lineage link of the business to be identified, and obtain the table lineage link or field lineage link of the business to be identified. The device also includes: A lineage link display unit is used to obtain the business data lineage link corresponding to the business query information input into the front-end page; the business lineage link includes the job lineage link, the table lineage link or the field lineage link; and the corresponding business data lineage link is sent to the front-end page so that the front-end page can display it.

8. The device according to claim 6, characterized in that The data lineage analysis module is used to perform lineage analysis on the business to be identified according to the job running data, obtain the job lineage link of the business to be identified, and obtain the table lineage link or field lineage link of the business to be identified. The device also includes: An exception analysis unit is used to process a business lineage link, wherein the business lineage link includes the job lineage link, the table lineage link or the field lineage link; if the business lineage link generates exception information, the exception information is processed; the exception information includes a lineage link information loop, a lineage link information missing or a lineage link information break, wherein the lineage link loop includes that the node information in the lineage link cannot be sent from the current node, the lineage link information missing includes that the upstream node information required by the node in the lineage link is missing, and the lineage link information break includes that the information generation in the lineage link is interrupted.

Images