Big data warehouse governance method based on homework consanguinity knowledge graph
By building a knowledge graph of the blood relationship and a hierarchical architecture for the operation, the problem of insufficient complexity and observability of the operation dependency relationship in the big data warehouse is solved, and efficient and orderly governance and data processing are achieved.
Patent Information
- Application Number
- CN202411389055.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-10-08
AI Technical Summary
In terms of data growth and operation and maintenance management, the existing big data warehouse has problems such as increasing frequency of operation card batches, difficulty in clearly observing and monitoring of operation dependencies, and lack of observable indicator statistics, resulting in low efficiency and incorrect governance work.
Adopt the governance method based on the operation blood relationship knowledge graph, and by collecting and preprocessing metadata, constructing the operation blood relationship knowledge graph, forming a hierarchical structure, realizing source data traceability and batch operation positioning, statistical observability indicators, and optimizing data processing paths and Job dependency.
It effectively solves the problems of rapid operation positioning and observability indicator statistics, improves the efficiency and orderliness of big data warehouse governance, and improves the efficiency and observability of data processing.
Smart Images

Figure CN118939740A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data management technology, and in particular to a big data warehouse governance method based on a job lineage knowledge graph. Background Art
[0002] In the process of promoting the digital transformation of banking business, big data warehouse, as the core technology of data management and analysis, plays a vital role. It can not only integrate multi-channel data resources and improve the operational efficiency of banking business, but also provide solid data support for multiple key areas such as risk management, market analysis, decision support and customer relationship management through efficient data processing and analysis capabilities.
[0003] With the rapid growth of data volume, the pressure on bank big data warehouses is increasing day by day. In addition, since there was no standardized management system in the early days, most of the operation and maintenance work still relied on manual operations, resulting in an increasing frequency of card batches in operation operations.
[0004] The big data warehouse is responsible for data processing and storage. Since data is time-sensitive, it needs to be scheduled in sequence according to the dependencies between jobs. Only when all the current jobs are completed can the latest data be obtained for processing. In the case of limited computing resources, factors such as the architecture design and data quality of the big data warehouse will affect the normal operation of data processing.
[0005] Existing big data warehouses use structured data to record the dependencies between jobs, which makes it difficult to clearly observe the global architecture of the entire data warehouse, resulting in the omission of non-compliant job dependency designs. Secondly, when problems occur, recursive queries are usually used to specify the job path and then check them one by one, which is computationally intensive and difficult to process in batches; at the same time, the current main monitoring target is the real-time status of job operation, which makes it difficult to provide targets for problem prevention and early repair.
[0006] Therefore, it is necessary to provide a big data warehouse governance method based on job lineage knowledge graph to solve the problems in the existing technology that it is difficult to quickly locate problem jobs and lack statistics of observable indicators, so as to make the big data warehouse governance work more efficient and orderly. Summary of the invention
[0007] The purpose of the present invention is to provide a big data warehouse management method based on job lineage knowledge graph to solve the problems in the prior art that it is difficult to quickly locate problem jobs and lack statistics of observable indicators, so as to make the big data warehouse management work more efficient and orderly.
[0008] In order to solve the problems existing in the prior art, the present invention provides a big data warehouse governance method based on job lineage knowledge graph, comprising the following steps: Collect and preprocess metadata of the big data warehouse to form a unique identifier and status table for each job; Based on the unique identifier and status table of each job, a job lineage knowledge graph is constructed; According to the operation lineage knowledge graph, a hierarchical architecture of a big data warehouse is constructed. The hierarchical architecture of the big data warehouse is divided into six layers, namely, the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer. The data transmission path is the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer in turn, and is irreversible; The source data traceability function is implemented based on the hierarchical architecture of the big data warehouse as follows: Among them, G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; ; Where A(v) is the set of all upstream dependent jobs of job v, and job v is the job specified in the job set; is the source data job set of job v; is the in-degree of job v, that is, the number of jobs that the job directly depends on; P(v) is the set of all jobs that job v directly depends on; p belongs to P(v); is the complete set of all source data jobs of job v that directly depend on job P(v); Find the source data job set of the specified job according to A(v); Based on the hierarchical architecture of the big data warehouse, the following method is used to find the stuck batch jobs: Specify the job to be queried, obtain the job set that the job to be queried directly depends on, and merge the directly dependent job set into the unfinished job set; Take out the jobs in the unfinished job set in order. If the completion time of any job in the directly dependent job set of the currently taken out job is greater than the start execution time of the currently taken out job, it means that the currently taken out job is the current stuck batch job; if there is a job in the directly dependent job set of the currently taken out job whose completion time is less than or equal to the start execution time of the currently taken out job, all jobs in the directly dependent job set of the currently taken out job that meet this condition are put into the unfinished job set; Repeat the above steps until all jobs in the unfinished job set are taken out for inspection, and all blocked batch jobs at the current time can be found.
[0009] Optionally, in the big data warehouse governance method based on job lineage knowledge graph, The unique identifier consists of the job-to-system field and the job name field for each job; The job status table includes the system to which the job belongs, the job name, data date, job start time, job end time and job completion status.
[0010] Optionally, in the big data warehouse governance method based on the job lineage knowledge graph, the job lineage knowledge graph contains three types of entities and two types of associated edges. The three types of entities are database nodes, system nodes, and job nodes, and the two types of associated edges are subordinate relationships and dependent relationships.
[0011] Optionally, in the big data warehouse governance method based on job lineage knowledge graph, The source data layer contains copies of original data, which comes from various business systems, external data sources, and other data warehouses. Other data warehouses are data warehouses other than the main data warehouse. The source layer is used to complete data extraction, conversion and loading from the source data layer to the big data warehouse; The historical detail layer is used to store historical data versions to support time series analysis; The standard layer is responsible for data standardization; The basic marketplace layer divides data into different subject domains according to business needs, including customers, accounts, and transactions; The application marketplace layer provides data views and services for specific business units or applications.
[0012] Optionally, the big data warehouse governance method based on the job lineage knowledge graph further includes the following steps: Statistics on observability indicators of big data warehouse operation and maintenance, including: average running time of Top 100 jobs, job depth, average start time of source data layer jobs in the past month, and number of days source data layer jobs were late in starting in the past month.
[0013] Optionally, in the big data warehouse governance method based on the job lineage knowledge graph, for the abnormal situation that the source data layer jobs are delayed for a certain number of days in the past month, the longest processing path and the latest processing path of the delayed jobs are found.
[0014] Optionally, in the big data warehouse governance method based on the job lineage knowledge graph, based on the big data warehouse hierarchical architecture, the longest processing path of the delayed job is found in the following manner: , where G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; Calculate the longest processing time and the longest processing path: ; ; Wherein, job v is the job specified in the job set; L(v) is the longest processing time of job v; LP(v) is the longest processing path of job v; T(v) is the average processing time of job v; is the in-degree of job v, that is, the number of jobs that the job directly depends on; P(v) is the set of all jobs that job v directly depends on; L(p) is the longest processing time of job p, a direct dependency of job v; is the source data job set of job v; The job that takes the longest processing time among the set of directly dependent jobs of job v; It is the longest processing path in the set of directly dependent jobs of job v.
[0015] Optionally, in the big data warehouse governance method based on the job lineage knowledge graph, based on the big data warehouse hierarchical architecture, the latest processing path of the late job is found in the following manner: ; Where WP(v) represents the latest processing path; E(p) represents the average completion time of job p, P(v) is the set of all directly dependent jobs of job v, and job p belongs to P(v); It is the job with the largest E(p) among the set of directly dependent jobs of job v.
[0016] In the big data warehouse governance method based on job lineage knowledge graph provided by the present invention, based on the dependency relationship between batch jobs, a job lineage knowledge graph is constructed to sort out the complex processing paths inside the data warehouse. In addition, the present invention also realizes the location function of problem jobs and the statistics of observability indicators, effectively solving the problem of lack of direction and grip in big data warehouse governance, making governance more efficient and orderly. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A schematic diagram of a big data warehouse management process provided by an embodiment of the present invention; Figure 2 A schematic diagram of a hierarchical architecture of a big data warehouse provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.
[0019] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0020] Existing big data warehouses use structured data to record the dependencies between jobs, which makes it difficult to clearly observe the global architecture of the entire data warehouse, resulting in the omission of non-compliant job dependency designs. Secondly, when problems occur, recursive queries are usually used to specify the job path and then check them one by one, which is computationally intensive and difficult to process in batches; at the same time, the current main monitoring target is the real-time status of job operation, which makes it difficult to provide targets for problem prevention and early repair.
[0021] In order to solve the problems existing in the prior art, the present invention provides a big data warehouse governance method based on job lineage knowledge graph. The present invention constructs a lineage knowledge graph between jobs, sorts out the global call path of jobs within the big data warehouse, and makes improvements in multiple aspects such as data architecture, problem location and observable indicators, thus providing direction and grip for big data warehouse governance. The overall application scenario flow chart is shown in the figure below: Figure 1 Specifically, the big data warehouse governance method includes the following steps: S1: Collect and preprocess the metadata of the big data warehouse to form a unique identifier and status table for each job; Collect data related to batch jobs in the big data warehouse, including basic job information, job directory, job dependencies, job current status, and job running history information.
[0022] Furthermore, the existing big data warehouse job dependencies are recorded in the database in the form of structured data. Each piece of data represents a job dependency. Each job is uniquely identified by two fields, namely the job system and the job name. The job dependency representation is shown in Table 1. The job status table records the historical running information of each job. The main information includes the job system, job name, data date used by the job, job start time, job end time and job completion status. The job status representation is shown in Table 2.
[0023] Table 1 Example of job dependency table Table 2 Example of job status table Preferably, preprocessing also includes data cleaning. During data cleaning, for the job dependency table, it is necessary to clean up the data that does not conform to the specification according to the system and name naming rules of the job, and finally clean up the duplicate data. For the job status table, it is necessary to process the data format, including job naming, date format, status enumeration value range, and remove duplicate data.
[0024] The preprocessing here is for the metadata of the big data warehouse. Metadata refers to the information of each table in the warehouse (such as table name, status, etc.), rather than the specific data in the table, and is independent of the specific data.
[0025] S2: Based on the unique identifier and status table of each job, a job lineage knowledge graph is constructed; Due to system requirements, data characteristics or historical factors, banks usually deploy multiple databases and data processing platforms, and there is necessary data interaction between these platforms. Sorting out the operation links within each data processing platform and between different platforms will help build a comprehensive knowledge graph, and then form a full path display from data source to data processing to final report, so as to ensure the integrity and accuracy of the analysis results. The lineage knowledge graph is mainly built based on the dependency relationship between operations, and other information is incorporated into node attributes or relationship attributes to sort out the complex global operation processing logic of the big data warehouse.
[0026] Specifically, the job lineage knowledge graph contains three types of entities and two types of associated edges. The three types of entities are database nodes, system nodes, and job nodes, and the two types of associated edges are subordinate relationships and dependent relationships. The entity and associated edge structures are shown in Table 3.
[0027] Table 3 Example of job dependency table S3: Based on the job lineage knowledge graph, a hierarchical architecture of the big data warehouse is constructed. The hierarchical architecture of the big data warehouse is crucial because it effectively improves the efficiency and value of the data warehouse by providing advantages in scalability, data lineage tracking, and strengthening data security and compliance.
[0028] In building the hierarchical architecture of the big data warehouse, a big data warehouse hierarchical strategy was formulated, and a reasonable hierarchical scheme was planned. The big data warehouse was divided into six levels based on the nature of the operation, reusability, development function, etc. According to the set rules, the non-compliant operation dependencies in the big data warehouse were reversely checked and handed over to the operation leader for adjustment.
[0029] like Figure 2 As shown, the hierarchical architecture of the big data warehouse is divided into six layers, namely the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer. The data transmission path is the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer in sequence, and is irreversible; Specifically, the source data layer contains copies of the original data, which comes from various business systems, external data sources and other data warehouses. Other data warehouses are data warehouses other than this big data warehouse; the source layer is used to complete data extraction, conversion and loading from the source data layer to the big data warehouse; the historical detail layer is used to store historical data versions to support time series analysis; the standard layer is responsible for data standardization, including data cleaning, conversion, merging, etc., to ensure data consistency and accuracy; the basic mart layer divides data into different subject domains according to business needs, including customers, accounts, and transactions; the application mart layer provides data views and services for specific business departments or applications.
[0030] In the above layered architecture, there are restrictions on the data transmission paths between different layers (that is, limiting non-compliant job dependencies), which can ensure the orderliness of data flow and the efficiency of data processing, including the following situations: data flows unidirectionally from the lower layer to the higher layer, that is, from the source data layer to the application mart layer, and cannot flow in the opposite direction; data moves in a predetermined path sequence to avoid directly skipping the middle layer from the lower layer to the higher layer; data is processed repeatedly between different layers as much as possible; different jobs in the application mart layer are not dependent on each other; there can only be one dependency relationship between any two jobs.
[0031] S4: The source data traceability function is implemented based on the hierarchical architecture of the big data warehouse. The job traceability function can provide a quick query function for the source data of a specified job.
[0032] The way to implement the source data traceability function is as follows: , where G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; ; Where A(v) is the set of all upstream dependent jobs of job v, and job v is the job specified in the job set; is the in-degree of job v, that is, the number of jobs that the job directly depends on; is the set of source data jobs of job v, that is, the starting job on the processing link, and its in-degree is 0; P(v) is the set of all directly dependent jobs of job v; p belongs to P(v); It is the complete set of all source data jobs of all directly dependent jobs P(v) of job v. U can represent a universal set or a complete set in mathematical symbols. Since p belongs to the set P(v) of all directly dependent jobs of v, A(p) represents the set of source data jobs of a job in P(v).
[0033] Find the source data job set of the specified job according to A(v); S5: Based on the hierarchical architecture of the big data warehouse, the stuck batch job is found. The stuck batch job location function helps the downstream job manager to directly locate the current stuck batch job, eliminating the process of asking questions at different levels in the middle, and improving the efficiency of problem handling.
[0034] The way to find the card batch operation is as follows: Specify the job w to be queried, obtain the job set P(w) that the job to be queried directly depends on, and merge the directly dependent job set into the unfinished job set WorkStack; Take out the jobs w from the unfinished job set in order c , if any job w in the set of directly dependent jobs P(w) of the currently retrieved job i The completion time E(w i ) are both greater than the current fetch job w c The start execution time S(w c ), then it means the currently retrieved job w c This is the current batch job; if the current take-out job w c There is a job whose completion time is less than or equal to the current job w in the set of directly dependent jobs P(w). c The start execution time S(w c )'s homework i , then all jobs w that meet this condition in the set of directly dependent jobs of the current job are taken out i Put it into the unfinished job collection WorkStack; Repeat the above steps until all jobs in the unfinished job set are taken out for inspection, and all blocked batch jobs at the current time can be found.
[0035] In one embodiment, the specific algorithm is as follows: 1: Input: Graph , job execution start time S(v), job execution end time E(v) 2: Job 3 to be queried: Directly dependent job set 4: 5: Non-empty do: 6: The first assignment in 7: Directly dependent job set 8: The default value for batch operation is 'batch operation' 9: : 10: if : 11: 12: Move into 13: end if 14:end for Optionally, the big data warehouse governance method based on the job lineage knowledge graph further includes the following steps: S6: For abnormal situations where jobs are delayed, find the longest processing path and the latest processing path of the delayed jobs, that is, analyze the observability indicators of the big data warehouse.
[0036] Optionally, the big data warehouse governance method based on the job lineage knowledge graph further includes the following steps: S6: For abnormal situations where jobs are delayed, find the longest processing path and the latest processing path of the delayed jobs, that is, analyze the observability indicators of the big data warehouse.
[0037] S7: In response to the abnormal situation that the source data layer jobs have been delayed for several days since the beginning of the month, the present invention explores the theoretical and actual delay cause paths, that is, the longest processing path and the latest processing path of the delayed jobs, to find out the causes of delays and optimize the jobs (i.e., problem processing path analysis).
[0038] S71: Based on the hierarchical architecture of the big data warehouse, find the longest processing path for the delayed operation in the following way: , where G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; calculate the longest processing time and the longest processing path: ; ; Wherein, job v is the job specified in the job set; L(v) is the longest processing time of job v; LP(v) is the longest processing path of job v; T(v) is the average processing time of job v; is the in-degree of job v, that is, the number of jobs that the job directly depends on; P(v) is the set of all jobs that job v directly depends on; L(p) is the longest processing time of job p, a direct dependency of job v; is the source data job set of job v; The job that takes the longest processing time among the set of directly dependent jobs of job v; It is the longest processing path in the set of directly dependent jobs of job v.
[0039] This formula can be used to find the longest time required for job processing. When the start time of jobs in the source data layer is equal, optimizing the jobs on this path can effectively advance the completion time of the target job.
[0040] S72: Based on the hierarchical architecture of the big data warehouse, find the latest processing path for the delayed operation in the following way: ; Where WP(v) represents the latest processing path; E(p) represents the average completion time of job p, P(v) is the set of all directly dependent jobs of job v, and job p belongs to P(v); It is the job with the longest average completion time among the set of directly dependent jobs of job v.
[0041] The latest processing path focuses on the latest processing path that causes the target job to be delayed during the actual operation. The target job delay can be effectively alleviated by optimizing the job completion time in the path, increasing the batch priority, or advancing the source data layer job start time.
[0042] In summary, in the big data warehouse governance method based on job lineage knowledge graph provided by the present invention, based on the dependency relationship between batch jobs, a job lineage knowledge graph is constructed to sort out the complex processing paths inside the data warehouse. In addition, the present invention also realizes the positioning function of problem jobs and the statistics of observability indicators, effectively solving the problem of lack of direction and grip in big data warehouse governance, making governance more efficient and orderly.
[0043] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A big data warehouse management method based on job lineage knowledge graph, characterized in that: The following steps are involved: Collect and preprocess metadata of the big data warehouse to form a unique identifier and status table for each job; Based on the unique identifier and status table of each job, a job lineage knowledge graph is constructed; According to the operation lineage knowledge graph, a hierarchical architecture of a big data warehouse is constructed. The hierarchical architecture of the big data warehouse is divided into six layers, namely, the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer. The data transmission path is the source data layer, the source layer, the historical detail layer, the standard layer, the basic market layer and the application market layer in turn, and is irreversible; The source data traceability function is implemented based on the hierarchical architecture of the big data warehouse as follows: , where G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; ; Where A(v) is the set of all upstream dependent jobs of job v, and job v is the job specified in the job set; is the source data job set of job v; is the in-degree of job v, that is, the number of jobs that the job directly depends on; P(v) is the set of all jobs that job v directly depends on; p belongs to P(v); is the complete set of all source data jobs of job v that directly depend on job P(v); Find the source data job set of the specified job according to A(v); Based on the hierarchical architecture of the big data warehouse, the following method is used to find the stuck batch jobs: Specify the job to be queried, obtain the job set that the job to be queried directly depends on, and merge the directly dependent job set into the unfinished job set; Take out the jobs in the unfinished job set in order. If the completion time of any job in the directly dependent job set of the currently taken out job is greater than the start execution time of the currently taken out job, it means that the currently taken out job is the current stuck batch job; if there is a job in the directly dependent job set of the currently taken out job whose completion time is less than or equal to the start execution time of the currently taken out job, all jobs in the directly dependent job set of the currently taken out job that meet this condition are put into the unfinished job set; Repeat the above steps until all jobs in the unfinished job set are taken out for inspection, and all blocked batch jobs at the current time can be found.
2. The big data warehouse management method based on job lineage knowledge graph according to claim 1 is characterized in that: The unique identifier consists of the job-to-system field and the job name field for each job; The job status table includes the system to which the job belongs, the job name, data date, job start time, job end time and job completion status.
3. The big data warehouse management method based on job lineage knowledge graph as claimed in claim 1 is characterized in that: The job lineage knowledge graph includes three types of entities and two types of associated edges. The three types of entities are database nodes, system nodes, and job nodes, and the two types of associated edges are subordinate relationships and dependent relationships.
4. The big data warehouse management method based on job lineage knowledge graph according to claim 1 is characterized in that: The source data layer contains copies of original data, which comes from various business systems, external data sources, and other data warehouses. Other data warehouses are data warehouses other than the main data warehouse. The source layer is used to complete data extraction, conversion and loading from the source data layer to the big data warehouse; The historical detail layer is used to store historical data versions to support time series analysis; The standard layer is responsible for data standardization; The basic marketplace layer divides data into different subject domains according to business needs, including customers, accounts, and transactions; The application marketplace layer provides data views and services for specific business units or applications.
5. The big data warehouse management method based on job lineage knowledge graph according to claim 1 is characterized in that: The following steps are also included: Statistics on observability indicators of big data warehouse operation and maintenance, including: average running time of Top 100 jobs, job depth, average start time of source data layer jobs in the past month, and number of days source data layer jobs were late in starting in the past month.
6. The big data warehouse management method based on job lineage knowledge graph according to claim 5 is characterized in that: For the abnormal number of days that the source data layer jobs have been delayed in the past month, find the longest processing path and the latest processing path of the delayed jobs.
7. The big data warehouse management method based on job lineage knowledge graph according to claim 6 is characterized in that: Based on the hierarchical architecture of the big data warehouse, the longest processing path for delayed operations is found as follows: , where G is the job lineage knowledge graph, V is the set of jobs, and E is the set of dependencies between jobs; Calculate the longest processing time and the longest processing path: ; ; Wherein, job v is the job specified in the job set; L(v) is the longest processing time of job v; LP(v) is the longest processing path of job v; T(v) is the average processing time of job v; is the in-degree of job v, that is, the number of jobs that the job directly depends on; P(v) is the set of all jobs that job v directly depends on; L(p) is the longest processing time of job p, a direct dependency of job v; is the source data job set of job v; The job that takes the longest processing time among the set of directly dependent jobs of job v; It is the longest processing path in the set of directly dependent jobs of job v.
8. The big data warehouse management method based on job lineage knowledge graph according to claim 7 is characterized in that: Based on the hierarchical architecture of the big data warehouse, the latest processing path for late operations is found as follows: ; Where WP(v) represents the latest processing path; E(p) represents the average completion time of job p, P(v) is the set of all directly dependent jobs of job v, and job p belongs to P(v); It is the job with the largest E(p) among the set of directly dependent jobs of job v.
Citation Information
Patent Citations
Metadata knowledge graph construction method, storage medium and system
CN114880483A
Service report integration method and apparatus, device, and storage medium
WO2022134363A1
