Data management method and system and storage medium

Abstract

The invention relates to a data management method and system and a storage medium. The method comprises the steps that an SQL script of multi-source heterogeneous data is optimized through an SQL optimization model, and an optimized SQL script and an optimization description of the optimized SQL script are obtained; repairing metadata in the multi-source heterogeneous data through a metadata compliance model to obtain optimized metadata of the multi-source heterogeneous data; and performing attribution analysis on the abnormal SQL scripts in the SQL scripts through the SQL attribution model to obtain an attribution result of the abnormal SQL scripts, and optimizing the scheme. According to the scheme provided by the invention, automation of data governance can be realized, the data governance efficiency and accuracy are improved, the manual intervention cost of data governance is reduced, and the adaptability of data governance to a complex data environment is enhanced.

Description

Technical Field

[0001] This application relates to the fields of data governance and artificial intelligence technology, and in particular to a data governance method, system and storage medium. Background Technology

[0002] Data governance refers to a series of management activities and technical means to ensure the accuracy, consistency, integrity, security and availability of data.

[0003] Data governance related technologies mainly focus on data quality management, metadata management, and lineage analysis. They lack a unified modeling and standardized processing mechanism for multi-source heterogeneous data, have limited automatic analysis and optimization capabilities for SQL scripts, rely on preset rules for data governance strategies, have poor flexibility and weak adaptability, and have not fully incorporated artificial intelligence technology, thus failing to achieve automatic learning and continuous optimization of data governance.

[0004] Data governance for related technologies faces challenges. When dealing with a large number of SQL (Structured Query Language) scripts and ETL (Extract-Transform-Load) tasks, it is difficult to quickly determine whether there are syntax errors, performance bottlenecks, or logical defects. Experienced engineers still need to manually check for issues such as data quality and unreasonable storage structures. It is also difficult to uniformly manage data from multiple data sources such as relational databases, big data platforms, and log systems. For error messages during data processing, non-professionals often cannot accurately identify the causes and propose effective solutions.

[0005] Therefore, the data governance of related technologies is difficult to cope with multi-source heterogeneous data environments, lacks intelligent means to assist decision-making, and has high labor costs and low efficiency. Summary of the Invention

[0006] To address or partially address the problems existing in related technologies, this application provides a data governance method, system, and storage medium that can automate data governance, improve data governance efficiency and accuracy, reduce the cost of manual intervention in data governance, and enhance the adaptability of data governance to complex data environments.

[0007] A first aspect of this application provides a data governance method, the method comprising: The SQL scripts for multi-source heterogeneous data are optimized using an SQL optimization model to obtain optimized SQL scripts and optimization descriptions of the optimized SQL scripts. The metadata in the multi-source heterogeneous data is repaired by using a metadata compliance model to obtain optimized metadata for the multi-source heterogeneous data; An attribution analysis of abnormal SQL scripts in an SQL script is performed using an SQL attribution model to obtain the attribution results and optimization schemes for the abnormal SQL scripts.

[0008] In one implementation, before optimizing the SQL script for multi-source heterogeneous data using an SQL optimization model to obtain the optimized SQL script and its optimization description, the process includes: Using a TOP-level LSTM model, based on the historical TOP-level data collection volume of the TOP-level multi-source heterogeneous data collected by the data governance system, the predicted TOP-level data collection volume of the TOP-level multi-source heterogeneous data at the target time of the data governance system is predicted. Using a business domain-level LSTM model, based on the historical data collection volume of business domain-level multi-source heterogeneous data collected by the data governance system and the predicted data collection volume of the top table level data of the business domain, the predicted data collection volume of business domain-level multi-source heterogeneous data collected by the data governance system at the target time is predicted. Using a system-level LSTM model, based on the system-level historical collection volume of system-level multi-source heterogeneous data collected by the data governance system and the predicted collection volume of business domain-level data of the data governance system, the predicted collection volume of system-level multi-source heterogeneous data collected by the data governance system at the target time is predicted. The predicted collection volume S is calculated based on the predicted collection volume of TOP table-level data, business domain-level data, and system-level data. (t) Among them, S (t) =W 1(t-1) ×S 1(t) +W 2(t-1) ×S 2(t) +W 3(t-1) ×S 3(t) W 1(t-1) The system-level data prediction and collection volume S represents the amount of data collected. 1(t) The first dynamic weighting coefficient; W 2(t-1) The data collection volume S at the business domain level is predicted. 2(t) The second dynamic weighting coefficient; W 3(t-1) This indicates the predicted data collection volume S at the TOP table level. 3(t) The third dynamic weighting coefficient; The data governance system pre-allocates system resources at the target time based on the predicted collection volume; If the predicted data collection amount exceeds the set data volume limit, a resource warning signal will be issued.

[0009] In one implementation, the predicted collection volume S is calculated based on the predicted collection volume of TOP table-level data, the predicted collection volume of business domain-level data, and the predicted collection volume of system-level data. (t)Previously, the process included: calculating the first dynamic weight coefficient, the second dynamic weight coefficient, and the third dynamic weight coefficient based on the first mean square error of the system-level data prediction collection amount at time t-1 predicted by the system-level LSTM model, the second mean square error of the business domain-level data prediction collection amount at time t-1 predicted by the business domain-level LSTM model, and the third mean square error of the TOP table-level data prediction collection amount at time t-1 predicted by the TOP table-level LSTM model.

[0010] In one embodiment, the method further includes: If the first average absolute percentage error of the system-level data prediction collection volume predicted by the system-level LSTM model is greater than or equal to the first set error threshold, then the second average absolute percentage error is greater than or equal to the second set error threshold. The second average absolute percentage error is the error of the business domain-level LSTM model predicting the business domain-level data prediction collection volume, and the business domain is determined to be an abnormal business domain. If the second average absolute percentage error of the business domain-level data prediction collection volume predicted by the business domain-level LSTM model is greater than or equal to the second set error threshold, then the third average absolute percentage error, which is greater than or equal to the third set error threshold, is obtained. The third average absolute percentage error is the error of the TOP table-level data prediction collection volume of the TOP table by the TOP table-level LSTM model, and the TOP table is determined to be an abnormal TOP table.

[0011] In one implementation, the step of optimizing the SQL script for multi-source heterogeneous data using an SQL optimization model to obtain an optimized SQL script and an optimization description of the optimized SQL script includes: The SQL optimization model includes an input layer, an encoding layer, a diagnostic layer, a generation layer, and an output layer. The input layer obtains the SQL text, execution plan, and metadata tags of the SQL script; The encoding layer generates the SQL semantic vector of the SQL script through a pre-trained language model, extracts local features of the execution plan through an execution plan feature extractor, and outputs a joint feature vector concatenated with the SQL semantic vector and the local features. The diagnostic layer, based on the joint feature vector of the encoding layer, outputs the optimization type of the SQL script through a multi-label classification task, and locates the problem location of the SQL script through sequence labeling. The generation layer, through the decoder, generates the optimized SQL script based on the optimization type and problem location of the diagnostic layer and the joint feature vector of the encoding layer, constrained by Hive optimization rules. The output layer outputs the optimized SQL script and an optimization description of the optimized SQL script.

[0012] In one implementation, the step of repairing the metadata in the multi-source heterogeneous data through a metadata compliance model to obtain optimized metadata of the multi-source heterogeneous data includes: Through the metadata compliance model and based on the metadata mapping knowledge base, the metadata in the multi-source heterogeneous data is standardized to obtain the standardized metadata in the multi-source heterogeneous data. According to preset metadata rules, the standardized metadata is repaired to obtain optimized metadata of the multi-source heterogeneous data. The repair includes at least one of completion, correction, cleaning, and deduplication.

[0013] In one implementation, the step of performing attribution analysis on abnormal SQL scripts in an SQL script using an SQL attribution model to obtain the attribution results and optimization schemes for the abnormal SQL scripts includes: The SQL optimization model is trained using the SQL script's execution logs to obtain the SQL attribution model; Based on the execution log of the SQL script, identify the SQL scripts with a runtime longer than or equal to the set duration or those that report an error as the abnormal SQL scripts; The abnormal SQL script is analyzed using the SQL attribution model to obtain the attribution results and optimization schemes for the abnormal SQL script.

[0014] In one embodiment, the method further includes: performing a data quality assessment on the multi-source heterogeneous data collected by the data governance system at a target time, and obtaining a quality score Score for the multi-source heterogeneous data, wherein Score = A × (S ′ (t) ÷S (t) ) + B × (R ′ (t) ÷R (t) )+C×(I ′ (t) (÷100), A, B, and C represent quality rating coefficients, S ′ (t) S represents the number of successfully acquired multi-source heterogeneous data at the target time. (t) R represents the predicted acquisition volume of multi-source heterogeneous data at the target time. ′ (t) R represents the number of valid ETL tasks at the target time. (t) I represents the total number of ETL tasks at the target time. ′ (t)This indicates the metadata completeness of metadata in the multi-source heterogeneous data at the target time.

[0015] A second aspect of this application provides a data governance system, the system comprising: The script optimization unit is used to optimize the SQL scripts of multi-source heterogeneous data through the SQL optimization model, and obtain the optimized SQL scripts and the optimization descriptions of the optimized SQL scripts. The metadata optimization unit repairs the metadata in the multi-source heterogeneous data through a metadata compliance model to obtain optimized metadata of the multi-source heterogeneous data. The script analysis unit is used to perform attribution analysis on abnormal SQL scripts in the SQL script using the SQL attribution model, and to obtain the attribution results and optimization schemes of the abnormal SQL scripts.

[0016] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0018] The fifth aspect of this application provides a computer program product comprising computer instructions that, when executed by a processor, implement the method described above.

[0019] The technical solution provided in this application may include the following beneficial results: The technical solution of this application combines artificial intelligence, natural language processing, code understanding and other technologies to automatically identify, diagnose and optimize key links in the data governance process. It can realize the automation of data governance, improve the efficiency and accuracy of data governance, reduce the cost of manual intervention in data governance, and enhance the adaptability of data governance to complex data environments.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0022] Figure 1 This is a flowchart illustrating the data governance method in an embodiment of this application; Figure 2 This is another schematic diagram of the data governance method shown in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the data governance system shown in the embodiments of this application; Figure 4 This is another structural schematic diagram of the data governance system shown in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0023] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0024] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0026] This application provides a data governance method that can automate data governance, improve data governance efficiency and accuracy, reduce the cost of manual intervention in data governance, and enhance the adaptability of data governance to complex data environments.

[0027] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1This is a flowchart illustrating the data governance method in an embodiment of this application.

[0029] See Figure 1 A data governance approach, comprising: Step 101: Optimize the SQL scripts of multi-source heterogeneous data using the SQL optimization model to obtain the optimized SQL scripts and optimization instructions.

[0030] In one embodiment, a pre-trained SQL optimization model can deeply understand the mapping relationship between the syntax and execution logic of SQL scripts. Based on the SQL scripts in the input multi-source heterogeneous data, the SQL optimization model optimizes the SQL scripts to obtain optimized SQL scripts and optimization descriptions.

[0031] Step 102: Repair the metadata in the multi-source heterogeneous data through the metadata compliance model to obtain optimized metadata of the multi-source heterogeneous data.

[0032] In one embodiment, a trained metadata compliance model is used to repair metadata in multi-source heterogeneous data, correcting errors, redundancies, and missing data, and obtaining optimized metadata for multi-source heterogeneous data.

[0033] Step 103: Perform attribution analysis on abnormal SQL scripts in the SQL script using the SQL attribution model to obtain the attribution results and optimization solutions for the abnormal SQL scripts.

[0034] The data governance method of this application combines artificial intelligence, natural language processing, code understanding and other technologies to automatically identify, diagnose and optimize key links in the data governance process. It can realize the automation of data governance, improve the efficiency and accuracy of data governance, reduce the cost of manual intervention in data governance, and enhance the adaptability of data governance to complex data environments.

[0035] Figure 2 This is another schematic diagram of the data governance method shown in the embodiments of this application. Figure 2 Compared to Figure 1 The scheme of this application is described in more detail. See also Figure 2 A data governance approach, comprising: Step 210: Collect multi-source heterogeneous data from different data sources.

[0036] In one embodiment, multi-source heterogeneous data, such as metadata, log data, and script data, from different data sources, such as PostgreSQL (relational database management system) database, MySQL (relational database management system) database, and data platform, can be collected through collection tools such as API (Application Programming Interface), DataX (heterogeneous data source offline synchronization tool), Flume (log collection system), and Kafka (distributed streaming processing platform).

[0037] Step 220: Based on the historical collection volume of multi-source heterogeneous data collected by the data governance system, predict the predicted collection volume of multi-source heterogeneous data collected by the data governance system at the target time, pre-allocate system resources of the data governance system at the target time according to the predicted collection volume, and issue a resource warning signal when the predicted collection volume exceeds the set data volume limit.

[0038] In one embodiment, the multi-source heterogeneous data includes table-level data, business domain-level data, and system-level data.

[0039] Table-level data collection focuses on database tables as the smallest unit, examining data collection at the level of individual tables within the database. It focuses on the fields, data types, and relationships between tables within the table structure, representing the most basic and granular level of data collection. Business domain-level data, on the other hand, is collected from a business perspective, organizing and collecting data around specific business domains (e.g., finance, customer, R&D, operations). It emphasizes the meaning and value of data within the business context and serves as a crucial basis for data governance and classification. System-level data is collected at the level of the entire system or platform, including system logs, operational status, and performance metrics, used for monitoring, maintenance, and system optimization.

[0040] In one embodiment, the target time can be a time in days. This is achieved using a TOP-level LSTM (Long Short-Term LSTM). The system employs a Long Short-Term Memory (LSTM) model to predict the target data collection volume of TOP-level heterogeneous data at the data governance system's timeframe, based on the historical data collection volume of TOP-level heterogeneous data from multiple sources collected by the data governance system. It also uses a business domain-level LSTM model to predict the target data collection volume of business domain-level heterogeneous data at the data governance system's timeframe, based on the historical data collection volume of business domain-level heterogeneous data and the predicted data collection volume of TOP-level data from the business domain. Furthermore, it uses a system-level LSTM model to predict the target data collection volume of system-level heterogeneous data at the data governance system's timeframe, based on the historical data collection volume of system-level heterogeneous data and the predicted data collection volume of business domain-level data from the data governance system. Finally, it calculates the predicted collection volume based on the predicted data collection volume of TOP-level, business domain-level, and system-level data. Based on the predicted collection volume, it pre-allocates system resources for the data governance system at the target time. If the predicted collection volume exceeds a set data volume limit, it issues a resource warning signal.

[0041] In one embodiment, the predicted collection volume of multi-source heterogeneous data collected by the data governance system at a target time is S. (t) , among which, S (t) =W 1(t-1) ×S 1(t) +W 2(t-1) ×S 2(t) +W 3(t-1) ×S 3(t) W 1(t-1) S represents the system-level data prediction and collection volume. 1(t) The first dynamic weighting coefficient; W 2(t-1) S represents the business domain-level data prediction and collection volume. 2(t) The second dynamic weighting coefficient; W 3(t-1) S represents the predicted data collection volume for the TOP table level. 3(t) The third dynamic weighting coefficient.

[0042] In one embodiment, the first dynamic weight coefficient + the second dynamic weight coefficient + the third dynamic weight coefficient = 1. The initial value of the first dynamic weight coefficient can be 0.4, the initial value of the second dynamic weight coefficient can be 0.4, and the initial value of the third dynamic weight coefficient can be 0.3. The first dynamic weight coefficient, the second dynamic weight coefficient, and the third dynamic weight coefficient are calculated based on the first mean square error of the system-level data prediction collection at time t-1 of the system-level LSTM model, the second mean square error of the business domain-level data prediction collection at time t-1 of the business domain-level LSTM model, and the third mean square error of the TOP table-level data prediction collection at time t-1 of the TOP table-level LSTM model.

[0043] In one embodiment, the first mean square error of the system-level data prediction acquisition at time t-1 using the system-level LSTM model is MSE. sys(t-1) The second mean square error of the business domain-level data prediction acquisition amount at time t-1 using the business domain-level LSTM model is MSE. dom(t-1) The third mean square error of the TOP-level LSTM model prediction data collection at time t-1 is MSE. top(t-1) .

[0044] System-level data prediction collection volume S 1(t) The first dynamic weighting coefficient is W 1(t-1) ,in, , Represents a small constant. .

[0045] Business domain-level data prediction collection volume S 2(t) The second dynamic weighting coefficient is W 2(t-1) ,in, .

[0046] S of TOP-level data prediction collection volume 3(t) The third dynamic weighting coefficient is W 3(t-1) ,in, .

[0047] In one embodiment, the TOP table refers to the top 5% of key data tables in terms of daily data collection volume (e.g., order tables, user tables, and other core business tables). A business domain refers to a data domain divided according to business logic (e.g., financial domain, bill domain, budget domain), and each business domain contains several data tables (including TOP tables and non-TOP tables). The system refers to the entire data governance system or data platform, encompassing all business domains and all data tables.

[0048] In one embodiment, the system-level LSTM model, business domain-level LSTM model, and TOP table-level LSTM model can be pre-trained LSTM models. LSTM is a special type of recurrent neural network, adept at processing and predicting long-term dependencies in time series data. The dataset for the pre-trained LSTM model includes multi-source heterogeneous data collected through different acquisition tools. This multi-source heterogeneous data can include raw data and derived data. The raw data includes table names, business domains, business systems, time, acquisition dates, and events. The derived data includes TOP table-level data acquisition volume (the top 5% of data tables collected daily), business domain-level data acquisition volume, and system-level data acquisition volume. The dataset is then cleaned to obtain a cleaned dataset. Data cleaning includes using interpolation to restore the temporal continuity of the data and labeling outlier samples. For example, if a dataset has missing data, such as a missing daily data collection volume for a single data form, and the number of consecutive missing days is less than or equal to 3 days, the average of the data collection volumes from the previous 7 days will be used to fill the gap. If the number of consecutive missing days is greater than 3 days, it will be marked as a "data gap," and interpolation methods (e.g., linear interpolation) will be used to restore the temporal continuity of the daily data collection volume for the single data form. If the business domain-level data collection volume for a certain business domain is missing, an alarm will be triggered. After confirming whether the data collection tool is faulty, the data collection tool will be repaired and the data will be reconnected.

[0049] In one embodiment, for anomalous data in the dataset, if the anomalous data is static, the IQR (interquartile range) rule is used to identify extreme values ​​of the data collection volume of a single data table (e.g., exceeding the mean by ±3 standard deviations). This static anomalous data is marked as is_exception=1 and is not directly deleted (to avoid losing real business fluctuations, such as peak periods during promotional events), but rather modeled separately as an "abnormal sample." If the anomalous data is dynamic, Isolation Forest (an unsupervised anomaly detection algorithm based on ensemble learning) is used to detect abrupt changes in the continuous data collection volume of a single data table (e.g., the data collection volume of a data table suddenly drops from 100,000 to 10,000). This is combined with the event_tag to determine whether it is a known event (e.g., "system rate limiting"). Unknown events are then included in the event features.

[0050] In one embodiment, the features of the system-level LSTM model, the business domain-level LSTM model, and the TOP table-level LSTM model adopt a three-level feature penetration design, so that the features of the system-level LSTM model, the business domain-level LSTM model, and the TOP table-level LSTM model satisfy "layer-specific + cross-layer sharing". The specific features of the system-level LSTM model, the business domain-level LSTM model, and the TOP table-level LSTM model are shown in Table 1 below.

[0051] Table 1:

[0052] For time features, extract the weektime position of dt (date) (e.g., Monday = 1), determine whether it is the end of the month, the end of the quarter, or a holiday by associating with a holiday table, and convert it into periodic features using one-hot encoding or sine / cosine encoding. For event features, convert event_tags (e.g., "618 Big Sale", "Weekly Wednesday Data Synchronization") into binary features (0 / 1), or use a time decay function (e.g., weights gradually decrease within 3 days after the event) to enhance timeliness.

[0053] In one embodiment, differentiated system-level LSTM models, business domain-level LSTM models, and TOP-table-level LSTM models can be designed based on the data characteristics of TOP-table-level data collection, business domain-level data collection, and system-level data collection (e.g., TOP-table-level data collection fluctuates more frequently, while system-level data collection is more stable). The correlation between these models is enhanced through an "attention mechanism + hierarchical constraints." The data collection volume of a single data table is affected by its own periodicity (e.g., daily incremental patterns) and business events (e.g., a surge in orders on promotional days), resulting in large fluctuations and high noise levels. The TOP-table-level LSTM model can capture the fine-grained fluctuations of a single data table. The TOP-table-level LSTM model includes an input layer, a core layer, and an output layer.

[0054] The input layer includes the time-series features (collect_vol), temporal features, and business features of the data table (a specific TOP table) for the previous 14 days. The input dimension is 14 × (1 + 3 + 2) = 84, where 1 represents the data collection volume, 3 represents the temporal features, and 2 represents the business features. The specific representations of symbols 1, 3, and 2 are shown in Table 2 below.

[0055] Table 2:

[0056] The core layers consist of a Bidirectional LSTM (BiLSTM) and an attention layer. The Bidirectional LSTM (2 layers, 64 hidden dimensions) captures the bidirectional dependency between past data collection volume and future data collection volume, and vice versa (e.g., data collection volume for a data table increases 3 days before a major promotion and decreases 2 days after). The attention layer uses an attention mechanism to assign weights to the 14-day data collection volume (e.g., the weight of the day before a major promotion is higher than that of a normal day), focusing on key time steps. Key time steps refer to days with unusually high data collection volume, such as the exceptionally large order volume during Singles' Day (November 11th).

[0057] The output layer outputs the predicted data collection volume of each TOP table for the next day. Monte Carlo Dropout (MC Dropout) is used to estimate uncertainty, and the 95% confidence interval of the predicted data collection volume is output. For example, if the predicted data collection volume of the TOP table is 1.2 million rows, the 95% confidence interval of the predicted data collection volume of the TOP table is [1.05 million rows, 1.35 million rows], which means that the TOP table LSTM model predicts that the data collection volume of the TOP table for the next day is likely (95%) between 1.05 million and 1.35 million rows.

[0058] In one embodiment, the business domain-level data collection includes the TOP table-level data collection and non-TOP table-level data collection within the business domain. The business domain-level LSTM model, through aggregation and calibration, can balance the strong fluctuations in TOP table-level data collection and the stability of non-TOP table-level data collection. The business domain-level LSTM model includes an input layer, a core layer, and an output layer. The input layer's inputs include the business domain-level data collection volume for each business domain in the previous 14 days, the sum of the TOP table-level data collection volume within each business domain (predicted by the TOP table-level LSTM model), and business characteristics (e.g., "whether it is a peak business period"). The input dimension of the input layer is 14 × (1 + 1 + 2) = 56. The core layer includes stacked LSTM and residual connections, which balance the fluctuations and stability of TOP table-level and non-TOP table-level data collection. The stacked LSTM consists of three LSTM layers with a hidden dimension of 128. The first LSTM layer learns the interaction patterns between the TOP and non-TOP tables of each business domain. The second layer learns the periodicity of the intra-domain business domain-level data collection volume within each business domain. The third layer captures the long-term trend of the intra-domain business domain-level data collection volume within each business domain. Residual connections mitigate the vanishing gradient problem in deep networks, preserving the signal of the sum of the intra-domain TOP table-level data collection volume of each business domain in the input. The output layer outputs the next-day business domain-level data prediction collection volume for each business domain, and also outputs the contribution percentage of the next-day business domain-level data prediction collection volume of the intra-domain next-day TOP table-level data prediction collection volume within each business domain. The contribution percentage of the next-day business domain-level data prediction collection volume of the intra-domain next-day TOP table-level data prediction collection volume is matched with the historical contribution percentage. If the deviation between the contribution percentage of the next-day business domain-level data prediction collection volume of the intra-domain next-day TOP table-level data prediction collection volume and the historical contribution percentage is greater than 10%, calibration is triggered. Non-TOP table-level data collection volume fluctuates little and can serve as a stable anchor point. If the difference between the sum of the predicted collection volume of TOP table-level data for the next day and the predicted collection volume of business domain-level data for the next day output by the business domain-level LSTM model exceeds the historical fluctuation range (e.g., ±20%), then the predicted collection volume of business domain-level data for the next day output by the business domain-level LSTM model is dynamically adjusted based on the historical average of the non-TOP table-level data collection volume within the business domain. An adjustment data collection is estimated based on the historical average of the non-TOP table-level data collection volume. By adjusting the data collection volume, the gap between the predicted collection volume and the actual collection volume of business domain-level data is narrowed, making the predicted collection volume of business domain-level data for the next day output by the business domain-level LSTM model closer to the actual situation.

[0059] In one embodiment, the system-level data collection volume is affected by the linkage between various business domains (e.g., the "user domain" and "transaction domain" grow synchronously during peak new user registration periods) and global resource constraints (e.g., bandwidth limits). The system-level LSTM model predicts the system-level data collection volume by adapting to global constraints and resources, balancing accuracy and feasibility. The system-level LSTM model includes an input layer, a core layer, and an output layer. The input layer includes the system-level data collection volume for the previous 30 days, the sum of the predicted business domain-level data collection volumes for each business domain (obtained through prediction by the business domain-level LSTM model), and global resource characteristics (e.g., the daily bandwidth limit). The input dimension of the input layer is 30×(1+N+2), where N is the number of business domains. The core layer includes an LSTM layer and a constraint layer. The LSTM layer includes two LSTM layers with a hidden dimension of 256. The LSTM layer learns the collaborative patterns of each business domain (e.g., the data collection volume of the "transaction domain" increases one day after the data collection volume of the "marketing domain" increases). The constraint layer calculates the predicted data collection volume for the data governance system the following day based on the predicted collection volume of TOP table-level data, business domain-level data, and system-level data. A hard resource limit (e.g., the system's maximum processing capacity) is introduced to truncate the predicted collection volume. For example, if the predicted collection volume exceeds the data governance system's maximum collection volume, the upper limit of the predicted collection volume is forcibly adjusted to 90% of the maximum collection volume to avoid overloading the data governance system's resources. The output layer outputs the system's predicted collection volume and, when the predicted collection volume exceeds 80% of the data governance system's maximum collection volume, marks it as "resource strained" and issues a resource warning signal.

[0060] In one embodiment, the training process of the TOP table-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model adopts a dynamic adaptation time-shifting strategy, and the training process includes: Step 221: Divide the dataset into training, validation, and test sets according to time sequence to avoid data leakage.

[0061] The training set includes data from the first 12 months (70%); the validation set includes data from the 13th and 14th months (15%); and the test set includes data from the 15th month (15%). For example, if the current time is December 2025, the dataset would be the TOP table-level data collection volume from September 2024 to November 2025, along with features related to the TOP table-level data collection volume (e.g., basic time-series features, business features, cross-layer correlation features). The training set would be the TOP table-level data collection volume from September 2024 to August 2025, along with features related to the TOP table-level data collection volume. The validation set would be the TOP table-level data collection volume from September 2025 to October 2025, along with features related to the TOP table-level data collection volume. The test set would be the TOP table-level data collection volume from November 2025, along with features related to the TOP table-level data collection volume.

[0062] In one embodiment, the TOP table-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model can be trained using a sliding window: the model is retrained every 7 days with the latest collected data (excluding the earliest 7 days) to adapt to the time-series distribution drift (for example, the addition of a new data table in the business domain causes the baseline of the data collection to rise).

[0063] Step 222: Construct the loss function and optimizer for the TOP table-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model, respectively.

[0064] In one embodiment, the TOP-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model each use the mean squared error as the loss function L. MSE To prevent the TOP-level LSTM model, business domain-level LSTM model, and system-level LSTM model from ignoring extreme values, a weight is added to the loss function for outlier samples (is_exception=1). For example, the loss function for outlier samples (is_exception=1) is 0.5 × L. MSE The TOP-level LSTM model, business domain-level LSTM model, and system-level LSTM model can be modified to L, where L = L. MSE-1+ 0.5×L MSE-2 L MSE-1 L represents the loss function for non-abnormal samples. MSE-2 This represents the loss function for outlier samples. The optimizer for training the TOP table-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model can be the Adam optimizer, with a learning rate of 0.001 that decays with each training epoch. Gradient clipping (clipvalue=1.0) is used to prevent gradient explosion.

[0065] In one embodiment, the prediction chain for predicting the amount of data collected moves from a single data table to the system's collaborative inference. The daily data collection forecasting process starts at 2 AM, predicting the data collection volume for TOP table-level data, business domain-level data, and system-level data, and outputs the daily forecast data collection volume, including the following steps: Step 2201: Predict the amount of data collected at the top table level using the TOP table-level LSTM model.

[0066] In one embodiment, the input of the TOP table-level LSTM model includes the data collection volume of the TOP table in the past 14 days, time features, and business features; the output is the predicted data collection volume of the TOP table on the current day (for example, the predicted data collection volume of the "order table" is 1.2 million records), and it is stored in the prediction result library.

[0067] Step 2202: Predict the amount of business domain-level data to be collected using a business domain-level LSTM model.

[0068] In one embodiment, the process of using a business domain-level LSTM model to predict the business domain-level data collection volume includes data aggregation, data prediction, and data calibration. Data aggregation: The input to the business domain-level LSTM model includes the daily business domain-level data collection volume for the previous 14 days within the business domain, the predicted data collection volume for the top tables of all top tables within the business domain, and business characteristics. The predicted data collection volume for the top tables of all top tables within the business domain is aggregated to obtain the aggregated value of the predicted data collection volume for the top tables of all top tables within the business domain. Data prediction: The business domain-level LSTM model outputs the predicted data collection volume for the business domain. Data calibration: If the deviation between the aggregated value and the predicted data collection volume output by the business domain-level LSTM model is greater than 10%, the predicted data collection volume output by the business domain-level LSTM model is corrected based on the historical average of the non-top table-level data collection volume within the business domain. The non-top table-level data collection volume ≈ historical data collection volume within the business domain × (1 - historical percentage of top table-level data collection volume).

[0069] Step 2203: Predict the system-level data acquisition volume using a system-level LSTM model.

[0070] In one embodiment, the input to the system-level LSTM model includes the daily system-level data collection volume for the previous 30 days, the predicted data collection volume for each business domain within the system (predicted by the business domain-level LSTM model), and global resource characteristics. The system-level LSTM model aggregates the predicted data collection volumes for each business domain within the system; outputs the predicted data collection volume for the system level; and calculates the predicted collection volume based on the predicted data collection volume for the TOP table level, the predicted data collection volume for each business domain, and the predicted data collection volume for the system level. Through constraints, system resources for the data governance system are pre-allocated based on the predicted collection volume; if the predicted collection volume exceeds a set data volume limit, a resource warning signal is issued. For example, if the predicted collection volume exceeds resource limits (e.g., bandwidth limits, storage limits), a resource scheduling suggestion is triggered to expand system capacity in advance and adopt off-peak data collection. The predicted data collection volume at the TOP table level, the predicted data collection volume at the business domain level, the predicted data collection volume at the system level, and the confidence interval are synchronized to the data collection scheduling module. The data collection scheduling module pre-allocates system resources of the data governance system based on the predicted data collection volume at the TOP table level, the predicted data collection volume at the business domain level, the predicted data collection volume at the system level, and the confidence interval. For example, more computing nodes are reserved for data tables with large predicted collection volumes.

[0071] In one embodiment, the system-level LSTM model, the business domain-level LSTM model, and the TOP table-level LSTM model calculate the MAE (mean absolute error) and MAPE (mean absolute percentage error) of the system-level data prediction collection volume, the business domain-level data prediction collection volume, and the TOP table-level data prediction collection volume, respectively. The MAPE of the TOP table-level data prediction collection volume is required to be less than 15%, the MAPE of the business domain-level data prediction collection volume is required to be less than 10%, and the MAPE of the system-level data prediction collection volume is required to be less than 8%.

[0072] In one embodiment, the deviation rate between the total predicted data collection volume of the TOP table level of the business domain and the actual data collection volume of the business domain level of the business domain is required to be less than 12%, and the deviation rate between the total predicted data collection volume of the business domain level of the system and the actual data collection volume of the system level of the system is required to be less than 12%.

[0073] In one embodiment, anomaly tracing and root cause analysis are performed based on MAPE. If the first mean absolute percentage error of the system-level data prediction acquisition volume predicted by the system-level LSTM model is greater than or equal to a first set error threshold (8%), then a second mean absolute percentage error greater than or equal to a second set error threshold (10%) is obtained. The second mean absolute percentage error is the error of the business domain-level LSTM model predicting the business domain-level data prediction acquisition volume, and the business domain is determined to be an abnormal business domain. If the second mean absolute percentage error of the business domain-level data prediction acquisition volume predicted by the business domain-level LSTM model is greater than or equal to a second set error threshold, then a third mean absolute percentage error greater than or equal to a third set error threshold (15%) is obtained. The third mean absolute percentage error is the error of the TOP table-level LSTM model predicting the TOP table-level data prediction acquisition volume, and the TOP table is determined to be an abnormal TOP table.

[0074] In one embodiment, when the average absolute percentage error of the system-level data prediction collection volume, the business domain-level data prediction collection volume, and the TOP table-level data prediction collection volume exceeds the requirements, drill-down analysis can be used to locate the specific location and cause of the average absolute percentage error exceeding the requirements by analyzing the system-level data prediction collection volume, the business domain-level data prediction collection volume, and the TOP table-level data prediction collection volume layer by layer. For example, if the MAPE of the system-level data prediction collection is greater than or equal to 8%, check the MAPE of the business domain-level data prediction collection for each business domain in the system. Business domains with a MAPE of 10% or higher are identified as abnormal business domains. If the MAPE of the business domain-level data prediction collection for a business domain is greater than or equal to 10%, check the MAPE of the TOP table-level data prediction collection for all TOP tables within that business domain. TOP tables with a MAPE of 15% or higher are identified as abnormal TOP tables. If the MAPE of the TOP table (abnormal TOP table) is greater than or equal to 15%, analyze the specific reasons why the MAPE of the TOP table (abnormal TOP table) is greater than or equal to 15%. Analyze whether it is due to new events not being included in the features (e.g., sudden traffic) or the TOP table-level LSTM model not capturing new cycles (e.g., newly added daily collection rules).

[0075] In one embodiment, the iteration strategies for the system-level LSTM model, business domain-level LSTM model, and TOP table-level LSTM model include periodic iteration, triggered iteration, and feature iteration. Periodic iteration: The system-level LSTM model, business domain-level LSTM model, and TOP table-level LSTM model are trained weekly using the latest 7 days of data (including freezing underlying parameters and fine-tuning the top layer), and fully retrained monthly. Triggered iteration: When the MAPE of any of the system-level LSTM model, business domain-level LSTM model, or TOP table-level LSTM model exceeds the requirement for 3 consecutive days, or the change rate of the TOP table list is >20% (e.g., adding or deleting key data tables), training of the system-level LSTM model, business domain-level LSTM model, and TOP table-level LSTM model is immediately initiated. Feature iteration: The effectiveness of event features is reviewed monthly (e.g., whether the weight of the "major promotion" tag is reasonable), and new high-frequency event tags (e.g., "weekly data reconciliation day") are added.

[0076] Step 230: Optimize the SQL scripts of multi-source heterogeneous data using the SQL optimization model to obtain the optimized SQL scripts and optimization instructions for the optimized SQL scripts.

[0077] In one embodiment, the SQL optimization model includes an input layer, an encoding layer, a diagnostic layer, a generation layer, and an output layer. The input layer acquires the SQL text, execution plan, and metadata tags of the SQL script from multi-source heterogeneous data. The encoding layer generates the SQL semantic vector of the SQL script using a pre-trained language model, extracts local features of the execution plan using an execution plan feature extractor, and outputs a joint feature vector concatenated with the SQL semantic vector and local features. The diagnostic layer, based on the joint feature vector from the encoding layer, outputs the optimization type of the SQL script through a multi-label classification task and locates the problem location in the SQL script through sequence labeling. The generation layer, using a decoder, generates an optimized SQL script based on the optimization type and problem location from the diagnostic layer, and the joint feature vector from the encoding layer, constrained by Hive optimization rules. The output layer outputs the optimized SQL script and an optimization description of the optimized SQL script.

[0078] In one embodiment, a BERT-based model is trained using an SQL script dataset to obtain an SQL optimization model. The SQL script dataset includes an SQL optimization case library, academic datasets, SQL script optimization records, execution plan logs, and paired samples of problematic SQL and optimized SQL. The SQL optimization case library includes optimization examples of SQL scripts from official documentation and comparison scripts before and after SQL optimization. The academic dataset includes samples from the SQL parsing dataset and the WikiSQL dataset (containing SQL and execution logic) involving big data scenarios (e.g., large table joins, full table scans). The SQL script optimization records include "original SQL - optimized SQL - execution time comparison - optimization labels" (e.g., "add partition filter", "adjust join order") extracted from data platform or data system logs. The table names and field names in the SQL script optimization records need to be replaced with common identifiers. The execution plan logs include the output of Hive's EXPLAIN (command for viewing the execution plan of SQL queries), which includes Stage partitioning, Map / Reduce task counts, and data skew markers. These are then correlated with the corresponding SQL text to construct a "SQL - execution plan - performance metric" triple. The rule engine automatically generates "problem SQL", and based on the optimization rules, it generates corresponding "optimized SQL" to form a paired sample to deal with high-frequency optimization scenarios (e.g., 10 types of typical Hive performance problems).

[0079] In one embodiment, basic fields for SQL scripts in the SQL script dataset can be defined. These basic fields may include raw_sql (raw SQL text), optimized_sql (optimized SQL text), performance_metrics (key metrics before and after optimization), optimization_type (optimization type label), and execution_plan (execution plan text). raw_sql (raw SQL text) can be, for example, a HiveQL (Hive Query Language, a query language similar to SQL) script. performance_metrics (key metrics before and after optimization) includes execution time, number of MapReduce (programming model) tasks, amount of data in Shuffle (a key data reorganization process in distributed computing), and skew rate. optimization_type (optimization type label) is represented by enumerated values, such as "partition pruning," "Join strategy adjustment," "data skew handling," and "predicate pushdown," totaling 15-20 categories. execution_plan (execution plan text) can be the execution plan text output by Hive EXPLAIN, serving as an enhancement feature.

[0080] In one embodiment, data preprocessing can be performed on the SQL scripts in the SQL script dataset. Data preprocessing may include SQL normalization, text segmentation and encoding, and data augmentation. SQL normalization involves removing comments, standardizing keyword case (e.g., select → SELECT), adjusting indentation, and avoiding format differences that could affect semantic understanding, thus unifying the SQL script format. Text segmentation and encoding are performed on the SQL scripts in the SQL script dataset, concatenating raw_sql and execution_plan into the input text (e.g., using special symbols as separators, such as [SQL][PLAN]), enabling the SQL optimization model to understand both syntax and execution logic. The SQL scripts are converted into token sequences according to BERT requirements, for example, using a dedicated SQL tokenizer, such as the sqlparse library, to split the SQL script's keywords, table names, and functions, preventing "JOIN" from being split into "J", "O", "I", and "N". The SQL scripts are paraphrased by performing equivalent transformations on the raw_sql of the SQL scripts (e.g., swapping IN and EXISTS, adjusting the order of WHERE conditions) to maintain semantic integrity while increasing sample diversity; common redundancies are randomly added (e.g., redundant DISTINCT, unused GROUP BY fields) to simulate non-standard writing practices in real-world development; and data augmentation is performed on the SQL scripts in the SQL script dataset through paraphrasing and noise injection.

[0081] In one embodiment, the encoding layer of the SQL optimization model uses bert-base-uncased (a pre-trained language model based on the Transformer architecture) as its foundation. The encoding layer is pre-trained for domain adaptation using a large-scale unlabeled HiveQL script dataset (1 million Hive SQL statements). The encoding layer replaces the vocabulary (adding SQL keywords and Hive-specific functions), and a Masked Language Model (MLM) is used to randomly mask SQL keywords, table names, and function names, enabling the SQL optimization model to learn the SQL syntax structure and generate SQL semantic vectors for the SQL scripts. The encoding layer performs execution plan prediction, predicting key information in the execution plan (e.g., whether MapJoin is triggered, data volume level in the Shuffle stage) given the SQL text, enhancing the bert-base-uncased model's understanding of the "syntax-execution" mapping. An execution plan feature extractor (1D-CNN, a one-dimensional convolutional neural network) extracts local features from the execution plan text. The SQL semantic vector of the SQL script is concatenated with these local features to achieve a fusion of the SQL semantic vector and local features, generating a joint feature vector.

[0082] In one embodiment, the diagnostic layer of the SQL optimization model identifies and locates performance bottlenecks in the SQL script based on the joint feature vector of the encoding layer. The SQL script optimization type is determined through a multi-label classification task (e.g., simultaneously labeling "partition pruning" and "Join strategy adjustment"). Based on the token sequence of the SQL text, each token is labeled with a tag using problem location sequence labeling (similar to the NER task), such as O (no problem), B-PARTITION (partition filtering missing start), and I-JOIN (Join strategy problem in the middle).

[0083] In one embodiment, the generation layer of the SQL optimization model adopts a Seq2Seq structure of "BERT encoder + T5 decoder". Based on the optimization type of the diagnostic layer, the marked problem location, and the token sequence of the original SQL text, it generates an optimized SQL script. The constraint mechanism of the generation layer can be an embedded Hive optimization rule (e.g., a MapJoinhint must be added for joining a large table with a small table, and the partition key must appear in the WHERE condition) to avoid generating invalid optimized SQL scripts.

[0084] In one embodiment, the training strategy and parameter design of the SQL optimization model are shown in Table 3 below.

[0085] Table 3

[0086] In one embodiment, the training process of the SQL optimization model includes domain fine-tuning, multi-task fine-tuning, and reinforcement learning optimization.

[0087] Phase 1: Pre-training adaptation and domain fine-tuning. The bert-base-uncased encoding layer is pre-trained on the unlabeled Hive SQL corpus to optimize the MLM loss, enabling the SQL optimization model to learn SQL syntax and Hive-specific structures.

[0088] Phase 2: Multi-task fine-tuning. Using the labeled "Problem SQL - Optimized SQL" dataset, train the diagnostic layer (multi-label classification + problem location sequence labeling) and the generation layer, with a loss function of L. total L total =L cls +L ner +2×L gen L cls L represents the classification loss. ner L represents the sequence labeling loss at the problem location; gen This represents the generation loss. The classification loss L... cls And the problem location sequence labeling loss L nerIt can be BCE (Binary Cross-Entropy) loss, with generation loss L. gen It can be CCE (Categorical Cross-Entropy) loss.

[0089] Phase 3: Reinforcement Learning Optimization: Using the percentage reduction in execution time of the optimized SQL script as a reward, the generation layer is fine-tuned using the Proximal Policy Optimization (PPO) algorithm. This ensures the SQL optimization model prioritizes generating SQL scripts and optimization schemes with significantly improved performance. By constructing a Policy Network and a Value Network, defining a Reward Function and a PPO loss function, and employing PPO iteration, the SQL optimization model is optimized.

[0090] In one embodiment, the SQL optimization model can correctly identify that the proportion of SQL script samples to be optimized is greater than or equal to 85%; the proportion of Hive performance problem types that can be handled is greater than or equal to 90% of the total types; the proportion of optimized SQL scripts that can be executed normally and meet the optimization rules is greater than or equal to 90%; and the average execution time of the optimized SQL scripts is reduced by greater than or equal to 40%.

[0091] Step 240: Repair the metadata in the multi-source heterogeneous data through the metadata compliance model to obtain optimized metadata of the multi-source heterogeneous data.

[0092] In one embodiment, a metadata compliance model can be used to standardize the metadata in multi-source heterogeneous data based on a metadata mapping knowledge base to obtain standardized metadata in the multi-source heterogeneous data; the standardized metadata can be repaired according to preset metadata rules to obtain optimized metadata of the multi-source heterogeneous data, and the repair includes at least one of completion, correction, cleaning, and deduplication.

[0093] In one embodiment, a knowledge graph can be used to store standard type mapping relationships between different data sources. Standard mappings are generated by retrieving knowledge from the knowledge graph, creating a metadata mapping knowledge base for metadata in multi-source heterogeneous data. Through a metadata compliance model, standardized metadata in the multi-source heterogeneous data is obtained based on the metadata mapping knowledge base. Standardized metadata includes, but is not limited to, tables, fields, comments, paths, table creation statements, partitions, and indexes of the corresponding data. By running a data processing script, an abstract syntax tree (AST) is parsed to obtain the lineage information of the corresponding standardized metadata, facilitating subsequent governance of standardized metadata in multi-source heterogeneous data.

[0094] In one embodiment, a metadata compliance model can be used to organize the technical and business specifications of metadata into a metadata rule base. Through this model, the lineage information and standardized metadata from multi-source heterogeneous data are organized into a data stream. Based on the metadata rule base, multiple rounds of agent rule identification are performed on the data stream to conduct compliance verification checks on the standardized metadata, obtaining compliant standardized metadata and standardized metadata to be repaired. The metadata compliance model then repairs the standardized metadata to be repaired. Repair operations include, but are not limited to, completion, correction, cleaning, and deduplication, correcting errors, redundancy, and missing data in the standardized metadata to be repaired, improving the quality of metadata, ensuring its accuracy, completeness, and consistency, and enhancing its security, discoverability, and usability. By standardizing and repairing metadata, ambiguity is eliminated, enabling different data sources to have a consistent understanding of metadata, improving the efficiency of metadata analysis and application; standardized and repaired metadata can optimize indexing and querying, improving data retrieval speed and analytical accuracy.

[0095] Step 250: Perform attribution analysis on abnormal SQL scripts in the SQL script using the SQL attribution model to obtain the attribution results and optimization solutions for the abnormal SQL scripts.

[0096] In one embodiment, the SQL script execution logs can be used to train the SQL optimization model to obtain an SQL attribution model; based on the SQL script execution logs, SQL scripts with a runtime longer than or equal to a set duration or that report an error are identified as abnormal SQL scripts; attribution analysis is performed on the abnormal SQL scripts through the SQL attribution model to obtain the attribution results and optimization schemes for the abnormal SQL scripts.

[0097] In one embodiment, the execution of the SQL script is a data ETL process, which includes data extraction, transformation, and loading. The operational logs of the data ETL process during SQL script execution can be used as part of the training data flow to further train the SQL optimization model and obtain the SQL attribution model.

[0098] In one embodiment, a data collection tool can collect all data processing logs from the data platform (data source) daily. Based on the runtime and error status of the SQL scripts in the logs, SQL scripts with a runtime greater than or equal to a set duration are identified as abnormal SQL scripts, or SQL scripts that report errors are identified as abnormal SQL scripts. For example, SQL scripts with a runtime exceeding 30 minutes are identified as abnormal SQL scripts. The SQL attribution model performs attribution analysis on the abnormal SQL scripts, outputting the attribution results and optimization solutions. The attribution results output by the SQL attribution model include the main cause of the abnormality, the sub-type, the confidence level, and key evidence. The optimization solutions output by the SQL attribution model include optimization steps, an example of rewriting the optimized SQL, and the expected results achieved after optimization.

[0099] In one embodiment, the input information Prompt for the SQL attribution model attribution analysis includes resource features: {Resource Feature JSON}, data features: {Data Feature JSON}, SQL features: {SQL Feature JSON}, and environment features: {Environment Feature JSON}. The attribution rules of the SQL attribution model include the main cause and sub-types in Table 4 below.

[0100] Table 4

[0101] In one embodiment, the SQL attribution model outputs the attribution results of the abnormal SQL scripts, and the output format (JSON) of the attribution results is as follows: "main_cause": "main cause" "sub_cause": "sub-type", "confidence": "confidence level (0-100%)", "evidence": "Key evidence (e.g., "memory usage peaked at 98%", "OOM logs appeared")".

[0102] In one embodiment, the confidence level represents the degree of confidence that the SQL attribution model believes the conclusion that "abnormal SQL script timeouts are caused by the selected primary cause".

[0103] In one embodiment, the attribution results of abnormal SQL scripts output by the SQL attribution model can provide specific and executable optimization basis for SQL rewriting (optimization), resource adjustment, and parameter configuration of abnormal SQL scripts.

[0104] In one embodiment, the input information Prompt for optimizing the SQL attribution model's abnormal SQL script includes the original SQL: {SQL text}, attribution result: {attribution JSON}, and key execution plan information: {execution plan summary}. The optimization rules for optimizing the SQL attribution model's abnormal SQL script include: insufficient resources (e.g., adjusting YARN parameters, such as executor-memory=16g → 24g) and increasing parallelism (num_executors=5 → 10); data bloat (e.g., adding partitions, field filtering, limiting scanned columns, such as removing SELECT *), and sampling processing (e.g., LIMIT 10000); script defects (e.g., rewriting the join order, such as adding MapJoin Hint, adding PARTITION BY to window functions, and splitting large queries into CTEs); and environmental anomalies (e.g., staggered scheduling, such as from 8 am to 12 pm) and contacting the cluster administrator for capacity expansion).

[0105] In one embodiment, the SQL attribution model outputs the optimization steps, rewrite example, and expected results of the optimized abnormal SQL script. The output format of the optimization steps, rewrite example, and expected results can be as follows: "optimization_steps": [ Step 1: Specific operations (e.g., "Add partition filter condition WHERE dt='2023-10-01'") Step 2: Example of SQL rewriting (e.g., "SELECT id, name FROM sales → SELECT id,name FROM sales WHERE dt='2023-10-01'") Step 3: Adjust resource parameters (e.g., "executor-memory=16g → 24g") "expected_effect": "Expected results (e.g., "Scanned data volume reduced from 100GB to 10GB, runtime shortened to within 10 minutes")". "risk_note": "Risk warning (e.g., "Data integrity of partition dt='2023-10-01' needs to be confirmed)".

[0106] In one embodiment, a visual interface can be used to display in real time the main cause, sub-type, confidence level, key evidence, optimization steps, SQL rewriting examples, and expected results of the abnormal SQL scripts output by the SQL attribution model, enabling operations and maintenance personnel to efficiently handle abnormal SQL scripts. For example, for timeout SQL scripts (abnormal SQL scripts with a runtime greater than or equal to a set duration), the visual interface can display a list of timeout SQLs for the day (sorted in descending order of runtime), along with the task ID, SQL summary, runtime, and business domain of the timeout SQLs. A trend chart of timeout SQLs can be displayed in real time, showing the number of timeout SQLs and the average runtime changes over the past 7 days, and marking abnormal peaks (e.g., triggering an alarm if the number of timeout SQLs exceeds 5 on a certain day). The visual interface can also display in real time the main cause, sub-type, confidence level, and key evidence of specific timeout SQLs output by the SQL attribution model, along with original log snippets (e.g., OOM logs, execution plan screenshots).

[0107] In one embodiment, a visual interface can be used to display the optimization steps, SQL rewriting examples, and expected results of abnormal SQL scripts. It provides a correction entry point, allowing operations personnel to correct attribution results. After correction, the script is automatically added to the dataset for fine-tuning the SQL attribution model. The runtime of the optimized SQL script is recorded, and metrics before and after optimization (e.g., an 80% reduction in scanned data volume) are compared to automatically calculate the optimization benefits. This visual interface lowers the operational threshold for troubleshooting abnormal SQL scripts, reducing troubleshooting time by 50%.

[0108] Step 260: Perform a data quality assessment on the multi-source heterogeneous data collected by the data governance system at the target time and obtain a quality score for the multi-source heterogeneous data.

[0109] In one embodiment, the data quality of multi-source heterogeneous data collected by the data governance system at a target time can be assessed by comprehensively evaluating three dimensions: data acquisition success rate, ETL task effectiveness, and metadata completeness, using a weighted scoring method based on an AI large-scale model. This assessment yields a quality score for the multi-source heterogeneous data. The score is determined based on the successful acquisition volume S of the multi-source heterogeneous data at the target time. ′ (t) The predicted acquisition volume S of multi-source heterogeneous data at the target time (t) The number of effective ETL tasks at the target time, R ′ (t) The total number of ETL tasks at the target time, R (t) Metadata completeness I in multi-source heterogeneous data at the target time ′ (t)Data quality assessment is performed on the multi-source heterogeneous data collected by the data governance system at the target time, and a quality score (Score) is obtained for the multi-source heterogeneous data, where Score = A × (S ′ (t) ÷S (t) ) + B × (R ′ (t) ÷R (t) )+C×(I ′ (t) (÷100), A, B, and C represent quality rating coefficients, with A=0.3, B=0.4, and C=0.3.

[0110] In one embodiment, the health status of the collected multi-source heterogeneous data is represented by a quality score based on daily data quality. The governance of the multi-source heterogeneous data is assessed and implemented based on this quality score, providing a quantitative basis for governance and enabling traceability and optimization of the governance effect.

[0111] Step 270: Build a data governance knowledge base.

[0112] In one embodiment, a data governance knowledge base can be constructed based on the prediction of the data collection volume of multi-source heterogeneous data using the TOP table-level LSTM model, the business domain-level LSTM model, and the system-level LSTM model; the pre-allocation of system resources of the data governance system at the target time based on the predicted collection volume; the resource warning signal issued when the predicted collection volume exceeds the set data volume limit; the metadata of multi-source heterogeneous data - optimization metadata; the SQL script of multi-source heterogeneous data - optimization SQL script - optimization description of optimization SQL script; and the attribution results and optimization schemes of abnormal SQL scripts.

[0113] In one embodiment, the data governance knowledge base is the core infrastructure supporting the intelligent management of data governance across all scenarios. By collecting historical cases and rules from scenarios such as prediction, early warning, SQL script optimization, metadata repair, and abnormal SQL script attribution analysis, a data governance knowledge base is constructed, achieving a closed loop of case reuse, pattern mining, prompt optimization, and model iterative optimization. Case reuse based on the knowledge base reduces the time to solve new problems by 50%.

[0114] The data governance method in this application uses a TOP-level LSTM model to predict the predicted data collection volume at the TOP-level table, a business domain-level LSTM model to predict the predicted data collection volume at the business domain, and a system-level LSTM model to predict the predicted data collection volume at the system level. It employs a three-level differentiated LSTM structure (TOP-level, business domain-level, and system-level LSTM models) to match the characteristics of different data collection volumes at these three levels. A dynamic weight allocation algorithm is used to calculate the predicted data collection volume, dynamically adjusting the weights based on the historical prediction errors of each level of the TOP-level, business domain-level, and system-level LSTM models. This improves the accuracy and robustness of the fused predicted data collection volume.

[0115] Furthermore, the data governance method of this application embodiment pre-allocates system resources of the data governance system at the target time based on the predicted collection volume; if the predicted collection volume is greater than the set data volume limit, a resource warning signal is issued; it can avoid resource overload or data gap caused by sudden increases or decreases in data collection volume; it supports multi-source heterogeneous data access from different data sources, adapts to complex scenarios such as peak collection during promotional events and cross-business domain data synchronization; it identifies resource shortages in advance (predicted collection volume is greater than the set data volume limit), and improves system resource utilization by 25% through off-peak collection and capacity expansion suggestions, avoiding ineffective resource waste.

[0116] Furthermore, the data governance method of this application embodiment optimizes the SQL scripts of multi-source heterogeneous data through the SQL optimization model, obtains the optimized SQL scripts and the optimization descriptions of the optimized SQL scripts, shortens the debugging time of the SQL scripts (the debugging time of the SQL scripts is 40% of the original time), reduces the average execution time of the optimized SQL scripts by 40%, and reduces the amount of data scanned by full table scan SQL scripts by 90%.

[0117] Furthermore, the data governance method in this application embodiment repairs the metadata in multi-source heterogeneous data through a metadata compliance model to obtain optimized metadata of multi-source heterogeneous data; the automated metadata repair can cover more than 70% of the metadata, reduce the verification cost by 50%, and shorten the standardization cycle of multi-source heterogeneous data from 15 days to 3 days.

[0118] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a data governance system, electronic device, and corresponding embodiments.

[0119] Figure 3 This is a schematic diagram of the structure of the data governance system shown in the embodiments of this application.

[0120] See Figure 3A data governance system 300 includes a script optimization unit 310, a metadata optimization unit 320, and a script analysis unit 330.

[0121] The script optimization unit 310 is used to optimize the SQL scripts of multi-source heterogeneous data through the SQL optimization model, and obtain the optimized SQL scripts and the optimization descriptions of the optimized SQL scripts.

[0122] Metadata optimization unit 320 repairs metadata in multi-source heterogeneous data through a metadata compliance model, and obtains optimized metadata for multi-source heterogeneous data.

[0123] The script analysis unit 330 is used to perform attribution analysis on abnormal SQL scripts in SQL scripts through the SQL attribution model, and obtain the attribution results and optimization schemes of abnormal SQL scripts.

[0124] Figure 4 This is another structural schematic diagram of the data governance system shown in the embodiments of this application.

[0125] See Figure 4 A data governance system 300 includes a perception layer 410, an AI central layer 420, and a decision application layer 430. Through these three layers, the system achieves prediction, early warning, diagnosis, optimization, and evolution of multi-source heterogeneous data governance. The perception layer 410 collects multi-source heterogeneous data, such as metadata, log data, and script data, from different data sources including databases and data platforms, using various acquisition tools. The AI ​​central layer 420 trains the AI ​​model for the data governance system. This AI model includes a TOP table-level LSTM model, a business domain-level LSTM model, a system-level LSTM model, an SQL optimization model, a metadata compliance model, and an SQL attribution model. The decision application layer 430 includes the AI ​​model; it performs data governance on the multi-source heterogeneous data collected by the perception layer 310 using the AI ​​model.

[0126] Regarding the data governance system in the above embodiments, the specific methods by which each unit and layer performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0127] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0128] See Figure 5 The electronic device 1000 includes a memory 1010 and a processor 1020.

[0129] The processor 1020 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0130] Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1020 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 1010 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0131] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to execute part or all of the methods described above.

[0132] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0133] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) that, when executed by a processor of a data governance system (or electronic device, or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0134] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the method described above.

[0135] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data governance method, characterized by, The method comprises the following steps: optimizing SQL scripts of multi-source heterogeneous data through a SQL optimization model to obtain optimized SQL scripts and optimization descriptions of the optimized SQL scripts; repairing metadata in the multi-source heterogeneous data through a metadata integration model to obtain optimized metadata of the multi-source heterogeneous data; performing attribution analysis on abnormal SQL scripts in the SQL scripts through a SQL attribution model to obtain attribution results and optimization schemes of the abnormal SQL scripts.

2. The method of claim 1, wherein: Before the step of optimizing SQL scripts of multi-source heterogeneous data through a SQL optimization model to obtain optimized SQL scripts and optimization descriptions of the optimized SQL scripts, the method comprises the following steps: predicting a TOP table level data prediction collection amount of TOP table level multi-source heterogeneous data collected by a data governance system at a target time according to a TOP table level historical collection amount of the TOP table level multi-source heterogeneous data collected by the data governance system through a TOP table level LSTM model; predicting a business domain level data prediction collection amount of business domain level multi-source heterogeneous data collected by the data governance system at the target time according to a business domain level historical collection amount of the business domain level multi-source heterogeneous data collected by the data governance system and the TOP table level data prediction collection amount of the business domain through a business domain level LSTM model; predicting a system level data prediction collection amount of system level multi-source heterogeneous data collected by the data governance system at the target time according to a system level historical collection amount of the system level multi-source heterogeneous data collected by the data governance system and the business domain level data prediction collection amount of the data governance system through a system level LSTM model; According to the TOP table level data prediction collection amount, the business domain level data prediction collection amount and the system level data prediction collection amount, the prediction collection amount S is calculated (t) ; wherein, S (t) =W 1(t-1) ×S 1(t) +W 2(t-1) ×S 2(t) +W 3(t-1) ×S 3(t) , W 1(t-1) represents the first dynamic weight coefficient of the system level data prediction collection amount S 1(t) ; W 2(t-1) represents the second dynamic weight coefficient of the business domain level data prediction collection amount S 2(t) ; W 3(t-1) represents the third dynamic weight coefficient of the TOP table level data prediction collection amount S 3(t) ; pre-allocating system resources of the data governance system at the target time according to the prediction collection amounts; if the prediction collection amount is greater than a set data amount limit value, a resource pre-warning signal is sent.

3. The method of claim 2, wherein, The predicted collection amount S is calculated according to the TOP table level data predicted collection amount, the business domain level data predicted collection amount and the system level data predicted collection amount (t) Previously, the first dynamic weight coefficient, the second dynamic weight coefficient and the third dynamic weight coefficient are respectively calculated according to the first mean square error of the system level data predicted collection amount at time t-1 predicted by the system level LSTM model, the second mean square error of the business domain level data predicted collection amount at time t-1 predicted by the business domain level LSTM model, and the third mean square error of the TOP table level data predicted collection amount at time t-1 predicted by the TOP table level LSTM model.

4. The method of claim 2, wherein, The method further comprises the following steps: if a first mean absolute percentage error of the system level data prediction collection amount predicted by the system level LSTM model is greater than or equal to a first set error threshold value, a second mean absolute percentage error greater than or equal to a second set error threshold value is obtained, the second mean absolute percentage error is an error of the business domain level data prediction collection amount of the business domain predicted by the business domain level LSTM model, and the business domain is determined as an abnormal business domain; if a second mean absolute percentage error of the business domain level data prediction collection amount predicted by the business domain level LSTM model is greater than or equal to a second set error threshold value, a third mean absolute percentage error greater than or equal to a third set error threshold value is obtained, the third mean absolute percentage error is an error of the TOP table level data prediction collection amount of the TOP table predicted by the TOP table level LSTM model, and the TOP table is determined as an abnormal TOP table.

5. The method of claim 1, wherein, The step of optimizing SQL scripts of multi-source heterogeneous data through a SQL optimization model to obtain optimized SQL scripts and optimization descriptions of the optimized SQL scripts comprises the following steps: the SQL optimization model comprises an input layer, an encoding layer, a diagnosis layer, a generation layer and an output layer; the input layer obtains SQL texts, execution plans and metadata tags of the SQL scripts; The encoding layer generates an SQL semantic vector of the SQL script through a pre-trained language model, extracts local features of the execution plan through an execution plan feature extractor, and outputs a joint feature vector obtained by splicing the SQL semantic vector and the local features; The diagnosis layer outputs an optimization type of the SQL script through a multi-label classification task according to the joint feature vector of the encoding layer, and locates a problem position of the SQL script through sequence labeling; The generation layer generates the optimized SQL script according to the optimization type, the problem position of the diagnosis layer, and the joint feature vector of the encoding layer, with Hive optimization rules as constraints; The output layer outputs the optimized SQL script and an optimization explanation of the optimized SQL script.

6. The method of claim 1, wherein, The metadata in the multi-source heterogeneous data is repaired through the metadata integration model to obtain optimized metadata of the multi-source heterogeneous data, including: The metadata in the multi-source heterogeneous data is standardized through the metadata integration model according to a metadata mapping knowledge base to obtain standardized metadata of the metadata in the multi-source heterogeneous data; The standardized metadata is repaired according to a preset metadata rule to obtain the optimized metadata of the multi-source heterogeneous data, and the repairing includes at least one of completion, correction, cleaning, and deduplication.

7. The method of claim 1, wherein, The abnormal SQL script in the SQL script is attributed and analyzed through the SQL attribution model to obtain an attribution result and an optimization scheme of the abnormal SQL script, including: The SQL attribution model is obtained by training the SQL optimization model using the SQL script run log; SQL scripts with a runtime longer than or equal to a set time length or with a running error are identified as the abnormal SQL script according to the SQL script run log; The abnormal SQL script is attributed and analyzed through the SQL attribution model to obtain an attribution result and an optimization scheme of the abnormal SQL script.

8. The method of claim 2, wherein, The method further comprises: performing data quality evaluation on the multi-source heterogeneous data collected by the data governance system target time, to obtain a quality score Score of the multi-source heterogeneous data, wherein Score=A×(S ′ (t) ÷S (t) )+B×(R ′ (t) ÷R (t) )+C×(I ′ (t) ÷100), A, B and C represent quality score coefficients, S ′ (t) represents a successful collection amount of the multi-source heterogeneous data at the target time, S (t) represents a predicted collection amount of the multi-source heterogeneous data at the target time, R ′ (t) represents a number of effective ETL tasks at the target time, R (t) represents a total number of ETL tasks at the target time, I ′ (t) represents a metadata completeness of metadata in the multi-source heterogeneous data at the target time.

9. A data governance system, characterized by, It includes: A script optimization unit is configured to optimize SQL scripts of multi-source heterogeneous data through a SQL optimization model to obtain optimized SQL scripts and optimization explanations of the optimized SQL scripts; A metadata optimization unit is configured to repair metadata in the multi-source heterogeneous data through a metadata integration model to obtain optimized metadata of the multi-source heterogeneous data; A script analysis unit is configured to attribute and analyze abnormal SQL scripts in SQL scripts through a SQL attribution model to obtain attribution results and optimization schemes of the abnormal SQL scripts.

10. A computer-readable storage medium, characterized in that: Executable code is stored thereon, and when the executable code is executed by a processor, the processor executes the method of any one of claims 1-8.

Images