Data diagnostic methods, devices, equipment, storage media and software products

Abstract

This application discloses a data diagnostic method, apparatus, device, storage medium, and program product, belonging to the field of data processing technology. In this application, a target data table is acquired, and its current state statistics are determined. In response to the current state statistics meeting abnormal conditions, source tracing information for anomaly analysis of the target data table is generated based on the target data table and its upstream data tables. In a case database, target source tracing information that meets similar conditions to the source tracing information is searched, and the corresponding target diagnostic results and target adjustment schemes are obtained. The source tracing information, target source tracing information, target diagnostic results, and target adjustment schemes are input into a generative language model to diagnose the causes of anomalies in the target data table, obtaining diagnostic results and adjustment schemes. This application significantly improves the efficiency of data quality assurance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data diagnostic method, apparatus, device, storage medium, and program product. Background Technology

[0002] As the core infrastructure for enterprise data analysis, data warehouses have complex upstream and downstream dependencies between data tables. During the production process, data tables may experience quality issues such as sudden changes in the number of rows or a surge in the null value rate due to reasons such as abnormal data sources, changes in calculation logic, and system failures. If these issues are not detected and diagnosed in a timely manner, they will affect the output of downstream reports.

[0003] In related technologies, when an anomaly is found in a data table, engineers usually have to manually investigate and diagnose it. This involves checking scheduling logs, table structure changes, and calculation script modifications, and tracing back along the upstream dependency chain layer by layer to diagnose the cause of the anomaly. This process usually takes several hours to several days, which is too inefficient. Summary of the Invention

[0004] This application provides a data diagnostic method, apparatus, device, storage medium, and program product, which can solve the problem of low efficiency in data table diagnostics. The technical solution is as follows: On the one hand, a data diagnostic method is provided, the method comprising: Obtain the target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, generate source tracing information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table. In the case database, search for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis results and target adjustment schemes corresponding to the target traceability information; The source tracing information, the target source tracing information, the target diagnosis results, and the target adjustment plan are input into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnosis results and adjustment plans.

[0005] In one possible implementation, generating source information for anomaly analysis of the target data table based on the target data table and its upstream data tables includes: Obtain the data source relationship diagram corresponding to the target data table; Based on the data tracing relationship diagram, the upstream data table of the target data table is determined; The tracing information is generated based on the target data table and the upstream data table.

[0006] In another possible implementation, determining the upstream data table of the target data table based on the data tracing relationship diagram includes: Starting from the node corresponding to the target data table in the data tracing relationship diagram, the process traverses upstream layer by layer, and determines the data table corresponding to each upstream node as the upstream data table of the target data table.

[0007] In another possible implementation, generating the tracing information based on the target data table and the upstream data table includes: Based on the data processing dependency between the target data table and the upstream data table, perform upstream and downstream data reconciliation operations to obtain data reconciliation results; The traceability information is generated based on the data reconciliation results.

[0008] In another possible implementation, generating the tracing information based on the target data table and the upstream data table includes: Determine the abnormal feature data of the target data table and the first data output timestamp; Determine the data output status of the upstream data table before the first data output timestamp, and obtain the second data output timestamp of the upstream data table that has completed data output; In response to the time difference between the first data output timestamp and the second data output timestamp satisfying the specified cascading time window condition, it is determined that the upstream data table satisfies the cascading propagation condition, and cascading propagation information is generated based on the target data table and the upstream data table that satisfies the cascading propagation condition; The source tracing information is generated based on the abnormal feature data of the target data table, the data output status of the upstream data table, and the cascading propagation information.

[0009] In another possible implementation, generating the tracing information based on the target data table and the upstream data table includes: Obtain the table structure change record information and processing logic change record information of the target data table and the upstream data table within a specified time period. The specified time period is the time period corresponding to the specified time window before the first data output timestamp. The tracing information is generated based on the table structure change record information and the processing logic change record information.

[0010] In another possible implementation, the step of searching the case database for target tracing information that meets similar conditions to the tracing information, and obtaining the target diagnosis result and target adjustment plan corresponding to the target tracing information, includes: Extract the feature vector of the source information, and in the case database, find the target source information that meets the specified vector similarity condition with the source information based on the feature vector, and obtain the target diagnosis result and target adjustment plan corresponding to the target source information.

[0011] In another possible implementation, the step of inputting the source tracing information, the target source tracing information, the target diagnostic results, and the target adjustment plan into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnostic results and adjustment plans includes: The source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to diagnose the abnormal causes of the target data table based on the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme. The model prompts are input into a generative language model to obtain diagnostic results and adjustment plans.

[0012] On the other hand, a data diagnostic device is provided, the device comprising: The acquisition module is configured to acquire a target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, it generates source information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table. The search module is configured to search the case database for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis result and target adjustment plan corresponding to the target traceability information; The generation module is configured to input the source tracing information, the target source tracing information, the target diagnosis results, and the target adjustment scheme into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnosis results and adjustment schemes.

[0013] In one possible implementation, the acquisition module is configured to: Obtain the data source relationship diagram corresponding to the target data table; Based on the data tracing relationship diagram, the upstream data table of the target data table is determined; The tracing information is generated based on the target data table and the upstream data table.

[0014] In another possible implementation, the acquisition module is used for: Starting from the node corresponding to the target data table in the data tracing relationship diagram, the process traverses upstream layer by layer, and determines the data table corresponding to each upstream node as the upstream data table of the target data table.

[0015] In another possible implementation, the acquisition module is used for: Based on the data processing dependency between the target data table and the upstream data table, perform upstream and downstream data reconciliation operations to obtain data reconciliation results; The traceability information is generated based on the data reconciliation results.

[0016] In another possible implementation, the acquisition module is used for: Determine the abnormal feature data of the target data table and the first data output timestamp; Determine the data output status of the upstream data table before the first data output timestamp, and obtain the second data output timestamp of the upstream data table that has completed data output; In response to the time difference between the first data output timestamp and the second data output timestamp satisfying the specified cascading time window condition, it is determined that the upstream data table satisfies the cascading propagation condition, and cascading propagation information is generated based on the target data table and the upstream data table that satisfies the cascading propagation condition; The source tracing information is generated based on the abnormal feature data of the target data table, the data output status of the upstream data table, and the cascading propagation information.

[0017] In another possible implementation, the acquisition module is used for: Obtain the table structure change record information and processing logic change record information of the target data table and the upstream data table within a specified time period. The specified time period is the time period corresponding to the specified time window before the first data output timestamp. The tracing information is generated based on the table structure change record information and the processing logic change record information.

[0018] In another possible implementation, the lookup module is used for: Extract the feature vector of the source information, and in the case database, find the target source information that meets the specified vector similarity condition with the source information based on the feature vector, and obtain the target diagnosis result and target adjustment plan corresponding to the target source information.

[0019] In another possible implementation, the generation module is used for: The source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to diagnose the abnormal causes of the target data table based on the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme. The model prompts are input into a generative language model to obtain diagnostic results and adjustment plans.

[0020] On the other hand, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the method described in any of the above.

[0021] On the other hand, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in any of the preceding claims.

[0022] On the other hand, a computer program product is provided, including computer program instructions that, when run on a computer, cause the computer to perform the method described in any of the preceding claims.

[0023] The technical solution provided in this application offers the following benefits: By generating source tracing information containing the target table and upstream dependencies, it significantly shortens the anomaly localization time. Matching based on a case database allows for the reuse of historical investigation conclusions and remediation plans, avoiding repeated investigations of similar anomalies and effectively accumulating domain knowledge. Inputting the source tracing information and historical cases into a generative language model, integrating structured data with natural language reasoning capabilities, not only outputs accurate diagnostic results but also provides actionable adjustment plans, significantly improving data quality assurance efficiency and business continuity. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of the data diagnostic method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the data diagnostic device structure provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] As a core infrastructure for enterprise data analysis and business decision-making, data warehouses handle the entire data processing chain from data collection, cleaning, and processing to final consumption. In a typical data warehouse architecture, complex upstream and downstream dependencies exist between data tables. For example, in an e-commerce scenario, the final "order summary table" may depend on multiple upstream tables such as the "payment details table," "user information table," and "product inventory table." Any data anomaly in any upstream table can propagate through the dependency chain and affect the data quality of downstream tables. During the daily production of data tables, data quality issues may arise due to various reasons such as upstream data source anomalies, changes in computational logic, system failures, and data skew. Common anomalies include: sudden changes in row count (e.g., a table's row count drops from 1 million to 100,000), a sudden increase in null value rate (e.g., a field's null value rate jumps from 2% to 80%), enumeration value drift (e.g., a large number of new values ​​appear in an enumeration field), and extreme value anomalies (e.g., a monetary value field shows a significant deviation from the normal range). If these anomalies are not detected and diagnosed in a timely manner, they may lead to missing or distorted data in downstream reports, thereby affecting business decisions. For example, anomalies in order data may prevent the timely generation of sales reports, impacting management's daily operational decisions. Therefore, timely detection and accurate diagnosis of data anomalies are of great significance for ensuring data quality and maintaining business continuity.

[0028] Currently, data anomaly detection in data warehouses primarily relies on manually defined static rules for verification. A common practice is for data engineers to set a series of threshold rules for key data tables based on business experience, such as "the daily row count of the order table should not be less than 500,000," "the null value rate of mobile phone numbers in the user table should not exceed 5%," and "the category enumeration values ​​in the product table should not exceed 200." The scheduling system automatically executes these rule verifications after each table is generated, triggering an alarm notification when statistical data exceeds the preset thresholds. After an anomaly is detected and an alarm is received, data engineers typically intervene manually to investigate. The specific investigation process includes: first, logging into the scheduling system to check if the table's output task was successful and if there are any error logs; then, checking the table's DDL (Data Definition Language) records through the metadata system to confirm whether there have been recent table structure changes; next, checking if the table's calculation scripts have been recently modified; if no anomalies are found in the above steps, it is necessary to trace back along the upstream dependency chain layer by layer, checking the output status, data quality, and change records of upstream tables in turn.

[0029] However, when the data chain is long and involves multiple upstream tables, engineers need to manually query the output status, table structure changes, and calculation script changes of each upstream table one by one, and perform correlation analysis to locate the root cause of the anomaly. The entire investigation process can take several hours to several days. For example, after an alarm is triggered in the "Order Summary Table" of an e-commerce platform due to a sudden drop in the number of rows, engineers need to check the output status and change records of multiple upstream tables such as the "Payment Details Table," "User Information Table," and "Product Inventory Table." This may require traversing multiple business systems and contacting multiple teams for assistance before finally locating the root cause as a change in the calculation script of a certain upstream table that caused the data filtering anomaly. Moreover, due to the lack of an automated diagnostic mechanism, even if similar anomalies have been investigated multiple times before, each investigation still needs to start from scratch, and historical investigation conclusions and repair solutions cannot be automatically accumulated and reused.

[0030] This disclosure provides a data diagnostic method. This method can be applied to a terminal. For example... Figure 1 As shown, the terminal may include a processor 110, a memory 120, and a communication component 130.

[0031] Processor 110 can be a central processing unit (CPU), graphics processing unit (GPU), microcontroller unit (MCU), accelerated processing unit (APU), neural processing unit (NPU), tensor processing unit (TPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processor (DSP), etc. Processor 110 can be used to run generative language models, etc.

[0032] Memory 120 may include volatile memory and / or non-volatile memory. Volatile memory may include random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, non-volatile random access memory (NVRAM), etc. Memory 120 may be used to store case databases, etc.

[0033] The communication component 130 can be a wireless communication module (WCM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), an optical network unit (ONU), etc. The communication component 130 can be used for communication between the terminal and a server, or other terminals.

[0034] This application provides a data diagnostic method, such as Figure 2 As shown, in some embodiments, the method includes: S201. Obtain the target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, generate source tracing information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table.

[0035] In practice, aggregation and statistical operations can be performed on the target data table in the data warehouse to obtain the current state statistics of the target data table at the current detection time. The current state statistics include, but are not limited to, statistical indicators such as the number of rows, null value rate, enumeration value distribution, numerical extreme values, numerical mean, and numerical variance of the target data table. In some embodiments, the process of determining whether the current state statistics meet the abnormal conditions includes a first-level detection and a second-level detection, improving the accuracy of anomaly detection through a progressive approach.

[0036] The first layer of detection is rapid screening. After the target data table has produced data, aggregate SQL (Structured Query Language) statements are executed on the current produced partition of the target data table to calculate the current state statistics. These statistics are then compared with the statistical baseline to determine if they meet any anomaly conditions. The statistical baseline is based on statistical characteristics calculated from historical statistical snapshots of the target data table, including but not limited to mean, standard deviation, and quantiles. A statistical snapshot refers to the statistical indicator record generated after performing aggregate statistics on the produced partition of the data table at a specific point in time.

[0037] The first layer of detection includes at least one of the following: Null value rate mutation detection, which compares the null value rate of each field in the target data table with the mean and standard deviation of the null value rate in the statistical baseline. When the null value rate deviates from the baseline mean by more than a preset multiple (e.g., more than 3 times the standard deviation), it is determined to meet the abnormal condition; Row count fluctuation detection, which compares the number of rows in the target data table with the mean and standard deviation of the number of rows in the statistical baseline. When the number of rows deviates from the baseline mean by more than a preset multiple (e.g., more than plus or minus 3 times the standard deviation), it is determined to meet the abnormal condition; Enumeration value drift detection, which compares the distribution of enumeration values ​​in the target data table with the distribution of enumeration values ​​in the statistical baseline. When the number of enumeration values ​​exceeds a preset range or a new enumeration value not included in the statistical baseline appears, it is determined to meet the abnormal condition; Extreme value anomaly detection, which compares the extreme values ​​of the target data table with the extreme value range in the statistical baseline. When the extreme values ​​significantly deviate from the normal range of the statistical baseline, it is determined to meet the abnormal condition; Partition integrity detection, which checks whether the output partitions of the target data table are complete. When there are missing partitions, it is determined to meet the abnormal condition.

[0038] Before the first-level detection, a cold start strategy is implemented for newly connected target data tables in the data warehouse due to the lack of sufficient historical statistical snapshots to calculate the statistical baseline. During the cold start, only statistical indicator collection and statistical snapshot storage are performed; the baseline comparison and anomaly detection mentioned above are not performed. The number of executions during the cold start is a preset number (e.g., 7 times), and the first-level detection is enabled after sufficient samples are accumulated. During the cold start, extreme anomalies that can be determined without a historical baseline are still monitored, including zero rows (indicating data write failure) and a null value rate of all fields exceeding a preset extreme threshold (e.g., exceeding 99%). When the above extreme anomalies are detected, an alarm is triggered in the form of a low-level notification.

[0039] When the first-layer detection determines that the current statistical data meets the abnormal conditions, the second-layer detection is triggered. The second-layer detection is a deep analysis used to further confirm whether there has been a substantial change in the data distribution, in order to eliminate possible false alarms from the first-layer detection. The first-layer detection is based on aggregated statistical indicators, but in some scenarios, changes in aggregated statistical indicators do not reflect substantial data anomalies. For example, a proportional increase in the number of rows due to natural business growth (a sudden increase in the number of rows but normal proportions of each field, not an anomaly), local anomalies masked by aggregated statistics (the total number of rows remains unchanged, but the data distribution of a certain partition is skewed), and normal periodic fluctuations (such as a natural decrease in order volume on weekends) are falsely reported as anomalies. The second-layer detection addresses the blind spots of the first-layer detection by performing deep verification through at least one of the following: distribution offset detection, time-series pattern matching, and cluster anomaly detection.

[0040] Distribution shift detection quantifies the overall distribution difference between the current and historical data of a target data table using statistical tests. Specifically, it calculates the divergence between the current and historical data distributions. When the divergence exceeds a preset divergence threshold, a substantial shift in the data distribution is determined. Statistical tests include, but are not limited to, KL (Kullback-Leibler) divergence and JS (Jensen-Shannon) divergence. When the data distribution is complex and difficult to determine as substantial anomaly using rules, a generative language model is used to perform descriptive analysis of the distribution differences to assist in determining whether a substantial anomaly exists. The model prompt can include the following information areas: a current distribution description area for entering statistical descriptions of the current data distribution; a historical distribution description area for entering statistical descriptions of historical data distributions; a divergence value area for entering the divergence value between the current and historical data distributions; and an analysis task area to instruct the generative language model to determine whether the distribution difference constitutes a substantial anomaly and to explain the reasons. For example, the generated model prompt might be: "You are a data warehouse quality analysis expert." The following is the statistical information on the data distribution of key fields in a data table. Please determine whether the difference between the current distribution and the historical distribution constitutes a substantial anomaly, and explain your reasoning. Current Distribution (2025-03-15): Field "Order Amount": Mean 85.3 yuan, Median 62.0 yuan, Standard Deviation 120.5 yuan, Skewness 3.2, Kurtosis 12.8. Bucket Distribution (by Amount Range): [0,50) 35%, [50,100) 28%, [100,200) 18%, [200,500) 12%, [500,1000) 5%, [1000,+∞) 2%. Field "User Level": Enumerated value distribution is {"Regular User": 55%, "Silver Card User": 25%, "Gold Card User": 12%, "Diamond User": 6%, "New User": 2%}. Historical Distribution (30-day baseline): Field "Order Amount": Mean 82.1 yuan, Median 60.5 yuan, Standard Deviation 95.0 yuan, Skewness 2.1, Kurtosis 8.5. Bucket Distribution: [0,50) 38%, [50,100) 30%, [100,200) 17%, [200,500) 10%, [500,1000) 4%, [1000,+∞) 1%. Field "User Level": Enumerated value distribution is {"Regular User": 58%, "Silver Card User": 24%, "Gold Card User": 11%, "Diamond User": 5%, "New User": 2%}. KL Divergence Values: Order Amount field 0.12, User Level field 0.03.Please analyze whether the above distribution differences constitute a substantial anomaly, and give the reasons for judgment from the following perspectives: (1) whether the magnitude of the difference has business significance; (2) whether the distribution pattern of the difference shows an abnormal pattern (such as unilateral skewness, long-tail mutation, etc.); (3) whether the difference may be caused by normal business changes (such as promotional activities, seasonal fluctuations, etc.).

[0041] After analyzing the model prompts mentioned above, the generative language model outputs its analysis conclusions. For example, when the distribution difference is mainly caused by a small number of extreme values ​​and the overall distribution pattern has not fundamentally changed, the generative language model outputs "The distribution difference does not constitute a substantial anomaly. The current distribution is basically consistent with the overall pattern of the historical distribution. The difference is mainly caused by a small number of high-value orders. It is recommended to pay attention to the extreme value detection item." When the distribution difference shows obvious structural changes, the generative language model outputs "The distribution difference constitutes a substantial anomaly. The current distribution shows a significant increase in right skewness, and the proportion of high-value order intervals has increased significantly. It is recommended to further investigate whether there are changes in data processing logic or upstream data anomalies." Based on the analysis conclusions output by the generative language model, combined with the results of the first layer of detection, a comprehensive determination is made as to whether the current state statistical data meets the anomaly conditions.

[0042] Time-series pattern matching compares the current state statistics of the target data table with historical data from the same period last week, the same day last month, and other historical periods. Specifically, it retrieves the current state statistics of the target data table for the same period last week, the same day last month, etc., calculates the difference between the current fluctuation and the historical fluctuation, and determines that the current fluctuation does not belong to the normal periodic fluctuation and meets the abnormal condition when the difference exceeds a preset threshold. When the current fluctuation shows a consistent periodic pattern with the historical fluctuation, it is determined that the current fluctuation belongs to the normal periodic fluctuation, does not meet the abnormal condition, and excludes false alarms.

[0043] Cluster anomaly detection involves performing multidimensional clustering analysis on data from a target data table to precisely locate anomalous data segments. Specifically, multidimensional clustering is performed on the data based on key fields of the target data table. The distribution characteristics of each clustered data segment are then compared with the corresponding segments in a statistical baseline. When the statistical indicator of a data segment deviates from the baseline by more than a preset deviation threshold, that data segment is determined to have a local anomaly, meeting the anomaly criteria. Through cluster anomaly detection, local anomalies masked by aggregated statistics can be discovered.

[0044] Let's take a specific order detail table in a data warehouse as an example. Assume the data warehouse contains the following tables and their dependencies: Order detail table D depends on order log table A, user information table B, and product dimension table C. Order log table A records the original order log data, user information table B stores basic user information, and product dimension table C stores product attribute information. The data processing task for order detail table D retrieves order log data from order log table A and associates it with user information table B and product dimension table C to generate complete order detail data. The data traceability graph includes the following nodes and directed edges: node A (order log table), node B (user information table), node C (product dimension table), node D (order detail table), and directed edges A→D, B→D, and C→D.

[0045] Execute an aggregate SQL statement on the target data table "Order Details Table D" to obtain the following current status statistics: 320,000 rows of data for the day; null value rate for each field is 15.6%; the average value of the key field "Order Amount" is 85.3 yuan; the number of enumerated values ​​for the key field "Order Status" is 25; and the extreme value of the key field "Order Amount" is -50,000 yuan. First-level detection: Obtain the statistical baseline of the target data table "Order Details Table D". This table has been generated normally for more than 30 days. The statistical baseline is calculated based on the statistical snapshots of the last 30 times. The statistical baseline includes: the average number of rows is 510,000 rows, and the standard deviation of the number of rows is 15,000 rows; the average null value rate of each field is 2%, and the standard deviation of the null value rate is 0.5%; the number of enumeration values ​​for "order status" is 18, and the set of enumeration values ​​included in the baseline is {"pending payment", "paid", "shipped", "received", "cancelled", "refunding in progress", "refunded"}; the extreme value range of "order amount" is [0, 100,000] yuan. The current status statistics are compared with the baseline: the number of rows (320,000) deviates from the baseline mean of 510,000 rows by approximately 12.7 standard deviations, exceeding the preset threshold of 3 standard deviations, indicating an anomaly in the row count fluctuation detection; the null value rate (15.6%) deviates from the baseline mean of 2% by approximately 27.2 standard deviations, exceeding the preset threshold of 3 standard deviations, indicating an anomaly in the null value rate mutation detection; the number of "order status" enumeration values ​​(25) exceeds the baseline enumeration value number of 18, and a new enumeration value not included in the baseline appears ("abnormally frozen"), indicating an anomaly in the enumeration value drift detection; the extreme value of "order amount" (-50,000 yuan) significantly deviates from the baseline range [0, 100,000] yuan, indicating an anomaly in the extreme value anomaly detection. The first layer of detection determines that the current status statistics meet the anomaly conditions, triggering the second layer of detection. The second layer of detection is distribution offset detection. The KL divergence between the current key field data distribution and the historical data distribution of the target data table "Order Details Table D" is calculated. The calculated KL divergence for the row distribution is 0.85, exceeding the preset divergence threshold of 0.5, indicating a substantial data distribution offset. Time-series pattern matching is then performed, obtaining the current status statistics of the target data table for the same day last week (March 8, 2025) and the same day last month (February 15, 2025). The number of rows on the same day last week was 505,000 with a null value rate of 1.8%, and the number of rows on the same day last month was 498,000 with a null value rate of 2.1%. The current number of rows (320,000) is approximately 37% lower than the historical average for the same period, with the difference far exceeding the preset difference threshold of 10%, indicating that the current fluctuation is not a normal periodic fluctuation. Cluster anomaly detection: Based on the "order source" field, multidimensional clustering was performed on the target data table data. It was found that the number of rows in the data segment from "channel A" dropped sharply from the normal 350,000 to 130,000, which deviated from the baseline by more than the preset deviation threshold. It was determined that there was a local anomaly in the data segment.The second layer of detection confirms that the current status statistics meet the abnormal conditions, determines that there is data anomaly in the target data table "Order Details Table D", and triggers the anomaly diagnosis process.

[0046] When the first-layer detection and / or the second-layer detection determine that the current state statistical data meets the abnormal conditions, it is determined that there is a data anomaly in the target data table, triggering the anomaly diagnosis process. At this time, based on the target data table and its upstream data tables, source tracing information for anomaly analysis of the target data table is generated. In some embodiments, firstly, a data source tracing relationship diagram corresponding to the target data table is obtained; then, based on the data source tracing relationship diagram, the upstream data tables of the target data table are determined; finally, source tracing information is generated based on the target data table and its upstream tables.

[0047] S202. In the case database, search for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis results and target adjustment schemes corresponding to the target traceability information.

[0048] In practice, feature vectors of the source tracing information are extracted. A feature vector is a numerical vector representation of the source tracing information obtained after vectorization, used to represent the features of the source tracing information in a vector space. In the case database, target source tracing information that meets a specified vector similarity condition with the source tracing information is searched based on the feature vectors, and the corresponding target diagnostic results and target adjustment schemes are obtained. The case database stores multiple historical anomaly diagnostic records. Each historical anomaly diagnostic record includes the feature vector of the historical source tracing information, the corresponding historical diagnostic results, and the historical adjustment scheme. The vector similarity condition stipulates that the similarity between feature vectors should meet a preset similarity threshold. When the similarity between the feature vector and the feature vector of the historical source tracing information of a certain historical anomaly diagnostic record in the case database is greater than or equal to the preset similarity threshold, the historical source tracing information is identified as the target source tracing information, and its corresponding historical diagnostic results are obtained as the target diagnostic results, and its corresponding historical adjustment schemes are obtained as the target adjustment schemes.

[0049] S203. Input the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment plan into a generative language model to diagnose the abnormal causes of the target data table and obtain the diagnosis result and adjustment plan.

[0050] In practice, the source information, target source information, target diagnostic results, and target adjustment plans are first inserted into the prompt word template to obtain model prompt words. The prompt word template is a predefined template structure used to construct input prompt words for the generative language model. It contains multiple information insertion positions for inserting source information, target source information, target diagnostic results, and target adjustment plans, respectively. The model prompt words instruct the generative language model to diagnose the causes of anomalies in the target data table based on the source information, target source information, target diagnostic results, and target adjustment plans. Then, the model prompt words are input into the generative language model to obtain the diagnostic results and adjustment plans. Based on the anomaly features described in the source information, the status of upstream data tables, cascading propagation information, data reconciliation results, change records, and other contextual information, combined with the diagnostic results and adjustment plans of historical similar cases corresponding to the target source information, the generative language model comprehensively analyzes the causes of anomalies in the target data table and outputs the diagnostic results and adjustment plans.

[0051] The diagnostic results include an analysis and explanation of the causes of anomalies in the target data table. These causes include, but are not limited to: anomalies in the target data table's own data processing logic, anomalies in cascading propagation from upstream data tables, anomalies caused by changes in table structure, and anomalies caused by changes in processing logic. The adjustment plan includes recommended remedial measures for the diagnosed anomalies, including but not limited to modifying data processing scripts, rolling back table structure changes, re-executing data processing tasks, and repairing upstream data table data.

[0052] In some embodiments, the method further includes a self-accumulation step, in which the diagnostic results and adjustment plans obtained from the current diagnosis are stored as new case data in the case database, and the feature vector of the corresponding source information is updated. Through the self-accumulation mechanism, the case database is continuously enriched, and the accuracy and efficiency of subsequent anomaly diagnoses are continuously improved.

[0053] In this embodiment, source tracing information is generated based on the target data table and its upstream data table. Combined with diagnostic results and adjustment plans from historical similar anomalies in the case database, a generative language model is used for comprehensive analysis and diagnosis, achieving intelligent diagnosis of the causes of data anomalies. This significantly improves anomaly diagnosis efficiency and shortens anomaly location time (from hours or days to minutes). Furthermore, the comprehensive analytical capabilities of the generative language model enhance the accuracy and comprehensiveness of the diagnostic results.

[0054] In some embodiments, generating source information for anomaly analysis of the target data table based on the target data table and its upstream data tables includes: Obtain the data tracing relationship diagram corresponding to the target data table; based on the data tracing relationship diagram, determine the upstream data table of the target data table; based on the target data table and the upstream data table, generate the tracing information.

[0055] In practice, the data lineage relationships between tables in the data warehouse are obtained through the API (Application Programming Interface) provided by the data lineage system. Data lineage relationships refer to the upstream and downstream dependencies formed during the generation, processing, and flow of data to final consumption. A data lineage graph is constructed based on these relationships. This graph is a directed acyclic graph (DAG), where each node represents a data table, and each directed edge indicates the direction of data flow between tables, pointing from upstream to downstream tables. In the data lineage graph, traversal is performed starting from the node corresponding to the target data table and proceeding in the opposite direction of the directed edges (i.e., upstream). Traversal strategies can employ breadth-first search (BFS) or depth-first search (DFS). The number of traversal levels can be configured according to actual needs, such as traversing all upstream nodes, only traversing directly upstream nodes (i.e., one level upstream), or traversing upstream nodes within a specified number of levels. During the traversal, the data tables corresponding to each upstream node are identified as upstream data tables of the target data table. Upstream data tables include direct upstream data tables (upstream tables that have a direct data processing dependency on the target data table) and indirect upstream data tables (upstream tables that have an indirect data processing dependency on the target data table through direct upstream data tables). After identifying the target data table and upstream data tables, relevant information about them is collected, including but not limited to: abnormal characteristic data of the target data table (abnormality type, abnormal statistical index values, etc.), data output status of each upstream data table (whether output has been completed, output completion time, etc.), data reconciliation results between the target data table and each upstream data table, cascading propagation analysis results between the target data table and each upstream data table, table structure change records and processing logic change records of the target data table and each upstream data table, etc. This information is then integrated to generate traceability information.

[0056] In this embodiment, by obtaining the data source relationship diagram corresponding to the target data table, determining the upstream data table based on the data source relationship diagram, and generating source information based on the target data table and the upstream data table, a systematic upstream dependency tracing is achieved. This avoids the tedious process of manual layer-by-layer investigation, supports multi-level cascading anomaly analysis, significantly shortens the upstream dependency identification time, and improves the efficiency of anomaly diagnosis.

[0057] In some embodiments, determining the upstream data table of the target data table based on the data tracing relationship diagram includes: Starting from the node corresponding to the target data table in the data tracing relationship diagram, the process traverses upstream layer by layer, and determines the data table corresponding to each upstream node as the upstream data table of the target data table.

[0058] In practice, the node corresponding to the target data table in the data source graph is located and designated as the current node. A traversal queue (or stack) is initialized, and the current node is added to the queue. A set of visited nodes is initialized to record visited nodes, avoiding duplicate traversals and infinite loops. The current node is added to the set of visited nodes. Then, a node is retrieved from the traversal queue and designated as the node to be processed. All directed edges pointing to the node to be processed are queried in the data source graph, and the starting nodes of these edges are identified as the direct upstream nodes of the node to be processed. For each direct upstream node, it is determined whether it is already in the set of visited nodes. If not, the data table corresponding to the direct upstream node is identified as the upstream data table of the target data table, and the direct upstream node is added to both the traversal queue and the set of visited nodes.

[0059] Repeat the above process until the traversal queue is empty, meaning all reachable upstream nodes have been traversed. After the traversal is complete, the data tables corresponding to all nodes in the visited node set, except for the node corresponding to the target data table, are determined to be the upstream data tables of the target data table.

[0060] In this embodiment, by starting from the node corresponding to the target data table in the data tracing relationship graph and traversing layer by layer along the upstream direction, the data tables corresponding to each upstream node traversed are determined as the upstream data tables of the target data table. This realizes an automated and systematic upstream data table discovery process, avoiding the problem of missing some upstream tables during manual investigation.

[0061] In some embodiments, generating the tracing information based on the target data table and the upstream data table includes: Based on the data processing dependency between the target data table and the upstream data table, upstream and downstream data reconciliation operations are performed to obtain data reconciliation results; the traceability information is generated based on the data reconciliation results.

[0062] In practice, the definition information of the data processing tasks for the target data table is obtained. This definition includes the data processing script, input table list, and output table list for the target data table. From this definition, the data processing dependencies between the target data table and each upstream data table are identified. This includes how data from each upstream data table participates in the generation of data in the target data table (e.g., as an input table for join queries, as a data source for data aggregation), and the type of data processing logic (e.g., join queries, aggregation calculations, data filtering, field mapping). Then, reconciliation SQL statements are derived and generated based on these dependencies. These reconciliation SQL statements are used to verify the consistency between the data in the upstream and target data tables at the data level. The derivation rules for these reconciliation SQL statements include, but are not limited to: generating SQL statements that compare the number of rows in the target data table with the number of rows in the upstream data tables to verify whether the data are consistent in magnitude. For example, when the target data table is generated from the upstream data table through simple filtering, the number of rows in the target data table should not exceed the number of rows in the upstream data table. SQL statements are also generated that compare the aggregated values ​​of key numerical fields in the target data table with the aggregated values ​​of the corresponding fields in the upstream data tables. For example, when the target data table is generated by grouping and aggregating upstream data tables, the SUM value of a field in the target data table should be consistent with the SUM value of the corresponding field in the upstream data table. Generate SQL statements to compare the record matching between the target data table and the upstream data table, checking for isolated records in the target data table that cannot be found in the upstream data table, and for any missing records in the upstream data table that are not reflected in the target data table.

[0063] Next, the reconciliation SQL statements are executed to obtain the data reconciliation results. These results include the verification conclusions (pass or fail) for each verification item, as well as specific verification data (such as differences in row counts, aggregate value differences, and match rates). Finally, the data reconciliation results are used as part of the source tracing information. When the data reconciliation results indicate a data inconsistency between the target data table and an upstream data table, it suggests a potential anomaly in the data processing between the upstream and target data tables, providing direct data-level evidence for subsequent root cause diagnosis.

[0064] In this embodiment, by performing upstream and downstream data reconciliation operations based on the data processing dependency relationship between the target data table and the upstream data table, the data reconciliation results are obtained, and traceability information is generated based on the data reconciliation results. This can help discover inconsistencies in the data flow process between upstream and downstream, providing direct data-level evidence for locating the root cause of anomalies.

[0065] In some embodiments, generating the tracing information based on the target data table and the upstream data table includes: The process involves: identifying the abnormal characteristic data of the target data table and its first data output timestamp; determining the data output status of the upstream data table before the first data output timestamp and obtaining the second data output timestamp of the upstream data table that has completed data output; determining that the upstream data table meets the cascading propagation condition based on the time difference between the first and second data output timestamps, and generating cascading propagation information based on the target data table and the upstream data table that meets the cascading propagation condition; and generating the tracing information based on the abnormal characteristic data of the target data table, the data output status of the upstream data table, and the cascading propagation information.

[0066] In practice, anomaly characteristic data includes anomaly type identifiers (such as a sudden drop in row count, a sudden increase in null value rate, enumeration value drift, extreme value anomalies, etc.) and anomaly statistical indicator values ​​(such as current row count, current null value rate, current enumeration value count, etc.). The first data output timestamp is the completion time of the current data output of the target data table, which can be obtained through the scheduling system. The status and completion time of the most recent data output of each upstream data table before the first data output timestamp are queried through the scheduling system's API. For upstream data tables that have completed data output, their data output completion time is obtained as the second data output timestamp. For upstream data tables that have not yet completed data output, their current output status is recorded as "output in progress" or "output failed". The cascading time window condition stipulates that the time difference should be within a preset time window range, for example, the time difference should not exceed 60 minutes. For each upstream data table that has completed data output, the time difference between the first data output timestamp and the second data output timestamp is calculated. When the time difference is within the time window, the upstream data table is determined to meet the cascading propagation condition, meaning that anomalies in the upstream data table may temporally affect the data output of the target data table. When the time difference exceeds the time window, the upstream data table is determined not to meet the cascading propagation condition, meaning that anomalies in the upstream data table are unlikely to affect the target data table through cascading propagation. When an upstream data table meets the cascading propagation condition, cascading propagation information is generated based on the target data table and the upstream data tables that meet the cascading propagation condition. The cascading propagation information includes: the identifier of the upstream data table that meets the cascading propagation condition, the anomaly characteristics of the upstream data table, the time difference between the first data output timestamp and the second data output timestamp, and the hierarchical relationship of the upstream data table relative to the target data table. Based on the anomaly characteristic data of the target data table, the data output status of the upstream data tables, and the cascading propagation information, source tracing information is generated. The source tracing information contains the results of the cascading propagation analysis, used to indicate which upstream data tables' anomalies may have caused the target data table's anomalies, providing temporal-level evidence for subsequent root cause diagnosis.

[0067] This embodiment can effectively distinguish whether the anomaly of the target data table is caused by its own reasons or by the propagation of upstream data tables. It avoids the cumbersome process in the prior art where engineers need to manually compare the output time of each upstream table and manually judge whether there is cascading propagation. This improves the accuracy of root cause localization and shortens the anomaly diagnosis time.

[0068] In some embodiments, generating the tracing information based on the target data table and the upstream data table includes: Obtain table structure change records and processing logic change records of the target data table and the upstream data table within a specified time period, wherein the specified time period is the time period corresponding to a specified time window before the first data production timestamp; generate the tracing information based on the table structure change records and the processing logic change records.

[0069] In practice, the specified time period is the period corresponding to the specified time window preceding the timestamp of the first data output. The specified time window can be configured according to actual needs, such as 24 hours, 48 ​​hours, or 72 hours. For example, when the timestamp of the first data output is "2025-03-15 06:30:00" and the specified time window is 24 hours, the specified time period is from "2025-03-14 06:30:00" to "2025-03-15 06:30:00".

[0070] Then, through the metadata query interface provided by the database management system, the DDL logs of each data table within a specified time period are queried. Table structure change records include, but are not limited to: field additions, deletions, modifications (including changes to field names, field types, field lengths, etc.), index creation or deletion, and partitioning strategy changes. For each data table, the system parses the DDL logs into structured change records, recording change time, change type, and change details. Next, through the version management system for data processing tasks, the change records of each data table's data processing tasks within a specified time period are queried. Processing logic change records include, but are not limited to: modifications to data processing scripts (such as changes to SQL statements), changes to calculation formulas, changes to data filtering conditions, changes to join logic, and changes to scheduling parameters. For each data processing task, by comparing the script content of different versions (such as text diffs), the specific changes are extracted, and the change time, change type, and change details are recorded.

[0071] Finally, the table structure change records and processing logic change records of the target data table and each upstream data table are integrated into the source information. When an upstream data table undergoes a table structure change or processing logic change within a specified time period, the source information records detailed information about the change, indicating that an anomaly in the target data table may be caused by the change in that upstream data table. For example, if an upstream data table performs a field renaming operation within a specified time period, but the data processing script of the target data table still references the old field names, the change records in the source information can provide crucial clues.

[0072] In this embodiment, change record information within a specified time period is obtained and integrated into the source information, providing a complete change context for the generative language model. This enables the model to accurately identify anomalies caused by changes, improving the accuracy and relevance of the diagnosis.

[0073] In some embodiments, the step of searching for target tracing information in the case database that meets similar conditions to the tracing information, and obtaining the target diagnosis result and target adjustment plan corresponding to the target tracing information, includes: Extract the feature vector of the source information, and in the case database, find the target source information that meets the specified vector similarity condition with the source information based on the feature vector, and obtain the target diagnosis result and target adjustment plan corresponding to the target source information.

[0074] In practice, the source tracing information is structured and then mapped to fixed-dimensional feature vectors using a pre-trained vector encoder (which can be all source tracing information or only the abnormal feature data within it). The vector encoder can be a deep learning-based text encoder (such as BERT, Sentence-BERT, etc.) or an encoder specifically trained for abnormal features. Then, a case database stores the feature vectors of historical source tracing information corresponding to multiple historical abnormal diagnosis records. The vector similarity between the feature vector of the current source tracing information and the feature vectors of each historical source tracing information in the case database is calculated. The calculation methods for vector similarity include, but are not limited to, cosine similarity, Euclidean distance, and Manhattan distance. When using cosine similarity, the similarity value ranges from [0,1], with a larger value indicating greater similarity between the two vectors. The vector similarity condition stipulates that the similarity between feature vectors should be greater than or equal to a preset similarity threshold (e.g., 0.85).

[0075] In one possible implementation, case retrieval employs a two-layer search strategy. The first layer is intra-table retrieval: it prioritizes searching the case database for historical anomaly records from tables identical to the target table, calculating vector similarity. If a historical anomaly record satisfying the vector similarity condition is found, its historical source information is designated as the target source information. The second layer is cross-table extended retrieval: when intra-table retrieval fails to find a historical anomaly record satisfying the vector similarity condition, the search is extended to historical anomaly records from other tables. During cross-table extended retrieval, weighted averages can be applied based on anomaly type similarity, with historical records of the same or similar anomaly types receiving higher search weights.

[0076] Once the target source information is determined, the historical diagnostic results corresponding to that target source information are retrieved from the case database as the target diagnostic results, and the corresponding historical adjustment plans are retrieved as the target adjustment plans. The target diagnostic results are the analytical conclusions regarding the causes of data anomalies in similar historical anomaly scenarios, while the target adjustment plans are the remedial measures taken for data anomalies in similar historical anomaly scenarios. The target diagnostic results and target adjustment plans will be used as reference information input into the generative language model to assist the generative language model in diagnosing the current anomaly.

[0077] This embodiment automatically matches historical similar cases through vector similarity retrieval, enabling the accumulation and reuse of anomaly diagnosis experience. Simultaneously, a two-layer retrieval strategy—prioritizing cases within the same table and expanding across tables—ensures both the accuracy of case matching (cases within the same table have higher relevance) and the coverage of case retrieval (expanding to cases in other tables when there are insufficient cases within the same table), thus improving both the accuracy and coverage of case matching.

[0078] In some embodiments, inputting the source tracing information, the target source tracing information, the target diagnostic results, and the target adjustment plan into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnostic results and adjustment plans includes: The source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment plan are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to diagnose the abnormal causes of the target data table based on the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment plan. The model prompt words are then input into the generative language model to obtain the diagnosis result and the adjustment plan.

[0079] The prompt word template includes settings for the following information insertion locations: anomaly description area, upstream dependency analysis area, reconciliation results area, change record area, output time series area, cascading propagation analysis area, and historical case area. Examples of generated model prompt words include: You are a data warehouse anomaly diagnosis expert. Based on the following information, diagnose the cause of the anomaly in the target data table and provide remediation suggestions.

[0080] I. Description of Target Data Table Anomalies Target data table: Order details table D.

[0081] Anomaly characteristics: The number of data rows dropped sharply from the normal 500,000 rows to 320,000 rows; the null value rate of key fields increased from the normal 2% to 15.6%.

[0082] Current status statistics: 320,000 rows, null value rate 15.6%, average order amount 85.3 yuan.

[0083] II. Upstream Data Table Dependencies The upstream data tables include: Order Transaction Table A (direct upstream), User Information Table B (direct upstream), and Product Dimension Table C (direct upstream).

[0084] Data processing dependencies: The data processing task of Order Details Table D obtains order transaction data from Order Transaction Table A, and generates it by associating it with User Information Table B and Product Dimension Table C.

[0085] III. Results of Upstream and Downstream Data Reconciliation Reconciliation with Order Transaction Table A: Row count mismatch failed. Order Transaction Table A generated 510,000 records, while Order Details Table D had 320,000 rows, resulting in a discrepancy of 190,000 records.

[0086] Reconciliation with user information table B: The correlation verification pass rate is 98.7%, which is basically successful.

[0087] Reconciliation with product dimension table C: The correlation verification pass rate is 99.1%, which is basically successful.

[0088] IV. Change Records The target data table "Order Details Table D" has no changes in table structure or processing logic within the specified time period.

[0089] Upstream data table "Order Transaction Table A": The table structure change record shows that the field "customer_addr" has been renamed to "customer_address" and a new field "logistics_no" has been added; the processing logic change record shows that the ETL script has been updated.

[0090] Upstream data table "User Information Table B": No change records.

[0091] Upstream data table "Product Dimension Table C": No change records.

[0092] V. Output Time Sequence of Upstream Data Tables Order Flow Sheet A Production Completion Time: 2025-03-15 06:20:00.

[0093] User Information Table B was generated on 2025-03-15 at 05:00:00.

[0094] Product dimension table C was completed on 2025-03-15 at 04:30:00.

[0095] Target data table D output completion time: 2025-03-15 06:30:00.

[0096] VI. Cascade Propagation Analysis Order Flow Chart A: Time difference of 10 minutes, meets the conditions for cascading propagation.

[0097] User Information Table B: The time difference is 90 minutes, which does not meet the conditions for cascading propagation.

[0098] Product Dimension Table C: Time difference of 120 minutes does not meet the conditions for cascading propagation.

[0099] VII. Similar Historical Cases Historical cases with a similarity of 0.92 were retrieved from the anomaly knowledge base: Anomaly: Upstream table DDL changes cause downstream table join failures, resulting in a sharp drop in the number of data rows.

[0100] Historical diagnostic results: Renaming of upstream table fields caused downstream join queries to fail, resulting in a significant decrease in the number of data rows.

[0101] Historical adjustment plan: Roll back the upstream table field renaming changes, update the downstream table data processing script field references, and re-execute the data processing task.

[0102] Based on the above information, analyze the root cause of the anomaly in the target data table "Order Details Table D" and provide specific repair suggestions.

[0103] The generative language model performs a comprehensive analysis based on the above model prompts and generates the following diagnostic results: Diagnostic Results: The root cause of the anomaly in the target data table "Order Details Table D" was a change in the table structure of the upstream data table "Order Transaction Table A" (the field "customer_addr" was renamed to "customer_address") and a change in processing logic (ETL script update). This caused the old field name "customer_addr" referenced in the data processing script of the target data table "Order Details Table D" to be unavailable in the new version of Order Transaction Table A, resulting in a failed join query. Consequently, the number of data rows plummeted from 510,000 to 320,000. Simultaneously, the corresponding fields of the failed join records contained null values, causing the null value rate to rise from 2% to 15.6%. Cascading propagation analysis confirmed that the output time of Order Transaction Table A differed from that of the target data table D by only 10 minutes, satisfying the cascading propagation condition. Furthermore, the data reconciliation results confirmed a row count inconsistency between Order Transaction Table A and Order Transaction Table A, further verifying that the anomaly originated from Order Transaction Table A.

[0104] Adjustment plan: Roll back the renamed field "customer_address" in Order Transaction Table A to its original name "customer_addr", or update the data processing script in the target data table "Order Details Table D" to change the field reference "customer_addr" to "customer_address"; check and adapt the impact of the newly added field "logistics_no" on the data processing logic; re-execute the data processing task in the target data table "Order Details Table D"; after execution, re-perform data reconciliation and verification to confirm that the number of data rows has returned to normal levels and the null value rate has dropped below the threshold.

[0105] At the same time, the diagnostic results and adjustment plan will be stored as new case data in the case database, and the feature vector of the source information will be updated to achieve self-accumulation of the knowledge base.

[0106] This embodiment uses a structured design of prompt word templates to organize multi-dimensional information such as anomaly features, upstream dependency analysis, reconciliation results, change records, output sequence, cascading propagation analysis, and historical cases in an orderly manner. This enables the generative language model to conduct comprehensive analysis based on sufficient and structured background information, avoiding information omissions or confusion, and improving the interpretability of diagnostic results and the pertinence of adjustment plans.

[0107] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0108] Based on the same inventive concept, and corresponding to the data diagnostic method provided in the embodiments of this application, this application also provides a data diagnostic device.

[0109] refer to Figure 3 The data diagnostic device includes: The acquisition module 301 is configured to acquire a target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, it generates source information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table. The search module 302 is configured to search in the case database for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis result and target adjustment plan corresponding to the target traceability information; The generation module 303 is configured to input the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme into a generative language model to diagnose the abnormal causes of the target data table and obtain the diagnosis result and the adjustment scheme.

[0110] In one possible implementation, the acquisition module 301 is used to: Obtain the data source relationship diagram corresponding to the target data table; Based on the data tracing relationship diagram, the upstream data table of the target data table is determined; The tracing information is generated based on the target data table and the upstream data table.

[0111] In another possible implementation, the acquisition module 301 is used for: Starting from the node corresponding to the target data table in the data tracing relationship diagram, the process traverses upstream layer by layer, and determines the data table corresponding to each upstream node as the upstream data table of the target data table.

[0112] In another possible implementation, the acquisition module 301 is used for: Based on the data processing dependency between the target data table and the upstream data table, perform upstream and downstream data reconciliation operations to obtain data reconciliation results; The traceability information is generated based on the data reconciliation results.

[0113] In another possible implementation, the acquisition module 301 is used for: Determine the abnormal feature data of the target data table and the first data output timestamp; Determine the data output status of the upstream data table before the first data output timestamp, and obtain the second data output timestamp of the upstream data table that has completed data output; In response to the time difference between the first data output timestamp and the second data output timestamp satisfying the specified cascading time window condition, it is determined that the upstream data table satisfies the cascading propagation condition, and cascading propagation information is generated based on the target data table and the upstream data table that satisfies the cascading propagation condition; The source tracing information is generated based on the abnormal feature data of the target data table, the data output status of the upstream data table, and the cascading propagation information.

[0114] In another possible implementation, the acquisition module 301 is used for: Obtain the table structure change record information and processing logic change record information of the target data table and the upstream data table within a specified time period. The specified time period is the time period corresponding to the specified time window before the first data output timestamp. The tracing information is generated based on the table structure change record information and the processing logic change record information.

[0115] In another possible implementation, the lookup module 302 is used for: Extract the feature vector of the source information, and in the case database, find the target source information that meets the specified vector similarity condition with the source information based on the feature vector, and obtain the target diagnosis result and target adjustment plan corresponding to the target source information.

[0116] In another possible implementation, the generation module 303 is used for: The source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to diagnose the abnormal causes of the target data table based on the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme. The model prompts are input into a generative language model to obtain diagnostic results and adjustment plans.

[0117] It should be noted that the data diagnostic device provided in the above embodiments is only illustrated by the division of the above functional modules during data diagnosis. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data diagnostic device and the data diagnostic method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0118] Based on the same inventive concept, corresponding to the data diagnostic method provided in the embodiments of this application, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data diagnostic method described in the above embodiments.

[0119] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0120] The processor 1010 can be implemented using a general-purpose CPU (central processing unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0121] The memory 1020 can be implemented in the form of ROM (read-only memory), RAM (random access memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0122] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0123] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0124] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0125] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0126] The electronic devices described above are used to implement the corresponding data diagnostic methods in the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the data diagnostic method described above. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (read-only memory), RAM (random access memory), CD-ROM (compact disc read-only memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0128] In an exemplary embodiment, a computer program product is also provided, including computer program instructions that, when executed on a computer, cause the computer to perform the data diagnostic method described above.

[0129] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0130] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0131] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0132] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data diagnostic method, characterized in that, include: Obtain the target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, generate source tracing information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table. In the case database, search for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis results and target adjustment schemes corresponding to the target traceability information; The source tracing information, the target source tracing information, the target diagnosis results, and the target adjustment plan are input into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnosis results and adjustment plans.

2. The data diagnostic method according to claim 1, characterized in that, The step of generating source information for anomaly analysis of the target data table based on the target data table and its upstream data table includes: Obtain the data source relationship diagram corresponding to the target data table; Based on the data tracing relationship diagram, the upstream data table of the target data table is determined; The tracing information is generated based on the target data table and the upstream data table.

3. The data diagnostic method according to claim 2, characterized in that, The step of determining the upstream data table of the target data table based on the data source relationship diagram includes: Starting from the node corresponding to the target data table in the data tracing relationship diagram, the process traverses upstream layer by layer, and determines the data table corresponding to each upstream node as the upstream data table of the target data table.

4. The data diagnostic method according to claim 2, characterized in that, The step of generating the tracing information based on the target data table and the upstream data table includes: Based on the data processing dependency between the target data table and the upstream data table, perform upstream and downstream data reconciliation operations to obtain data reconciliation results; The traceability information is generated based on the data reconciliation results.

5. The data diagnostic method according to claim 2, characterized in that, The step of generating the tracing information based on the target data table and the upstream data table includes: Determine the abnormal feature data of the target data table and the first data output timestamp; Determine the data output status of the upstream data table before the first data output timestamp, and obtain the second data output timestamp of the upstream data table that has completed data output; In response to the time difference between the first data output timestamp and the second data output timestamp satisfying the specified cascading time window condition, it is determined that the upstream data table satisfies the cascading propagation condition, and cascading propagation information is generated based on the target data table and the upstream data table that satisfies the cascading propagation condition; The source tracing information is generated based on the abnormal feature data of the target data table, the data output status of the upstream data table, and the cascading propagation information.

6. The data diagnostic method according to claim 5, characterized in that, The step of generating the tracing information based on the target data table and the upstream data table includes: Obtain the table structure change record information and processing logic change record information of the target data table and the upstream data table within a specified time period. The specified time period is the time period corresponding to the specified time window before the first data output timestamp. The tracing information is generated based on the table structure change record information and the processing logic change record information.

7. The data diagnostic method according to any one of claims 1 to 6, characterized in that, The step of searching the case database for target tracing information that meets similar conditions to the tracing information, and obtaining the target diagnosis result and target adjustment plan corresponding to the target tracing information, includes: Extract the feature vector of the source information, and in the case database, find the target source information that meets the specified vector similarity condition with the source information based on the feature vector, and obtain the target diagnosis result and target adjustment plan corresponding to the target source information.

8. The data diagnostic method according to any one of claims 1 to 6, characterized in that, The step of inputting the source tracing information, the target source tracing information, the target diagnosis results, and the target adjustment plan into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnosis results and adjustment plans includes: The source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme are inserted into the prompt word template to obtain model prompt words. The model prompt words are used to instruct the generative language model to diagnose the abnormal causes of the target data table based on the source tracing information, the target source tracing information, the target diagnosis result, and the target adjustment scheme. The model prompts are input into a generative language model to obtain diagnostic results and adjustment plans.

9. A data diagnostic device, characterized in that, include: The acquisition module is configured to acquire a target data table and determine the current status statistics of the target data table. In response to the current status statistics meeting the abnormal conditions, it generates source information for anomaly analysis of the target data table based on the target data table and the upstream data table of the target data table. The search module is configured to search the case database for target traceability information that meets similar conditions to the traceability information, and obtain the target diagnosis result and target adjustment plan corresponding to the target traceability information; The generation module is configured to input the source tracing information, the target source tracing information, the target diagnosis results, and the target adjustment scheme into a generative language model to diagnose the causes of anomalies in the target data table and obtain diagnosis results and adjustment schemes.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 8.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 8.

Images