A method, system, device and storage medium for generating data quality analysis report based on large model

Abstract

The present application discloses a method, system, device and storage medium for generating a data quality analysis report based on a large model. The generated quality report has in-depth analysis and standardized suggestions, so that the report content is relatively rich, thereby being able to meet complex and changing business needs. The method of the present application includes: obtaining input data; processing missing values, duplicate data and outliers on the input data to obtain preprocessed data; classifying the preprocessed data to obtain log type data and original data; integrating the target knowledge base RAG into a vertical large model, and fine-tuning the vertical large model to obtain a vertical fine-tuning large model; inputting the log type data and the original data into the vertical fine-tuning large model; obtaining the verification rules generated by the vertical fine-tuning large model; after determining that the verification rules meet the requirements, generating a high-quality analysis report according to the verification rules.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, device and storage medium for generating a data quality analysis report based on a large model. Background Art

[0002] In the digital age, data in all walks of life have shown a step-by-step growth. The surge in data volume has brought about the 4V characteristics of big data: large volume, fast speed, many types and value density. In the face of the low value density of data, a data governance solution has been introduced.

[0003] Data governance operates throughout the data life cycle, mainly addressing issues such as data quality, security, and standards, thereby increasing data value. Among them, data quality is the most important in data governance. Data quality is often set as a rule based on attribute dimensions, and data is verified according to the rule settings. Finally, a data quality report is generated for developers to audit data. However, traditional data quality management solutions often rely on hard-coded attribute dimensions or custom rules, and the generated quality reports lack in-depth analysis and standardized recommendations, resulting in a single report content that is difficult to meet complex and changing business needs. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a method, system, device and storage medium for generating data quality analysis report based on a large model.

[0005] The technical solution provided in this application is described below:

[0006] The first aspect of the present application provides a method for generating a data quality analysis report based on a large model, the method comprising:

[0007] Get input data;

[0008] Process the input data for missing values, duplicate data and outliers to obtain preprocessed data;

[0009] Classifying the preprocessed data to obtain log type data and original data;

[0010] Integrating the target knowledge base RAG into the vertical large model, and fine-tuning the vertical large model to obtain a vertical fine-tuned large model;

[0011] Inputting the log type data and the raw data into the vertical fine-tuning macro model;

[0012] Obtain verification rules generated by the vertical fine-tuning large model;

[0013] After determining that the verification rule meets the rule requirements, a high-quality analysis report is generated according to the verification rule.

[0014] Optionally, before integrating the trained target knowledge base RAG into the vertical large model and fine-tuning the vertical large model to obtain the vertical fine-tuned large model, the method includes:

[0015] Build a graph-enhanced knowledge base;

[0016] Integrating the graph-enhanced knowledge base into a generation model;

[0017] Associating historical data with nodes in the graph-enhanced knowledge base;

[0018] Based on the rule knowledge of the graph-enhanced knowledge base, define a verification rule template;

[0019] The verification rule template is input into the generation model to obtain the target knowledge base RAG.

[0020] Optionally, integrating the target knowledge base RAG into the vertical large model, and fine-tuning the vertical large model to obtain the vertical fine-tuned large model, including:

[0021] Building large vertical models;

[0022] Integrating the target knowledge base RAG into the vertical macro model;

[0023] The vertical large model is fine-tuned by using the LORA method to obtain a vertical fine-tuned large model.

[0024] Optionally, the vertical large model is fine-tuned by using a LORA method to obtain a vertical fine-tuned large model, including:

[0025] By analyzing the architecture of the vertical large model, a target adaptation layer is determined, where the target adaptation layer is a layer that needs to be adapted;

[0026] The original weight of the vertical large model is represented by the expression:

[0027] W′=W+ΔW;

[0028] Where W represents the original weight matrix, ΔW = A × B, A and B are low-rank matrices, and W′ is the modified target weight matrix;

[0029] Acquire a target training data set, wherein the target training data set includes an input text and a target output text;

[0030] Define training parameters;

[0031] In the forward propagation process, the vertical large model is predicted using the forward propagation formula and the training parameters, and the forward propagation formula is:

[0032] a=f(W′*x+b)

[0033] Where W′ is the target weight matrix, x is the input data, b is the bias term, f(W′*x+b) is the activation function, and a is the output result predicted by the vertical large model;

[0034] Calculate the loss based on the difference between the output result predicted by the vertical large model and the target output result;

[0035] When it is determined that the loss result meets the preset requirements, the adaptation parameters of the vertical large model are saved;

[0036] The vertical large model is fine-tuned according to the adaptation parameters to obtain a vertical fine-tuned large model.

[0037] Optionally, performing loss calculation based on the difference between the output result predicted by the vertical large model and the target output result includes:

[0038] Obtaining the output result and target output result predicted by the vertical large model;

[0039] Calculate the loss result of the output result predicted by the vertical large model and the target output result according to the loss calculation formula;

[0040] The loss calculation formula is:

[0041]

[0042] Among them, a is the output result predicted by the vertical large model, y is the target data result, N is the number of samples, and L (a, y) is the loss result;

[0043] When it is determined that the loss result does not meet the preset requirements, the low-rank matrix is ​​adjusted through the back-propagation algorithm to obtain the target low-rank matrix;

[0044] The back-propagation algorithm:

[0045] δ=L(a,y)*f′(z), where δ is the activation gradient, L(a,y) is the loss result, and f′ is the derivative of the activation function;

[0046] A′=A−η ∂L / ∂A;

[0047] B′=B−η∂L / ∂B;

[0048] Among them, ∂L / ∂A and ∂L / ∂B are the gradients calculated by back propagation, and A′ and B′ are the adjusted low-rank matrices;

[0049] Obtaining a target low-rank matrix, and fine-tuning the vertical large model according to the target low-rank matrix to obtain a vertical fine-tuned large model;

[0050] Evaluate the vertically fine-tuned large model using a validation set in the target training dataset;

[0051] When it is determined that the vertical fine-tuning large model meets the preset requirements, the vertical fine-tuning large model is saved.

[0052] Optionally, after obtaining the verification rules generated by the vertical fine-tuning large model, the method further includes:

[0053] Determine whether the verification rule meets the requirements;

[0054] After determining whether the verification rule meets the requirement or not, the step of obtaining the verification rule generated by the vertical fine-tuning large model is repeated.

[0055] Optionally, after determining that the verification rule meets the rule requirements, generating a high-quality analysis report according to the verification rule includes:

[0056] After determining that the verification rule meets the rule requirements, the verification rule is processed by using a large model SQL method or coding rules in a Java method to obtain a target verification rule;

[0057] Generate a high-quality analysis report based on the target verification rules.

[0058] The second aspect of the present application provides a data quality analysis report generation system based on a large model, characterized in that the system includes:

[0059] A first acquisition unit, used for acquiring input data;

[0060] A processing unit, used for processing missing values, duplicate data and abnormal values ​​of input data to obtain preprocessed data;

[0061] A classification unit, used for classifying the preprocessed data to obtain log type data and original data;

[0062] A second acquisition unit is used to integrate the target knowledge base RAG into the vertical large model, and fine-tune the vertical large model to obtain a vertical fine-tuned large model;

[0063] An input unit, used for inputting the log type data and the original data into the vertical fine-tuning large model;

[0064] A third acquisition unit is used to acquire the verification rules generated by the vertical fine-tuning large model;

[0065] A generating unit is used to generate a high-quality analysis report according to the verification rule after determining that the verification rule meets the rule requirements.

[0066] The third aspect of the present application provides a device for generating a data quality analysis report based on a large model, the device comprising:

[0067] Processor, memory, input-output unit, and bus;

[0068] The processor is connected to the memory, the input and output unit, and the bus;

[0069] The memory stores a program, and the processor calls the program to execute the method as described in the first aspect and any one of the first aspects.

[0070] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method described in the first aspect and any one of the first aspects is performed.

[0071] It can be seen from the above technical solutions that this application has the following advantages:

[0072] 1. By integrating the target knowledge base RAG and fine-tuning the vertical large model, the model can better adapt to the data analysis needs of specific fields or business scenarios. The customized model can more accurately capture data features and improve the depth and breadth of analysis.

[0073] 2. The vertical fine-tuning large model can automatically generate verification rules based on the input data, which greatly reduces the time and effort of manual rule-making, while improving the objectivity and scientificity of the rules.

[0074] 3. After determining that the verification rules meet the requirements, high-quality analysis reports can be automatically generated according to these rules. This automated process not only improves the efficiency of report generation, but also reduces errors and deviations caused by human factors. In addition, due to the use of fine-tuned vertical large models and scientific verification rules, the generated analysis reports have been significantly improved in accuracy and credibility, which helps decision makers to more accurately grasp the data quality status and make more informed decisions.

[0075] 4. Fine-tune the vertical large model according to actual needs so that the model can flexibly adapt to the data characteristics of different fields or business scenarios. This flexibility ensures the applicability and effectiveness of the method in different environments. By integrating the target knowledge base RAG, the method can continuously absorb and integrate new knowledge and experience, thereby continuously improving analytical capabilities and accuracy.

[0076] 5. The generated quality report has in-depth analysis and standardized suggestions, making the report content richer, thus meeting complex and changing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] Figure 1 A schematic diagram of an embodiment of a method for generating a data quality analysis report based on a large model in this application;

[0080] Figure 2 This is a schematic diagram of another embodiment of the method for generating a data quality analysis report based on a large model of the present application;

[0081] Figure 3 This is a schematic diagram of another embodiment of the method for generating a data quality analysis report based on a large model of the present application;

[0082] Figure 4 This is a schematic diagram of another embodiment of the method for generating a data quality analysis report based on a large model of the present application;

[0083] Figure 5 This is a schematic diagram of another embodiment of the method for generating a data quality analysis report based on a large model of the present application;

[0084] Figure 6 This is a schematic diagram of another embodiment of the method for generating a data quality analysis report based on a large model of the present application;

[0085] Figure 7 A schematic diagram of an embodiment of a data quality analysis report generation system based on a large model of the present application;

[0086] Figure 8 This is a schematic diagram of an embodiment of a device for generating a data quality analysis report based on a large model of the present application. DETAILED DESCRIPTION

[0087] It should be noted that the data quality analysis report generation method based on a large model provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for explanation.

[0088] See also Figure 1 The present application first provides an embodiment of a method for generating a data quality analysis report based on a large model, the embodiment comprising:

[0089] 101. Get input data;

[0090] 102. Process the input data for missing values, duplicate data and outliers to obtain preprocessed data;

[0091] 103. Classify the preprocessed data to obtain log type data and original data;

[0092] 104. Integrate the target knowledge base RAG into the vertical large model, and fine-tune the vertical large model to obtain a vertical fine-tuned large model;

[0093] 105. Input the log type data and the original data into the vertical fine-tuning large model;

[0094] 106. Obtain verification rules generated by the vertical fine-tuning large model;

[0095] 107. After determining that the verification rule meets the rule requirements, generate a high-quality analysis report according to the verification rule.

[0096] In the present application, input data is first obtained, and then missing values, duplicate data and outliers are processed on the input data to obtain preprocessed data, and then the obtained preprocessed data is classified to obtain log type data and original data; further, the target knowledge base RAG is integrated into the vertical large model, and the vertical large model is fine-tuned to obtain a vertical fine-tuned large model, and the log type data and the original data are input into the vertical fine-tuned large model, and the generated verification rules are obtained through the vertical fine-tuned large model;

[0097] After making a judgment on the verification rule and determining that the verification rule meets the rule requirements, a high-quality analysis report is generated according to the verification rule.

[0098] In step 101, data needs to be collected and acquired first. The data can come from various sources, such as databases, file systems, API interfaces, etc. The format of the input data can be structured, semi-structured or unstructured data. The input data can ensure the integrity and accuracy of the data. After the input data is acquired, step 102 is executed.

[0099] In step 102, the input data is processed for missing values, duplicate data and outliers to obtain preprocessed data. Specifically, the purpose of preprocessing the input data is to clean and process the input data to eliminate potential problems. Interpolation, mean filling or deletion of records containing missing values ​​can be used, and duplicate rows can be identified and removed by comparing specific fields of the records. Finally, statistical methods (such as Z-score, IQR) are used to identify outliers, and they are selected for removal or correction according to the situation. After the above processing, preprocessed data can be obtained.

[0100] In step 103, the preprocessed data is further classified to obtain log type data and raw data, so as to better analyze and generate reports based on the log type data and the raw data. Dividing the data into log type data and raw data helps to make subsequent analysis more targeted.

[0101] Among them, log type data usually contains timestamp, event, status and other information, which are used for system monitoring and auditing. The original data is usually core business data, including user information, transaction records, etc. Through the classification capability of the big model, the pre-processed data can be automatically classified and processed. After the pre-processed data is classified and processed, execute step 104.

[0102] In step 104, the target knowledge base RAG is integrated into the vertical big model, and the vertical big model is fine-tuned to obtain the vertical fine-tuned big model. Specifically, the knowledge base RAG is introduced and integrated into the vertical big model, and fine-tuned to obtain the vertical fine-tuned big model, so that the vertical fine-tuned big model can be optimized for data in a specific field.

[0103] Among them, the target knowledge base RAG contains domain knowledge, rules, patterns, etc. RAG is integrated into the vertical large model. After the integration is completed, the model is fine-tuned using domain-specific data sets through model fusion and feature injection to improve the performance of the model on specific tasks. The fine-tuning process of the vertical large model is explained in subsequent embodiments and is not specifically limited here.

[0104] In step 105, after obtaining the vertical fine-tuning large model, the log type data and the original data are input into the vertical fine-tuning large model. In this stage, the data input format of the log type data and the original data needs to match the requirements of the vertical fine-tuning large model. After the data is input, the vertical fine-tuning large model is processed according to the log type data and the original data.

[0105] In step 106, further, the verification rules generated by the vertical fine-tuning large model are obtained. Specifically, according to the input log type data and original data, the vertical fine-tuning large model will generate a series of verification rules, which are used to judge the quality and integrity of the data. Among them, the verification rules can include data format check, range check, and logical relationship check. The generated verification rules can fully reflect the characteristics of the data and business requirements.

[0106] In step 107, after determining that the verification rules meet the requirements, a high-quality analysis report is generated according to the verification rules; specifically, a data quality analysis report will be generated while ensuring that the rules meet the requirements. The analysis report should include various indicators of data quality, such as the proportion of missing values, the proportion of repeated data, the number of outliers, etc., and be able to provide detailed data quality analysis results and improvement suggestions to help relevant personnel make decisions. Finally, the generated high-quality analysis report will be visualized and the analysis results will be displayed to improve the readability and practicality of the analysis report.

[0107] The vertical fine-tuning large model can automatically generate verification rules based on the input data, which greatly reduces the time and effort of manual rule-making, while improving the objectivity and scientificity of the rules. The final quality report has in-depth analysis and standardized suggestions, making the report content richer, thus meeting complex and changing business needs.

[0108] For code examples of the processing process of the data quality analysis report generation method based on a large model, please refer to the following:

[0109] import pandas as pd

[0110] def load_data(source: str) ->pd.DataFrame:

[0111] # Assume the source is a CSV file

[0112] return pd.read_csv(source)

[0113] def preprocess_data(df: pd.DataFrame) ->pd.DataFrame:

[0114] # Handling missing values

[0115] df.fillna(method='ffill', inplace=True)# forward filling

[0116] # Handling duplicate data

[0117] df.drop_duplicates(inplace=True)

[0118] # Handling outliers (assuming Z-score as an example)

[0119] from scipy import stats

[0120] df = df[(np.abs(stats.zscore(df.select_dtypes(include=[np.number])))<3).all(axis=1)]

[0121] return df

[0122] def classify_data(df: pd.DataFrame) ->(pd.DataFrame, pd.DataFrame):

[0123] # Assume that the log data contains a timestamp field

[0124] log_data = df[df['type'] == 'log']

[0125] raw_data = df[df['type'] != 'log']

[0126] return log_data, raw_data

[0127] # This step usually involves a deep learning framework such as TensorFlow or PyTorch.

[0128] # Pseudocode, the specific implementation depends on the model and data.

[0129] class VerticalModel:

[0130] def __init__(self, rag_knowledge_base):

[0131] self.model = self.initialize_model()

[0132] self.rag_knowledge_base = rag_knowledge_base

[0133] def initialize_model(self):

[0134] # Initialize model logic

[0135] pass

[0136] def fine_tune(self, preprocessed_data):

[0137] # Fine-tune model logic

[0138] pass

[0139] # Assume we have a RAG knowledge base

[0140] rag_knowledge_base = "path_to_rag_knowledge_base"

[0141] model = VerticalModel(rag_knowledge_base)

[0142] model.fine_tune(preprocessed_data)

[0143] def input_data_to_model(model, log_data, raw_data):

[0144] # Input data to the model

[0145] model_output_log = model.predict(log_data)

[0146] model_output_raw = model.predict(raw_data)

[0147] return model_output_log, model_output_raw

[0148] def extract_validation_rules(model_output) ->list:

[0149] # Extract verification rules, pseudo code

[0150] validation_rules = []

[0151] for output in model_output:

[0152] rules = output['validation_rules'] # Assume the output contains rules

[0153] validation_rules.extend(rules)

[0154] return validation_rules

[0155] def generate_quality_report(validation_rules: list) ->str:

[0156] report = "Data Quality Analysis Report\n\n"

[0157] for rule in validation_rules:

[0158] report += f"Rule: {rule}\n"# Build report content according to validation rules

[0159] return report

[0160] # Putting the whole process together

[0161] if __name__ == "__main__":

[0162] data = load_data('data.csv')

[0163] preprocessed_data = preprocess_data(data)

[0164] log_data, raw_data = classify_data(preprocessed_data)

[0165] model = VerticalModel(rag_knowledge_base)

[0166] model.fine_tune(preprocessed_data)

[0167] model_output_log, model_output_raw = input_data_to_model(model, log_data, raw_data)

[0168] validation_rules = extract_validation_rules(model_output_log)

[0169] report = generate_quality_report(validation_rules)

[0170] print(report)

[0171] This code example involves the process of data transmission. Specifically, first, the input data is obtained, and then the input data is processed for missing values, duplicate data, and outliers to obtain pre-processed data. The pre-processed data is then classified to obtain log type data and original data. Further, the target knowledge base RAG is integrated into the vertical large model, and the vertical large model is fine-tuned to obtain the vertical fine-tuned large model, and the log type data and original data are input into the vertical fine-tuned large model, and the generated verification rules are obtained through the vertical fine-tuned large model; after making a judgment on the verification rules and determining that the verification rules meet the requirements, a high-quality analysis report is generated according to the verification rules.

[0172] In this application, by integrating the target knowledge base RAG and fine-tuning the vertical large model, the model can better adapt to the data analysis needs of specific fields or business scenarios. The customized model can more accurately capture data features and improve the depth and breadth of analysis. The generated quality report has in-depth analysis and standardized suggestions, making the report content richer, thus meeting complex and changing business needs.

[0173] Please refer to Figure 2 According to some embodiments of the present invention, before integrating the target knowledge base RAG into the vertical large model and fine-tuning the vertical large model in step 104 to obtain the vertical fine-tuned large model, the steps may specifically include, but are not limited to, the following:

[0174] 201. Constructing a graph-enhanced knowledge base;

[0175] 202. Integrate the graph-enhanced knowledge base into a generation model;

[0176] 203. Associating historical data with nodes in the graph-enhanced knowledge base;

[0177] 204. Based on the rule knowledge of the graph-enhanced knowledge base, define a verification rule template;

[0178] 205. Input the verification rule template into the generation model to obtain a target knowledge base RAG.

[0179] In an embodiment of the present application, a graph-enhanced knowledge base is first constructed, wherein the graph-enhanced knowledge base is a knowledge representation and management method based on a knowledge graph, which combines the nodes, edges and attributes of the knowledge graph to represent entities, relationships and attributes, and enriches the content of the knowledge base through enhanced methods such as adding rules, patterns, constraints, etc. The graph-enhanced knowledge base can provide powerful knowledge support for subsequent data quality analysis.

[0180] After building the graph-enhanced knowledge base, the graph-enhanced knowledge base is integrated into the generative model. Specifically, integrating the graph-enhanced knowledge base into the generative model can enable the model to utilize the rich knowledge in the knowledge base when generating data or verifying rules, thereby improving the accuracy and reliability of the generated results. By selecting a suitable generative model according to task requirements, such as GPT, BERT, T5, etc., the knowledge in the graph-enhanced knowledge base is encoded into a format that the model can understand, such as embedded vectors, graph structures, etc.

[0181] The encoded knowledge is integrated into the generative model using model fusion, feature injection, and attention mechanism methods, and the integrated generative model is trained and adjusted to ensure that it can effectively utilize the knowledge in the knowledge base.

[0182] The historical data is then associated with the nodes in the knowledge base in order to use the knowledge in the knowledge base to verify and correct the historical data in subsequent data quality analysis. Specifically, the historical data is preprocessed by cleaning, deduplication, formatting and other preprocessing operations, and then entity recognition technology is further used to identify entities from the historical data, match the identified entities with the nodes in the knowledge base, find the corresponding nodes, and after a successful match, establish an association relationship between the historical data and the nodes in the knowledge base.

[0183] Furthermore, by utilizing the rule knowledge in the graph-enhanced knowledge base, rule templates for data quality verification can be defined. The rule templates can automatically check the consistency, completeness, accuracy and other quality dimensions of the data.

[0184] Specifically, first extract the rule knowledge related to data quality from the knowledge base, and then design the verification rule template based on the rule knowledge. The template should include elements such as rule description, verification logic, threshold, etc. Then use historical data to verify the template to ensure its effectiveness and accuracy.

[0185] After obtaining the verification rule template, the verification rule template is input into the generation model. Through the reasoning and generation capabilities of the model, a more refined and personalized target knowledge base RAG can be obtained. The target knowledge base RAG not only contains the original rule knowledge, but also integrates the intelligent reasoning and generation capabilities of the model.

[0186] Therefore, by constructing a graph-enhanced knowledge base and integrating it into the generative model, and then using historical data and rule knowledge in the knowledge base to define the verification rule template, and finally obtaining the target knowledge base RAG, this process fully utilizes the advantages of knowledge graphs and generative models, providing strong support for data quality analysis.

[0187] Please refer to Figure 3 According to some embodiments of the present invention, in step 104, integrating the target knowledge base RAG into the vertical large model and fine-tuning the vertical large model to obtain the vertical fine-tuned large model may specifically include, but is not limited to, the following:

[0188] 301. Build a vertical large model;

[0189] 302. Integrate the target knowledge base RAG into the vertical large model;

[0190] 303. Fine-tune the vertical large model to obtain a vertical fine-tuned large model.

[0191] In the embodiment of the present application, the vertical large model has powerful feature extraction and pattern recognition capabilities, and can handle complex tasks such as data classification, prediction, generation, etc. In the present application, a suitable vertical large model is selected according to the task requirements, and then the target knowledge base RAG is integrated into the vertical large model. The vertical large model can utilize the rich knowledge and rules in the knowledge base when processing tasks, which can improve the accuracy and reliability of the processing results.

[0192] Specifically, the target knowledge base RAG is integrated into the vertical big model. First, the knowledge in the target knowledge base RAG, such as rules, patterns, constraints, etc., is encoded into a format that can be understood by the vertical big model, such as embedded vectors, graph structures, etc.

[0193] The encoded knowledge is then fused with the vertical large model, including adding additional feature vectors to the input layer of the model or introducing a knowledge attention mechanism in the hidden layer of the model. Furthermore, the encoded knowledge is injected as features into the input layer or hidden layer of the vertical large model, so that the vertical large model can take this knowledge into account during reasoning and decision-making.

[0194] After the knowledge base is integrated into the model, the model is jointly trained to ensure that the integrated vertical large model can effectively utilize the knowledge in the knowledge base. After integration, the vertical large model is fine-tuned. The fine-tuning is intended to make the vertical large model better adaptable to specific tasks or data sets. After the target knowledge base RAG is integrated into the vertical large model, the model is fine-tuned to further improve the performance of the model on specific tasks. The specific fine-tuning process is described in detail in the following embodiments.

[0195] Please refer to Figure 4 According to some embodiments of the present invention, fine-tuning the vertical large model in step 303 to obtain the vertical fine-tuned large model may specifically include, but is not limited to, the following:

[0196] 401. Analyze the architecture of the vertical large model to determine a target adaptation layer, where the target adaptation layer is a layer that needs to be adapted;

[0197] 402. The original weight of the vertical large model is represented by an expression, wherein the expression is:

[0198] W′=W+ΔW;

[0199] Where W represents the original weight matrix, ΔW = A × B, A and B are low-rank matrices, and W′ is the modified target weight matrix;

[0200] 403. Obtain a target training data set, where the target training data set includes an input text and a target output text;

[0201] 404. Define training parameters;

[0202] 405. In the forward propagation process, the vertical large model is predicted using the forward propagation formula and the training parameters, wherein the forward propagation formula is:

[0203] a=f(W′*x+b)

[0204] Where W′ is the target weight matrix, x is the input data, b is the bias term, f(W′*x+b) is the activation function, and a is the output result predicted by the vertical large model;

[0205] 406. Calculate the loss according to the difference between the output result predicted by the vertical large model and the target output result;

[0206] 407. When it is determined that the loss result meets the preset requirements, the adaptation parameters of the vertical large model are saved;

[0207] 408. Fine-tune the vertical large model according to the adaptation parameters to obtain a vertical fine-tuned large model.

[0208] In an embodiment of the present application, in the vertical large model, different layers are responsible for extracting different features and information. In order to perform effective adaptation, the architecture of the vertical large model must be analyzed to determine which layers are the target adaptation layers that need to be adapted. The target adaptation layer is the key layer in the model, and this key layer has a greater impact on the output of the vertical large model.

[0209] Specifically, we first review the architecture of the vertical large model to understand the functions of each layer. Then, based on the task requirements and the model performance of the vertical large model, we identify the key layers that have a greater impact on the output of the vertical large model. These layers will then be identified as target adaptation layers.

[0210] In order to adapt the vertical large model, the weights of the vertical large model need to be further modified. The original weight matrix can be represented and modified by low-rank matrix decomposition to reduce the complexity of calculation and improve the flexibility of adaptation.

[0211] Original weight representation: W is used to represent the original weight matrix of the target adaptation layer in the vertical large model.

[0212] Low-rank matrix decomposition: The modification ΔW of the weight matrix is ​​expressed as the product of two low-rank matrices A and B, that is, ΔW = A × B, which can reduce the number of parameters that need to be learned and improve adaptation efficiency.

[0213] Target weight calculation: Based on the original weight W and the modification ΔW, calculate the modified target weight matrix W′ = W + ΔW.

[0214] After the target weights are adapted, in order to perform adaptation training, a training data set related to the target task is obtained. The training data set contains input text and target output text so that the model can learn how to map input text to target output text.

[0215] Before model training, it is necessary to further define training parameters, such as learning rate, optimizer, batch size, number of training rounds, etc., which are not specifically limited here. These training parameters can control the process and effect of vertical large model training.

[0216] After the above steps are completed, the vertical large model is trained. In the forward propagation process of the vertical large model training, the vertical large model is predicted using the defined forward propagation formula and training parameters. During the entire training process, the input data will be passed through the layers of the vertical large model to finally obtain the output result of the model.

[0217] Specifically, the input text is converted into a format that the model can understand, such as word vectors, sentence vectors, etc., and then the forward propagation formula a = f(W′* x + b) is used to calculate the output of each layer, where W′ is the target weight matrix, x is the input data, b is the bias term, and f is the activation function.

[0218] The predicted output result of the vertical large model is obtained through forward propagation calculation, and then the loss is calculated based on the difference between the output result predicted by the vertical large model and the target output result. The calculated loss will be used to guide the update of the model parameters.

[0219] During the training of the vertical large model, the changes in the loss value must be continuously monitored. When the loss value meets the preset requirements, it means that the model has achieved good performance. At this time, the adaptation parameters of the vertical large model can be saved, including the modified weight matrix W′ and other related parameters.

[0220] Finally, the vertical large model is fine-tuned using the saved adaptation parameters to obtain a vertical fine-tuned large model optimized for a specific task. The fine-tuned vertical large model can further improve the performance of the model on the target task.

[0221] Figure 5 According to some embodiments of the present invention, the loss calculation based on the difference between the output result predicted by the vertical large model and the target output result in step 406 may specifically include, but is not limited to, the following:

[0222] 501. Obtain the output result and target output result predicted by the vertical large model;

[0223] 502. Calculate the loss result of the output result predicted by the vertical large model and the target output result according to the loss calculation formula;

[0224] The loss calculation formula is:

[0225]

[0226] Among them, a is the output result predicted by the vertical large model, y is the target data result, N is the number of samples, and L (a, y) is the loss result;

[0227] 503. When it is determined that the loss result does not meet the preset requirements, the low-rank matrix is ​​adjusted by a back-propagation algorithm to obtain a target low-rank matrix;

[0228] The back-propagation algorithm:

[0229] δ=L(a,y)*f ′ (z), where δ is the activation gradient, L(a,y) is the loss result, and f′ is the derivative of the activation function;

[0230] A′=A−η ∂L / ∂A;

[0231] B′=B−η∂L / ∂B;

[0232] Among them, ∂L / ∂A and ∂L / ∂B are the gradients calculated by back propagation, and A′ and B′ are the adjusted low-rank matrices;

[0233] 504. Obtain a target low-rank matrix, and fine-tune the vertical large model according to the target low-rank matrix to obtain a vertical fine-tuned large model;

[0234] 505. Evaluate the vertical fine-tuning large model using the validation set in the target training data set;

[0235] 506. When it is determined that the vertical fine-tuning large model meets the preset requirements, the vertical fine-tuning large model is saved.

[0236] In the embodiment of the present application, firstly obtain the output result and target output result of the vertical large model prediction

[0237] The output result predicted by the vertical large model refers to the output result of inputting the input data into the trained vertical large model, and the model predicts based on the learned parameters and features. The target output result refers to the real label or expected output result corresponding to the input data. In training and learning, the target result is used to train the model, and the loss is calculated by comparing the model prediction result and the target result, thereby guiding the model parameter update.

[0238] After obtaining the output results predicted by the vertical large model and the target output results, the loss results are calculated according to the loss calculation formula. The loss calculation formula is:

[0239] , where a represents the output predicted by the model, y represents the target output, N is the number of samples, and L(a, y) is the loss result.

[0240] Then, the output results predicted by the model and the target output results are substituted into the loss calculation formula to calculate the loss value of the current data set. The calculated loss value reflects the degree of difference between the prediction results of the vertical large model and the target results.

[0241] When it is determined that the calculated loss result does not meet the preset requirements, the loss function is minimized by calculating the gradient of the loss function with respect to the model parameters and updating the parameters in the opposite direction of the gradient.

[0242] First, activate the gradient: δ=L(a,y)*f′(z), where f′(z) is the derivative of the activation function. The activation gradient is used to calculate the gradient of the loss function with respect to the internal parameters of the model, which is one of the key steps of the back propagation algorithm.

[0243] Then the parameters are updated: A′=A−η∂L / ∂A and B′=B−η∂L / ∂B, where ∂L / ∂A and η∂L / ∂B are the gradients calculated by back propagation, and A′ and B′ are the adjusted low-rank matrices. η is the learning rate, which controls the step size of the parameter update.

[0244] The low-rank matrix adjusted by the back-propagation algorithm contains a more effective and compact feature representation learned by the model. The target low-rank matrix is ​​then used to fine-tune the vertical large model, that is, the parameters of the model are adjusted according to the new low-rank matrix to further improve the performance of the model.

[0245] The validation set is then used to evaluate the performance of the fine-tuned vertical large model, including indicators such as accuracy, recall, and F1 score. The indicators in the validation set are used to measure the generalization ability of the vertical fine-tuned large model on unseen data.

[0246] Finally, when it is determined that the fine-tuned vertical large model meets the preset requirements, the model is saved and used for subsequent prediction tasks or further research.

[0247] Figure 6 According to some embodiments of the present invention, in step 107, after determining that the verification rule meets the rule requirements, generating a high-quality analysis report according to the verification rule may further include, but not limited to, the following:

[0248] 601. After determining that the verification rule meets the rule requirements, the verification rule is processed by using a large model SQL method or coding rules in a Java method to obtain a target verification rule;

[0249] 602. Generate a high-quality analysis report according to the target verification rule.

[0250] In an embodiment of the present application, after determining that the verification rules meet the requirements, the verification rules are processed using the large model SQL method or the Java method to encode the rules to obtain the target verification rules. Specifically, a set of verification rules has been generated in the aforementioned steps, and these rules have been preliminarily verified to confirm that they meet the requirements of the business logic and data specifications. Then the verification rules are actually implemented and compiled into an implementable method. Then, according to the target verification rules, the data to be analyzed is prepared, including the original data extracted from the database, the preprocessed data set, or the real-time data obtained through the API. Then the target verification rules are applied to the prepared data, and the results obtained after applying the rules are analyzed in depth, including the distribution of data, the detection of outliers, the prediction of trends, etc. At the same time, the results need to be interpreted and evaluated according to the requirements of the business logic and data specifications.

[0251] Finally, based on the analysis results, a high-quality analysis report is generated, which includes clear charts, detailed explanations, accurate conclusions, and suggestions and solutions for problems. In addition, the report also has a clear logical structure and accurate language to ensure that the content of the report can be easily understood to support business decision-making and data management.

[0252] See also Figure 7 In a second aspect, the present application provides a data quality analysis report generation system based on a large model, the system comprising:

[0253] A first acquisition unit 701 is used to acquire input data;

[0254] The processing unit 702 is used to process missing values, duplicate data and abnormal values ​​of the input data to obtain pre-processed data;

[0255] A classification unit 703, used for classifying the pre-processed data to obtain log type data and original data;

[0256] The second acquisition unit 704 is used to integrate the target knowledge base RAG into the vertical large model, and fine-tune the vertical large model to obtain the vertical fine-tuned large model;

[0257] An input unit 705, used for inputting the log type data and the original data into the vertical fine-tuning large model;

[0258] The third acquisition unit 706 is used to acquire the verification rules generated by the vertical fine-tuning large model;

[0259] The generating unit 707 is used to generate a high-quality analysis report according to the verification rule after determining that the verification rule meets the rule requirements.

[0260] See also Figure 8 The present application also provides a data quality analysis report generation device based on a large model, comprising:

[0261] Processor 801, memory 802, input and output unit 803, bus 804;

[0262] The processor 801 is connected to the memory 802, the input and output unit 803 and the bus 804;

[0263] The memory 802 stores a program, and the processor 801 calls the program to execute any of the above methods.

[0264] The present application also relates to a computer-readable storage medium on which a program is stored, wherein when the program is run on a computer, the computer is caused to execute any of the above methods.

[0265] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0266] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0267] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0268] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0269] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for generating a data quality analysis report based on a large model, characterized in that: The method comprises: Get input data; Process missing values, duplicate data and outliers on the input data to obtain preprocessed data; Classifying the preprocessed data to obtain log type data and original data; Build a graph-enhanced knowledge base; Integrating the graph-enhanced knowledge base into a generation model; Associating historical data with nodes in the graph-enhanced knowledge base; Based on the rule knowledge of the graph-enhanced knowledge base, define a verification rule template; Inputting the verification rule template into the generation model to obtain a target knowledge base RAG; Construct a vertical large model, integrate the target knowledge base RAG into the vertical large model, analyze the architecture of the vertical large model to determine the target adaptation layer, the target adaptation layer is the layer that needs to be adapted, and express the original weight of the vertical large model by an expression: W′=W+ΔW, wherein W represents the original weight matrix, ΔW=A×B, A and B are low-rank matrices, and W′ is the modified target weight matrix. Obtain a target training set, wherein the target training set includes input text and target output text, define training parameters, and in the forward propagation process, use the forward propagation formula and the training parameters to predict the vertical large model. The forward propagation formula is: a=f(W ′*x+b), wherein W ′ is the target weight matrix, x is the input data, b is the bias term, and f(W ′*x+b) is an activation function, a is the output result predicted by the vertical large model, and the loss is calculated according to the difference between the output result predicted by the vertical large model and the target output result. When it is determined that the loss result meets the preset requirements, the adaptation parameters of the vertical large model are saved, and the vertical large model is fine-tuned according to the adaptation parameters to obtain a vertical fine-tuned large model; Inputting the log type data and the raw data into the vertical fine-tuning macro model; Obtain verification rules generated by the vertical fine-tuning large model; After determining that the verification rule meets the rule requirements, a high-quality analysis report is generated according to the verification rule.

2. The method for generating a data quality analysis report based on a large model according to claim 1, characterized in that: The loss calculation is performed based on the difference between the output result predicted by the vertical large model and the target output result, including: Obtaining the output result and target output result predicted by the vertical large model; Calculate the loss result of the output result predicted by the vertical large model and the target output result according to the loss calculation formula; The loss calculation formula is: Among them, a is the output result predicted by the vertical large model, y is the target data result, N is the number of samples, and L (a, y) is the loss result; When it is determined that the loss result does not meet the preset requirements, the low-rank matrix is ​​adjusted through the back-propagation algorithm to obtain the target low-rank matrix; The back-propagation algorithm: δ=L(a,y)*f ′ (z), where δ is the activation gradient, L(a,y) is the loss result, and f′ is the derivative of the activation function; A′=A−η ∂L / ∂A; B′=B−η∂L / ∂B; Among them, ∂L / ∂A and ∂L / ∂B are the gradients calculated by back propagation, and A′ and B′ are the adjusted low-rank matrices; Obtaining a target low-rank matrix, and fine-tuning the vertical large model according to the target low-rank matrix to obtain a vertical fine-tuned large model; Evaluate the vertically fine-tuned large model using a validation set in the target training dataset; When it is determined that the vertical fine-tuning large model meets the preset requirements, the vertical fine-tuning large model is saved.

3. The method for generating a data quality analysis report based on a large model according to claim 1, characterized in that: After obtaining the verification rules generated by the vertical fine-tuning large model, the method further includes: Determine whether the verification rule meets the requirements; After determining whether the verification rule meets the requirement or not, the step of obtaining the verification rule generated by the vertical fine-tuning large model is repeated.

4. The method for generating a data quality analysis report based on a large model according to claim 1, characterized in that: After determining that the verification rule meets the requirements, a high-quality analysis report is generated according to the verification rule, including: After determining that the verification rule meets the rule requirements, the verification rule is processed by using a large model SQL method or coding rules in a Java method to obtain a target verification rule; Generate a high-quality analysis report based on the target verification rules.

5. A data quality analysis report generation system based on a large model, characterized in that: The system comprises: A first acquisition unit, used for acquiring input data; A processing unit, used for processing missing values, duplicate data and abnormal values ​​of input data to obtain preprocessed data; A classification unit, used for classifying the preprocessed data to obtain log type data and original data; A construction unit, used to construct a graph-enhanced knowledge base; An integration unit, used for integrating the graph-enhanced knowledge base into a generation model; An association unit, used to associate historical data with nodes in the graph-enhanced knowledge base; A definition unit, which defines a verification rule template based on the rule knowledge of the graph-enhanced knowledge base; The first input unit is used to input the verification rule template into the generation model to obtain the target knowledge base RAG The second acquisition unit is used to build a vertical large model, integrate the target knowledge base RAG into the vertical large model, analyze the architecture of the vertical large model to determine the target adaptation layer, the target adaptation layer is the layer that needs to be adapted, and express the original weight of the vertical large model by an expression: W′=W+ΔW, wherein W represents the original weight matrix, ΔW=A×B, A and B are low-rank matrices, and W′ is the modified target weight matrix. The target training number set is obtained, and the target training number set includes input text and target output text. The training parameters are defined. In the forward propagation process, the forward propagation formula and the training parameters are used to predict the vertical large model. The forward propagation formula is: a=f(W′*x+b), wherein W ′ is the target weight matrix, x is the input data, b is the bias term, and f(W ′*x+b) is an activation function, a is the output result predicted by the vertical large model, and the loss is calculated according to the difference between the output result predicted by the vertical large model and the target output result. When it is determined that the loss result meets the preset requirements, the adaptation parameters of the vertical large model are saved, and the vertical large model is fine-tuned according to the adaptation parameters to obtain a vertical fine-tuned large model; A second input unit, used for inputting the log type data and the original data into the vertical fine-tuning large model; A third acquisition unit is used to acquire the verification rules generated by the vertical fine-tuning large model; A generating unit is used to generate a high-quality analysis report according to the verification rule after determining that the verification rule meets the rule requirements.

6. A data quality analysis report generation device based on a large model, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 4 is performed.

Images