Data management method and device, electronic equipment and storage medium
By using multiple data quality detection models for data correlation analysis and automated governance script generation, the problems of low efficiency and poor intelligence in manual review are solved, achieving efficient and intelligent data governance.
Patent Information
- Application Number
- CN202511911377.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-27
AI Technical Summary
In existing technologies, data quality detection and governance rely on manual review, which is inefficient and difficult to cope with complex and ever-changing data environments, and lacks intelligence.
By acquiring the data to be tested, quality testing is performed using at least two data quality testing models. Correlation analysis is conducted to determine the cause of the problem, and a governance script is generated based on the cause of the problem for automated governance.
It improves the efficiency and intelligence of data governance, enables automatic monitoring of data and timely detection of problems, accurately locates the root cause, and improves the accuracy of monitoring and governance.
Smart Images
Figure CN121743316A_ABST
Abstract
Description
Technical Field
[0001] This application relates to artificial intelligence technology, and more particularly to a data governance method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rapid development of the Internet and the Internet of Things, data has become an important asset and fundamental support for enterprises. However, due to the many stages of data generation, the diversity of data types, and the huge volume of data, data quality is difficult to guarantee. Therefore, it is necessary to detect and manage data in order to correct problems in a timely manner.
[0003] In existing technologies, data quality inspection and governance rely on manual review, rule setting, and statistical aggregation. However, manual review and governance are inefficient, difficult to cope with complex and ever-changing data environments, and lack intelligence. Summary of the Invention
[0004] This application provides a data governance method, apparatus, electronic device, and storage medium to improve the efficiency and intelligence of data governance.
[0005] In a first aspect, embodiments of this application provide a data governance method, which includes:
[0006] Acquire the data to be tested, and perform quality testing on the data to be tested using at least two data quality testing models to obtain the test results;
[0007] Based on the detection results, correlation analysis was performed to determine the cause of the problem;
[0008] Determine the solution based on the cause of the problem;
[0009] A governance script is generated based on the governance plan, and the data to be tested is then governed using the governance script.
[0010] Secondly, embodiments of this application also provide a data governance apparatus, which includes:
[0011] The detection result acquisition module is used to acquire the data to be detected and perform quality detection on the data to be detected using at least two data quality detection models to obtain the detection result.
[0012] The detection result analysis module is used to perform correlation analysis based on the detection results to determine the cause of the problem.
[0013] The governance solution determination module is used to determine the governance solution based on the cause of the problem;
[0014] The data to be tested governance module is used to generate governance scripts according to the governance plan, and to govern the data to be tested through the governance scripts.
[0015] Thirdly, embodiments of this application also provide an electronic device, which includes:
[0016] One or more processors;
[0017] Storage device for storing one or more programs;
[0018] When one or more programs are executed by one or more processors, the one or more processors implement any of the data governance methods provided in the embodiments of this application.
[0019] Fourthly, embodiments of this application also provide a storage medium including computer-executable instructions, which, when executed by a computer processor, are used to perform any of the data governance methods provided in embodiments of this application.
[0020] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the data governance methods provided in embodiments of this application.
[0021] This application acquires data to be tested and performs quality checks on the data using at least two data quality inspection models to obtain inspection results. This enables automatic monitoring and detection of the data, allowing for timely problem identification. The accuracy of the quality inspection models improves the accuracy of monitoring. Correlation analysis is performed based on the inspection results to identify the causes of problems, and joint analysis accurately pinpoints the root causes. Based on the causes, a remediation plan is determined, automatically identifying targeted remediation solutions. A remediation script is generated based on the remediation plan, and this script is used to remediate the data to be tested. By generating the remediation script, automatic data remediation is achieved, improving the efficiency and intelligence of data governance. Therefore, the technical solution of this application solves the problems of low efficiency in manual review and remediation, and the inability to cope with complex and ever-changing data environments, achieving the effect of improving the efficiency and intelligence of data governance. Attached Figure Description
[0022] Figure 1 This is a flowchart of a data governance method according to Embodiment 1 of this application;
[0023] Figure 2 This is a flowchart of a data governance method according to Embodiment 2 of this application;
[0024] Figure 3 This is a flowchart of a data governance method according to Embodiment 3 of this application;
[0025] Figure 4 This is a schematic diagram of the structure of a data governance device according to Embodiment 4 of this application;
[0026] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of this application. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0028] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0029] Example 1
[0030] Figure 1 This is a flowchart of a data governance method provided in Embodiment 1 of this application. This embodiment is applicable to the case of automatic data governance. The method can be executed by a data governance device, which can be implemented in software and / or hardware and specifically configured in a data processing platform.
[0031] See Figure 1 The data governance method shown includes the following steps:
[0032] S110. Obtain the data to be tested, and perform quality testing on the data to be tested using at least two data quality testing models to obtain the test results.
[0033] The data to be tested can be data requiring data inspection and data governance, and can be obtained from relevant data platforms. For example, the data to be tested can be data from an enterprise's business platform. The data quality inspection model can be a deep learning model used to monitor data quality and promptly identify data quality issues. At least two data quality inspection models are included to detect different data problems, thereby improving the accuracy of the detection. For example, data quality inspection models can include decision tree models, isolated forest models, and logistic regression models. The detection result can be the model output obtained by performing quality inspection on the data to be tested using at least two data quality inspection models. For example, the detection result can include two cases: no problem and problematic. When the detection result is problematic, it can also include specific problems. The data to be tested is sequentially input into at least two data quality inspection models, and the output is the detection result. For example, the detection result could be problematic or inconsistent data.
[0034] S120. Based on the detection results, perform correlation analysis to determine the cause of the problem.
[0035] Correlation analysis can be used to analyze related data to identify the root causes of problems and determine the reasons behind the detection results. When the detection results include multiple problems, correlation analysis can be performed based on these results to identify the root causes. For example, if the detection results are "problematic" or "inaccurate," and "problematic" or "inconsistent," correlation analysis can first determine if there is a correlation between the inaccurate and inconsistent data. If so, it can be determined whether the inaccuracy is caused by data inconsistency, thus identifying the root cause of the problem. These root causes can provide data support for determining remediation solutions.
[0036] Optionally, after identifying the causes of the problem, statistics on these causes can be compiled, and the results can be visualized using visualization technology. Visualization enhances the intuitiveness of the causes and provides data support for R&D personnel to improve their research and development. For example, bar charts, line charts, and heatmaps can be used to display the distribution, trends, and influencing factors of data quality problems, and generate analysis reports.
[0037] In one optional embodiment, correlation analysis is performed based on the detection results to obtain the cause of the problem, including: determining whether there is a correlation between the problematic data in the detection results; if so, correlation analysis is performed on the problematic data to be detected that has a correlation to obtain the cause of the problem.
[0038] Correlation can be based on shared sources or causal relationships. By examining the data sources and data flow paths, it's determined whether problematic data in the detection results are correlated. If so, meaning there's a correlation between the data, correlation analysis is performed on the related data. This can involve in-depth analysis of the causes of the problems to identify the root cause. For example, if data A and data B are combined to obtain data C, then data A and data C have a causal relationship. When both data A and data C have problems, correlation analysis can be performed on the problems of data A and data C. If data A causes problems in data C, the root cause can be identified promptly as a problem with data A. This allows for subsequent governance solutions determined based on the root cause, ensuring the accuracy of data governance.
[0039] By determining whether there is a correlation between the problematic data in the test results, and if so, a correlation analysis is performed on the problematic data to find the cause of the problem. This can help identify the root cause of the problem and improve the accuracy of subsequent governance.
[0040] S130. Determine the solution based on the cause of the problem.
[0041] A governance plan is a scheme for governing data based on the identified cause of a problem, used to determine the corresponding governance methods. A governance plan includes the processing method, the processing target, and the data storage location. For example, if the problem is duplicate data A, the governance plan may include the field name corresponding to data A, its storage location, and the processing method of deleting data A. Similarly, if the problem is missing data B, the governance plan may include the field name corresponding to data B, its storage location, and the processing method of adding data B. Optionally, when data is missing, numeric fields can be filled with the median, and categorical fields can be filled with the mode.
[0042] S140. Generate a governance script based on the governance plan, and use the governance script to govern the data to be tested.
[0043] Governance scripts can be automated and can be automatically generated using deep learning models. For example, a large model can be used to automatically write governance scripts based on a governance scheme. Running the governance script can perform governance on the data to be detected.
[0044] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.
[0045] In the field of big data, data quality is particularly important for business analysis and decision-making, directly impacting data value. Ensuring data quality requires regular data governance across various dimensions. Traditional data quality monitoring and governance rely on manual review, rule setting, and statistical summarization. Traditional manual data governance suffers from the following pain points:
[0046] 1) Inefficiency: Manually handling data governance tasks, such as data standard setting, metadata information improvement, and data quality management, requires a lot of time and manpower;
[0047] 2) Insufficient accuracy: During the data cleaning, classification and summarization process, manual operations are prone to omissions and errors;
[0048] 3) Difficulty in handling complex data: When faced with massive and complex data, especially unstructured data, human processing capabilities are limited.
[0049] The technical solution of this embodiment acquires the data to be tested and performs quality testing on the data using at least two data quality testing models to obtain test results. This enables automatic monitoring and testing of the data to be tested, allowing for timely detection of problems. The accuracy of the quality testing models can improve the accuracy of monitoring. Correlation analysis is performed based on the test results to identify the causes of problems, and joint analysis accurately locates the root causes. Based on the causes of problems, a governance plan is determined, automatically identifying targeted governance solutions. A governance script is generated based on the governance plan, and the data to be tested is governed using the governance script. By generating the governance script, automatic data governance can be achieved, improving the efficiency and intelligence of data governance. Therefore, the technical solution of this application solves the problems of low efficiency in manual review and governance, and poor intelligence in dealing with complex and ever-changing data environments, achieving the effect of improving the efficiency and intelligence of data governance.
[0050] Example 2
[0051] Figure 2 This is a flowchart of a data governance method provided in Embodiment 2 of this application. The technical solution of this embodiment is further refined based on the above technical solution.
[0052] Furthermore, the process of "generating a governance script based on the governance scheme and governing the data to be detected using the governance script" is further refined into: "determining the script type and automatically generating a governance script based on the governance scheme; governing the data to be detected using the governance script to obtain the data to be verified; verifying the data to be verified based on the governance scheme, and obtaining the governance result after passing the verification," in order to achieve automatic data governance.
[0053] See Figure 2 One data governance method shown includes:
[0054] S210. Obtain the data to be tested, and perform quality testing on the data to be tested using at least two data quality testing models to obtain the test results.
[0055] S220. Based on the detection results, perform correlation analysis to determine the cause of the problem.
[0056] S230. Determine the solution based on the cause of the problem.
[0057] S240. Based on the governance plan, determine the script type and automatically generate the governance script.
[0058] The script type can be the type of script used, determined according to the governance scheme. For example, script types can be divided into general scripts and big data processing scripts. General scripts are automated scripts suitable for processing small amounts of data. Big data processing scripts are scripts suitable for processing large volumes of data, relying on a big data processing engine for efficient data processing to ensure data processing efficiency. The governance scheme includes the amount of data to be processed. The script type is determined based on the data volume, and a script generation program or model of the script type is called to automatically generate the governance script according to the governance scheme. For example, a program to generate governance scripts based on the governance scheme can be pre-written, obtaining relevant governance elements from the governance scheme to generate the governance script. Governance elements include governance objects and governance methods, etc. For example, an artificial intelligence model can also be trained to automatically generate governance scripts based on the governance scheme and script type. For instance, a large model can be used to automatically generate governance scripts based on the governance scheme.
[0059] In one optional embodiment, the script type is determined and a governance script is automatically generated according to the governance scheme, including: when the amount of data to be detected exceeds a preset data amount threshold, a governance script that can run in the big data processing engine is automatically generated according to the governance scheme.
[0060] The preset data volume threshold can be a pre-defined maximum data volume. When the preset data volume threshold is exceeded, it can be determined that the amount of data to be processed is large. The preset data volume threshold can be determined by professional technicians based on experience or experimentation. For example, the preset data volume threshold can be 100 million records. The big data processing engine can be a data processing engine with a high data processing speed, suitable for processing large amounts of data. When the amount of data to be detected exceeds the preset data volume threshold, a governance script that can run in the big data processing engine is automatically generated according to the governance plan, so that data governance can be quickly achieved by relying on the powerful processing efficiency of the big data processing engine. For example, the governance script includes parts such as initializing the session, loading data, and defining the governance method.
[0061] When the amount of data to be detected exceeds a preset data volume threshold, a governance script that can run in the big data processing engine is automatically generated according to the governance plan. This can improve data governance efficiency by leveraging the high efficiency of the big data processing engine.
[0062] S250. The data to be tested is processed using a processing script to obtain the data to be verified.
[0063] The data to be verified is the data after the data to be detected has been processed by the processing script, and is used for further verification to obtain the processing results. Running the processing script processes the data to be detected, resulting in the data to be verified.
[0064] S260. According to the governance plan, the data to be verified is verified, and the governance results are obtained after the verification is passed.
[0065] Validating the data to be validated verifies whether it meets the governance requirements of the governance scheme. For example, if the governance scheme is to remove spaces from numbers, the validation method is to check whether the numbers in the data to be validated contain spaces. If the validation passes, the data to be validated is determined as the governance result. Optionally, if the validation fails, the process returns to step S240 to continue data governance until a governance result is obtained, ensuring the accuracy of the data governance.
[0066] The technical solution of this embodiment determines the script type and automatically generates a governance script according to the governance scheme. Determining the script type ensures the efficiency of governance, and automatically generating the governance script improves the intelligence of data governance and reduces labor costs. The governance script is used to govern the data to be detected to obtain the data to be verified. According to the governance scheme, the data to be verified is verified, and after passing the verification, the governance result is obtained. Verifying the data to be verified ensures the accuracy of the governance result and the effectiveness of data governance.
[0067] Example 3
[0068] Figure 3 This is a flowchart of a data governance method provided in Embodiment 3 of this application. The technical solution of this embodiment is further refined based on the above technical solution.
[0069] Furthermore, before "acquiring the data to be tested and performing quality testing on the data to be tested using at least two data quality testing models to obtain the testing results", add: "acquiring historical problem logs and classifying the problem data in the historical problem logs according to the problem type to obtain sample data for each problem type; the problem type includes at least accuracy, completeness, consistency, and repeatability; training the corresponding data quality testing model using the sample data for each problem type to obtain the data quality testing model for each problem type", so as to obtain the data quality testing model for each problem type.
[0070] See Figure 3 One data governance method shown includes:
[0071] S310. Obtain historical issue logs and classify the issue data in the historical issue logs according to issue type to obtain sample data for each issue type; issue types include at least accuracy, completeness, consistency and repeatability.
[0072] Historical issue logs can be historical error logs, used to obtain sample data for each issue type. The historical issue logs are categorized according to issue type to obtain sample data for each issue type. Issue types can include accuracy, completeness, consistency, and repeatability, etc., which are not specifically limited in this application.
[0073] S320. Train the corresponding data quality detection model using sample data of each problem type to obtain the data quality detection model for each problem type.
[0074] Different data quality inspection models can be assigned to different problem types to improve the accuracy of data quality inspection. Supervised training of the corresponding data quality inspection models is performed using sample data from each problem type to obtain the data quality inspection models for each problem type.
[0075] For example, when the problem type is accuracy, the corresponding data quality inspection model can be an Isolation Forest model. For example, when the problem type is completeness, the corresponding data quality inspection model can be a Logistic Regression or Support Vector Machine model. For example, when the problem type is consistency, the corresponding data quality inspection model can be a Decision Tree or Random Forest model. For example, when the problem type is repeatability, the corresponding data quality inspection model can be a Mixed Effects Model or a Multilevel Linear Model.
[0076] In an optional embodiment, the method further includes: periodically acquiring the latest data definition file or new data samples, incrementally training at least two data quality detection models, and updating at least two data quality detection models.
[0077] The periodicity can be a pre-defined incremental training period, which can be determined by professional technicians based on experience or experimentation; this application does not impose specific limitations on this. The data definition file can be a specification document defining various data points, such as the data types of each field, used for incremental training of the data quality detection model. For example, a data definition file can be used to incrementally train a data quality detection model for consistency issues, updating the model to ensure that the trained model can adapt to new changes in data definitions, avoiding false positives and false negatives. New data samples can be samples consisting of data added within the current training period, used for incremental training of the data quality detection model. For example, new data samples can be used to incrementally train a data quality detection model for accuracy issues, updating the model to ensure that the trained model can adapt to new dynamic changes in data, avoiding false positives and false negatives.
[0078] Manual processing cannot monitor data changes in real time and is difficult to respond quickly to business needs. Incremental training can improve the robustness of data quality detection models to data changes and enhance their reliability.
[0079] For example, when selecting the Isolation Forest model as the corresponding data quality detection model for the accuracy problem type, the contamination parameter of the model can be dynamically adjusted according to the proportion of outliers in the data, and the model can be retrained periodically using a dataset containing the latest data features to adapt to changes in data distribution.
[0080] For example, a data quality detection model corresponding to the integrity problem type can adopt an incremental learning approach when new data is added. This involves adding the features of the new data and the corresponding integrity labels to the training set, and then using algorithms such as stochastic gradient descent to update and train the model, enabling the model to continuously learn new data patterns.
[0081] For example, a data quality detection model corresponding to a consistency problem type can periodically collect new data with different consistency states and incrementally train the model. Alternatively, an online learning approach can be used, where new data is immediately incorporated into the training process as it is generated, and model parameters are adjusted through mini-batch updates to maintain applicability to changes in data consistency.
[0082] For example, when selecting a mixed-effects model or a multilevel linear model for data quality assessment of repetitive problem types, the random and fixed effects parameters in the model can be updated based on new repeated data. Specifically, new repeated measurements are added to the training set, and the model parameters are re-evaluated to better capture individual differences and group trends in the data.
[0083] By periodically acquiring the latest data definition files or adding new data samples, incrementally training and updating at least two data quality detection models, the adaptability of the data quality detection models to data changes can be ensured, and the accuracy and reliability of the data quality detection models can be improved.
[0084] S330. Obtain the data to be tested, and perform quality testing on the data to be tested using at least two data quality testing models to obtain the test results.
[0085] S340. Based on the detection results, perform correlation analysis to determine the cause of the problem.
[0086] S350. Determine the solution based on the cause of the problem.
[0087] S360. Generate a governance script based on the governance plan, and then use the governance script to process the data to be tested.
[0088] The technical solution of this embodiment obtains historical problem logs and classifies the problem data in the historical problem logs according to the problem type to obtain sample data for each problem type. The problem types include at least accuracy, completeness, consistency, and repeatability. The data quality detection model for each type is trained using the sample data of each problem type to obtain the data quality detection model for each problem type. By training different data quality detection models for different problem types, the detection accuracy of each data quality detection model for the corresponding data problems can be improved, thereby improving the effectiveness and accuracy of subsequent data governance.
[0089] Example 4
[0090] Figure 4 The diagram shown is a structural schematic of a data governance device according to Embodiment 4 of this application. This embodiment is applicable to situations involving automatic data governance. The specific structure of the data governance device is as follows:
[0091] The detection result acquisition module 410 is used to acquire the data to be detected and perform quality detection on the data to be detected using at least two data quality detection models to obtain the detection result.
[0092] The detection result analysis module 420 is used to perform correlation analysis based on the detection results to determine the cause of the problem.
[0093] The governance solution determination module 430 is used to determine the governance solution based on the cause of the problem;
[0094] The data to be tested governance module 440 is used to generate governance scripts according to the governance scheme and to govern the data to be tested through the governance scripts.
[0095] The technical solution of this embodiment acquires the data to be tested and performs quality testing on the data using at least two data quality testing models to obtain test results. This enables automatic monitoring and testing of the data to be tested, allowing for timely detection of problems. The accuracy of the quality testing models can improve the accuracy of monitoring. Correlation analysis is performed based on the test results to identify the causes of problems, and joint analysis accurately locates the root causes. Based on the causes of problems, a governance plan is determined, automatically identifying targeted governance solutions. A governance script is generated based on the governance plan, and the data to be tested is governed using the governance script. By generating the governance script, automatic data governance can be achieved, improving the efficiency and intelligence of data governance. Therefore, the technical solution of this application solves the problems of low efficiency in manual review and governance, and poor intelligence in dealing with complex and ever-changing data environments, achieving the effect of improving the efficiency and intelligence of data governance.
[0096] Optionally, the data governance module 440 to be detected includes:
[0097] The governance script generation unit is used to determine the script type and automatically generate the governance script based on the governance scheme.
[0098] The unit for obtaining data to be verified is used to process the data to be detected using the processing script to obtain the data to be verified.
[0099] The governance result obtaining unit is used to verify the data to be verified according to the governance scheme, and obtain the governance result after passing the verification.
[0100] Optionally, the governance script generation unit includes:
[0101] The big data processing engine script generation subunit is used to automatically generate a governance script that can run in the big data processing engine when the amount of data to be detected exceeds a preset data amount threshold, based on the governance scheme.
[0102] Optional, the data governance device also includes:
[0103] The sample data classification module is used to obtain historical issue logs and classify the issue data in the historical logs according to issue type to obtain sample data for each issue type; the issue types include at least accuracy, completeness, consistency and repeatability;
[0104] The data quality detection model training module is used to train the corresponding data quality detection model using sample data of each problem type, so as to obtain the data quality detection model for each problem type.
[0105] Optional, the data governance device also includes:
[0106] The incremental training module for data quality detection models is used to periodically obtain the latest data definition files or new data samples, perform incremental training on at least two data quality detection models, and update at least two data quality detection models.
[0107] Optionally, the detection result analysis module 420 includes:
[0108] The data correlation judgment unit is used to determine whether there is a correlation between problematic data in the detection results;
[0109] The problem cause analysis unit is used to perform correlation analysis on the problems in the data to be tested that are related, and to obtain the cause of the problem if the condition is found.
[0110] The data governance apparatus provided in this application embodiment can execute the data governance method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the data governance method.
[0111] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.
[0112] Example 5
[0113] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this application, as shown below. Figure 5 As shown, the electronic device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the electronic device can be one or more. Figure 5 Taking a processor 510 as an example; the processor 510, memory 520, input device 530, and output device 540 in the electronic device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.
[0114] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data governance method in this embodiment (e.g., detection result acquisition module 410, detection result analysis module 420, governance scheme determination module 430, and data governance module 440). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 520, thereby implementing the aforementioned data governance method.
[0115] The memory 520 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 520 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include memory remotely located relative to the processor 510, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0116] Input device 530 can be used to receive input character information and generate key signal inputs related to user settings and function control of the electronic device. Output device 540 may include display devices such as a display screen.
[0117] Example 6
[0118] Embodiment 6 of this application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a data governance method. The method includes: acquiring data to be tested, and performing quality testing on the data to be tested using at least two data quality testing models to obtain testing results; performing correlation analysis based on the testing results to obtain the cause of the problem; determining a governance scheme based on the cause of the problem; generating a governance script based on the governance scheme, and governing the data to be tested using the governance script.
[0119] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the method operations described above, but can also perform related operations in the data governance method provided in any embodiment of this application.
[0120] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0121] It is worth noting that in the embodiments of the above data governance device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.
[0122] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.
Claims
1. A data governance method, characterized in that, include: Acquire the data to be tested, and perform quality testing on the data to be tested using at least two data quality testing models to obtain the test results; Based on the detection results, a correlation analysis was performed to determine the cause of the problem. Based on the causes of the problem, determine the solution; A governance script is generated based on the governance scheme, and the data to be detected is governed using the governance script.
2. The method according to claim 1, characterized in that, The step of generating a governance script according to the governance scheme and governing the data to be detected using the governance script includes: Based on the governance scheme, determine the script type and automatically generate the governance script; The data to be detected is processed using the processing script to obtain the data to be verified. According to the governance scheme, the data to be verified is verified, and the governance result is obtained after the verification is passed.
3. The method according to claim 2, characterized in that, The step of determining the script type and automatically generating the governance script according to the governance scheme includes: When the amount of data to be detected exceeds a preset data volume threshold, a governance script that can run in the big data processing engine is automatically generated according to the governance scheme.
4. The method according to claim 1, characterized in that, The at least two data quality detection models are obtained through the following methods: Retrieve historical issue logs and categorize the issue data in the historical logs according to issue type to obtain sample data for each issue type; the issue types include at least accuracy, completeness, consistency, and repeatability. By training the corresponding data quality detection model with sample data of each problem type, the data quality detection model for each problem type is obtained.
5. The method according to claim 1, characterized in that, The method further includes: Periodically obtain the latest data definition files or new data samples, incrementally train at least two data quality detection models, and update at least two data quality detection models.
6. The method according to claim 1, characterized in that, The step of performing correlation analysis based on the detection results to obtain the cause of the problem includes: Determine whether there is any correlation between the problematic data in the test results; If so, then perform correlation analysis on the issues in the data to be tested that are correlated to find the cause of the problem.
7. A data governance device, characterized in that, include: The detection result acquisition module is used to acquire the data to be detected and to perform quality detection on the data to be detected using at least two data quality detection models to obtain the detection result. The detection result analysis module is used to perform correlation analysis based on the detection results to determine the cause of the problem. The governance solution determination module is used to determine a governance solution based on the cause of the problem. The data to be detected governance module is used to generate a governance script according to the governance scheme, and to govern the data to be detected through the governance script.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data governance method as described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data governance method as described in any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data governance method as described in any one of claims 1-6.