Data quality management method and system, storage medium and electronic equipment

Abstract

The invention discloses a data quality management method and system, a storage medium and electronic equipment, and relates to the technical field of data management. The method comprises the steps of obtaining a target plan of a military-civilian fusion project, and determining key sub-projects in the target plan; collecting target data of each key sub-project, and determining associated data of the target data; checking and evaluating the target data to obtain a first evaluation result; evaluating the associated data based on the first evaluation result to obtain a second evaluation result of the target data; and generating a data quality report based on the first evaluation result and the second evaluation result, and visually displaying the data quality report. By implementing the technical scheme provided by the invention, the accuracy of data quality evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data management, and in particular to a data quality management method, system, storage medium and electronic device. Background Art

[0002] In the era of data-driven decision-making, data quality management has become a key factor in ensuring project success, especially in complex system projects involving the integration of defense and civilian technologies, where the accuracy, completeness and timeliness of data are crucial to the overall success of the project.

[0003] In military-civilian integration projects, a large amount of data processing and analysis is involved to support project decision-making and management. Common data quality management methods include data collection, verification and analysis. These methods generally involve separate evaluation of project data to assess whether the data meets the preset quality standards. They cannot fully evaluate the impact of the data on the quality of each data set, resulting in a low accuracy in data quality assessment. Summary of the invention

[0004] The present application provides a data quality management method, system, storage medium and electronic device, which can improve the accuracy of data quality assessment.

[0005] In a first aspect, the present application provides a data quality management method, the method comprising: Obtain the target plan of the military-civilian integration project and identify the key sub-projects in the target plan; Collecting target data of each of the key sub-items and determining associated data of the target data; Verifying and evaluating each of the target data to obtain a first evaluation result; Evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data; A data quality report is generated based on the first evaluation result and the second evaluation result, and the data quality report is visually displayed.

[0006] By adopting the above technical solution, the key sub-projects are determined by obtaining the target plan of the military-civilian integration project, collecting target data and its related data, establishing quality assessment indicators to evaluate the target data separately to obtain a first assessment result, and based on the first assessment result, evaluating the related data to obtain a second assessment result, and then generating a visual data quality report containing multi-dimensional quality analysis of the target data and related data. By associating the quality impact assessment of the target data with the related data, the quality impact of the target data on each data set can be more comprehensively evaluated, thereby improving the accuracy of data quality assessment.

[0007] Optionally, determining the key sub-items in the target plan includes: Conduct hierarchical analysis on the target plan and establish a project hierarchical structure model; Based on the project hierarchy model, calculating the contribution of each of the sub-projects to the overall goal; The sub-items whose contribution is higher than the contribution threshold are regarded as key sub-items.

[0008] By adopting the above technical solution and establishing a project hierarchy model through hierarchical analysis, we can comprehensively consider the logical relationship between each sub-project from the overall to the local, evaluate the contribution of each sub-project to the final goal, calculate the contribution of each sub-project, select sub-projects with a contribution higher than the threshold as key sub-projects, and focus on data quality management of the identified key sub-projects. This can improve the effectiveness of data-driven decision-making and focus data quality management on the most critical project aspects, thereby greatly improving the work efficiency of data quality management.

[0009] Optionally, the collecting target data of each of the key sub-items and determining associated data of the target data includes: Establishing a data collection template for each key sub-project, wherein the data collection template includes data type, source, format and collection cycle; According to the data collection template, collecting target data at the data source interface of each of the key sub-projects; A data association map of a military-civilian integration project is constructed, and data associated with the target data is identified as associated data based on the data association map.

[0010] By adopting the above technical solution and establishing a standardized data collection template for each key sub-project, the standardized collection of target data is achieved, ensuring the integrity and consistency of data collection. The technology based on the data association graph can automatically identify the logical relationship between target data and accurately determine all associated data related to the target data.

[0011] Optionally, the verifying and evaluating each of the target data to obtain a first evaluation result includes: Constructing quality assessment indicators for multi-dimensional data, the indicator system includes completeness, accuracy, consistency, timeliness and availability; Applying a fuzzy comprehensive evaluation method based on the target data, calculating the degree of membership of each of the quality assessment indicators; Determine the weight of each quality assessment indicator by using the analytic hierarchy process; Calculating a quality score of the target data based on the membership degree and the weight; According to a preset quality grade classification standard, the quality score is mapped to a quality grade, and a first evaluation result including the quality score and the quality grade is generated.

[0012] By adopting the above technical solution and building a quality assessment system with multi-dimensional indicators, we can comprehensively consider the impact of different quality factors on the target data, and then calculate the membership of the quality indicators, and determine the indicator weights in combination with hierarchical analysis, thereby realizing the quantification and weighting of quality assessment. Based on the mapping of quality scores and quality grades, the quality level of the target data can be intuitively reflected.

[0013] Optionally, evaluating the associated data based on the first evaluation result to obtain a second evaluation result of the target data includes: Construct a quality impact map of target data and associated data; Calculating the quality impact value of the target data on each associated data based on the first evaluation result and the quality impact map; Performing quality assessment on each associated data to obtain a quality assessment result of each associated data; Calculating a consistency index between the quality impact value and the quality assessment result; Based on the consistency index, the quality score in the first evaluation result is corrected to generate a second evaluation result including the corrected quality score and quality impact analysis.

[0014] By adopting the above technical solution, the quality impact of target data on associated data is evaluated based on the quality impact map technology, and the visual modeling of quality impact between data is realized. By calculating the quality impact value and combining it with the quality assessment results of the associated data itself, the transmission effect of the target data error on the associated data can be quantified. The calculation of the consistency index realizes the difference analysis between quality assessment and quality impact, and re-calibrates the target data quality score with the consistency index, making the assessment results more accurate and comprehensive.

[0015] Optionally, generating a data quality report based on the first evaluation result and the second evaluation result includes: Constructing a multi-level data quality assessment matrix, wherein the data quality assessment matrix includes a first quality dimension of the target data and a second quality dimension of the associated data; Mapping the first evaluation result to a first quality dimension of target data of the matrix, and mapping the second evaluation result to a second quality dimension of associated data of the matrix; Calculating a comprehensive quality score of the first quality dimension and the second quality dimension, and determining a comprehensive quality grade based on the comprehensive quality score; Generate a data quality report containing comprehensive quality score, comprehensive quality grade, first quality dimension and second quality dimension analysis.

[0016] By adopting the above technical solution and constructing a multi-level data quality assessment matrix, a visual expression of the multi-dimensional quality results of the target data and related data is achieved. The first assessment result and the second assessment result can be intuitively mapped to the different dimensions of the assessment matrix, and the comprehensive quality score is calculated and the comprehensive quality level is determined, thereby achieving an overall data quality evaluation of the target data and related data.

[0017] Optionally, the method further includes: Determine the key factors affecting data quality in each of the key sub-projects based on the data quality report; Generate a proposal for each of the key sub-projects based on the key factors.

[0018] By adopting the above-mentioned technical solutions and customized quality management solutions for key sub-projects, we can effectively solve data quality problems, control the data quality of key sub-projects at a reasonable level, and maximize the effectiveness of data in supporting decision-making in military-civilian integration projects.

[0019] In a second aspect of the present application, a data quality management system is provided, the system comprising: A sub-project determination module is used to obtain the target plan of the military-civilian integration project and determine the key sub-projects in the target plan; A related data determination module, used for collecting target data of each of the key sub-items and determining related data of the target data; A first evaluation module, used to verify and evaluate each of the target data to obtain a first evaluation result; A second evaluation module, configured to evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data; A quality report generation module is used to generate a data quality report based on the first evaluation result and the second evaluation result, and to visually display the data quality report.

[0020] In a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above method steps.

[0021] In a fourth aspect of the present application, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0022] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: This application determines key sub-projects by obtaining the target plan of the military-civilian integration project, collects target data and its related data, establishes quality assessment indicators to evaluate the target data separately to obtain a first assessment result, and evaluates the related data based on the first assessment result to obtain a second assessment result, and then generates a visual data quality report containing multi-dimensional quality analysis of the target data and the related data. By associating the quality impact assessment of the target data and the related data, the quality impact of the target data on each data set can be more comprehensively evaluated, thereby improving the accuracy of the data quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a data quality management method provided in an embodiment of the present application; Figure 2 It is a module schematic diagram of a data quality management system provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0024] Description of reference numerals: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0025] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0026] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0027] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0028] The following will provide a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0029] Please refer to Figure 1 , a flow chart of a data quality management method is proposed, the method can be implemented by a computer program, can be implemented by a single-chip microcomputer, or can be run on a data quality management system, the computer program can be integrated in a computer device, or can be run as an independent tool application, specifically, the method includes steps 10 to 50, the above steps are as follows: Step 10: Obtain the target plan for the military-civilian integration project and identify the key sub-projects in the target plan.

[0030] In the present application, the military-civilian integration project refers to a comprehensive project that organically combines military and civilian resources, technology, information and other elements. Such projects are aimed at promoting the coordinated development of military and civilian fields, and may involve multiple aspects such as technology research and development, product production, and resource sharing.

[0031] In the present application, the target plan refers to the overall planning and expected goals of the military-civilian integration project, which includes the specific goals, implementation steps, time nodes, resource allocation, etc. of the project.

[0032] In the embodiment of the present application, a key sub-project refers to a sub-project that has an important influence or a decisive role in the entire military-civilian integration project.

[0033] Specifically, first collect comprehensive information on the military-civilian integration project, including project background, overall goals, expected results, implementation cycle, etc., to form a detailed target plan document. Then, use the hierarchical analysis method to systematically decompose the target plan and establish a project hierarchy model. This method can clearly show the relationship between the various components of the project and facilitate subsequent analysis. On this basis, evaluate the contribution of each sub-project to the overall goal. Consider multiple factors during the evaluation process, such as technological innovation, resource requirements, expected benefits, etc., and assign contribution scores to each sub-project through a combination of quantitative and qualitative methods. Next, set a contribution threshold, and identify sub-projects with contributions higher than this threshold as key sub-projects, ensuring that the selected key sub-projects truly represent the core value and key links of the project.

[0034] Based on the above embodiment, as an optional embodiment, the step of determining the key sub-projects in the target plan may further include the following steps: Step 101: Conduct hierarchical analysis on the target plan and establish a project hierarchy model.

[0035] Specifically, first parse the target planning document of the military-civilian integration project to fully understand the overall goal, sub-goals and specific tasks of the project. Then use the hierarchical analysis method to decompose the project goals from top to bottom. In this process, the team takes the overall goal as the top level and gradually refines it into multiple levels of sub-goals and specific tasks to form a tree-like hierarchy. This decomposition method can clearly show the subordinate relationship and logical connection between goals. In the process of establishing the hierarchy, it is necessary to pay attention to maintaining the independence and integrity between the levels to ensure that each sub-goal can accurately reflect an aspect of its superior goal. At the same time, the particularity of the military-civilian integration project should also be considered to reflect the balance and integration of military and civilian needs in the hierarchy. After the decomposition is completed, the hierarchy is visualized to form a project hierarchy model, which intuitively shows the overall architecture of the project and can clearly understand the position and importance of each goal and task in the entire project.

[0036] Step 102: Based on the project hierarchy model, calculate the contribution of each sub-project to the overall goal.

[0037] Specifically, first read the established project hierarchy model data, start from the bottom sub-project, and calculate the contribution layer by layer. The calculation process adopts the pairwise comparison algorithm in the hierarchical analysis method. The system automatically compares the importance of each pair of sub-projects at the same level and assigns corresponding weight values. This comparison is based on multiple pre-set evaluation indicators, such as the degree of technological innovation, resource input, expected benefits, and military-civilian integration. The system calls the scoring data in the expert knowledge base to simulate the expert scoring process and determine the relative importance of each pair of sub-projects. Subsequently, the system uses the eigenvector algorithm to process these pairwise comparison results and automatically calculates the local weight of each sub-project in its own level. Then, a hierarchical comprehensive operation is performed to calculate the global weight of each sub-project to the overall goal, that is, the contribution.

[0038] Step 103: The sub-projects whose contribution is higher than the contribution threshold are regarded as key sub-projects.

[0039] Specifically, the calculated contribution data of each sub-project is read, and then the preset contribution threshold is called. The contribution threshold is usually a value pre-set based on project characteristics and management requirements. The system automatically compares the contribution of each sub-project with this threshold, and identifies sub-projects with contributions higher than the contribution threshold as key sub-projects.

[0040] Step 20: Collect target data for each key sub-project and determine the associated data of the target data.

[0041] In the present embodiment of the application, target data refers to core data directly related to the identified key sub-projects. These data are crucial to achieving the specific goals of the key sub-projects and are data sets that need to be managed and evaluated in a focused manner.

[0042] In the embodiments of the present application, associated data refers to data that is directly or indirectly associated with the target data and may affect the quality, completeness or accuracy of the target data. Although these data are not the core data of the key sub-projects, they play an important role in fully understanding and evaluating the quality of the target data.

[0043] Specifically, the system first automatically generates corresponding data collection templates based on the list of key sub-projects determined in the previous step. These templates contain key information such as data type, source, format and collection cycle, ensuring the standardization and consistency of data collection. Subsequently, the system automatically connects to the data sources of each key sub-project through the preset interface, and accurately extracts the target data according to the parameters defined in the data collection template. In this process, the system automatically checks the integrity and format compliance of the data. If an abnormality is found, it will be recorded and triggered immediately. After completing the collection of the target data, the system automatically starts the identification program of the associated data. This program first analyzes the characteristics and attributes of the collected target data, and then searches for data items that may be related to the target data in the database of the entire military-civilian integration project according to the preset association rules. The association rules include but are not limited to direct reference relationships, logical dependencies, time series relationships, etc. between data. The system uses data association graph technology to automatically build an association network centered on the target data, which clearly shows the various association relationships between data. This method can not only identify directly related data, but also discover potential indirect associations, laying the foundation for subsequent comprehensive quality assessment.

[0044] Based on the above embodiment, as an optional embodiment, the step of collecting target data of each key sub-item and determining associated data of the target data may further include the following steps: Step 201: Create a data collection template for each key sub-project, where the data collection template includes data type, source, format and collection cycle.

[0045] Specifically, the list of key sub-projects is read and the preset data collection rules are called. For each key sub-project, the system automatically analyzes its characteristics and data requirements and matches the appropriate data collection rules from the rule base. Based on these rules, the system generates an initial template containing data types, sources, formats, and collection cycles. The data type field defines the specific types of data that need to be collected, such as performance indicators, technical parameters, etc. The data source field indicates the location where the data is generated or stored, which may be a specific database, sensor, or external system; the data format field specifies the storage form of the data, such as numeric, text, or a specific file format; the collection cycle field specifies the time interval for data updates, which may be real-time, daily, or weekly. After the initial template is generated, the system automatically performs a consistency check to ensure that there is no conflict in the data collection requirements between different key sub-projects. If a potential conflict is found, the system will automatically adjust according to the preset priority rules to obtain the data collection templates for each key sub-project.

[0046] Step 202: According to the data collection template, target data is collected at the data source interface of each key sub-project.

[0047] Specifically, the data collection template of each key sub-project is read and the information such as data type, source, format and collection cycle is parsed. Subsequently, the system automatically connects to the data source interface of each key sub-project according to the data source specified in the template. These interfaces may be database systems, sensor networks, external APIs or other data generating devices. The system uses preset data transmission protocols, such as REST API, JDBC or custom protocols, to establish secure data communication channels with these interfaces. After the connection is established, the system constructs corresponding data query or extraction instructions according to the data type and format defined in the template. These instructions are sent to the data source interface to trigger the extraction process of the target data. The system will automatically schedule data collection tasks according to the collection cycle specified in the template to ensure timely updating of data.

[0048] Step 203: Construct a data association graph for the military-civilian integration project, and identify data associated with the target data as associated data based on the data association graph.

[0049] Specifically, the collected target data and all data sets in the project database are called, and then the system starts the data association analysis engine, which uses graph database technology and semantic analysis algorithms. Each data item is regarded as a node in the graph, and the connection between nodes is automatically established by analyzing multi-dimensional factors such as reference relationships, logical dependencies, and time series correlations between data items to form a preliminary data association network. Then, the system applies machine learning algorithms, such as association rule mining and cluster analysis, to further discover potential data associations and obtain a data association map. During the construction process, the system calculates the centrality and connection strength of each node in real time to quantify the importance and degree of association of the data item in the entire network. After the construction is completed, the system uses the target data node as the center of the association map and automatically tracks all nodes directly and indirectly connected to it. According to the preset association strength threshold, the system screens out data nodes that are significantly associated with the target data and marks the data corresponding to these nodes as associated data.

[0050] Step 30: Verify and evaluate each target data to obtain a first evaluation result.

[0051] In the embodiment of the present application, verification and evaluation refers to the system's comprehensive and systematic quality inspection and evaluation process of the collected target data to identify potential data quality issues.

[0052] The first evaluation result in the embodiment of the present application refers to the direct output generated by the verification and evaluation process, and this result usually includes content such as quality score or quality grade.

[0053] Specifically, the system first calls the target data and the preset data quality assessment rule library, and the system starts the multi-dimensional data quality assessment engine, which contains five core assessment modules: completeness, accuracy, consistency, timeliness and availability. For completeness assessment, the system automatically checks the filling status of each data field and calculates the data completeness rate; in accuracy assessment, the system uses the preset data range rules and format rules to verify the rationality and correctness of the data value; consistency assessment involves comparing related data items to detect potential contradictions or conflicts; timeliness assessment determines the update status of the data by comparing the data timestamp with the current time; availability assessment simulates data application scenarios to test whether the data meets specific business needs. During the assessment process, the system assigns weights to each dimension according to the preset scoring rules and calculates the weighted score. At the same time, the system also records the specific problems found in each dimension to form a detailed list of problems. After the assessment is completed, the system divides the data quality into excellent, good, general, poor and other levels based on the overall score and the preset grade classification standards. Finally, the system generates a comprehensive assessment report containing quality scores, quality grades, problem lists and statistical data. This report is the first assessment result.

[0054] Based on the above embodiment, as an optional embodiment, the step of verifying and evaluating each target data to obtain the first evaluation result may further include the following steps: Step 301: Construct quality assessment indicators for multi-dimensional data, where the indicator system includes completeness, accuracy, consistency, timeliness and availability.

[0055] Specifically, the preset data quality assessment rule library and project-specific demand parameters are called, and then the indicator construction engine is started. The engine automatically generates an indicator system that adapts to the characteristics of the current project based on machine learning algorithms and expert knowledge bases. For the integrity indicator, the system designs sub-indicators such as data field filling rate and required field completeness; the accuracy indicator includes sub-indicators such as the rationality of the numerical range and the correctness of the format; the consistency indicator covers sub-indicators such as cross-table consistency and logical relationship conformity; the timeliness indicator considers factors such as data update frequency and real-time performance; and the availability indicator includes aspects such as data readability and ease of use. During the construction process, the system will dynamically adjust the weights and thresholds of each indicator according to the specific needs of the project and the characteristics of the data. For example, for data with high real-time requirements, the system will automatically increase the weight of the timeliness indicator. At the same time, the system will also consider the relationship between indicators and build an indicator association network to more comprehensively evaluate data quality.

[0056] Step 302: Apply the fuzzy comprehensive evaluation method based on the target data to calculate the membership degree of each quality assessment index.

[0057] Specifically, a fuzzy evaluation set is first established for each quality assessment indicator (completeness, accuracy, consistency, timeliness and availability), which usually includes four levels: "excellent", "good", "average" and "poor". Then, the system calculates the membership of each indicator to each evaluation level based on the preset fuzzy rules and the actual situation of the target data. During the calculation process, the system uses fuzzy mathematical tools such as triangular fuzzy numbers or trapezoidal fuzzy numbers to uniformly convert quantitative and qualitative indicators into fuzzy numbers. For example, for the integrity indicator, the system may use the data field filling rate as a quantitative indicator and convert it into the membership of each evaluation level by setting a fuzzy interval. For indicators that are difficult to quantify, such as availability, the system will combine the expert knowledge base and machine learning algorithms to construct a fuzzy evaluation function. In the process of calculating the membership, the system will also consider the mutual influence between indicators and adjust the membership calculation results through fuzzy association rules. After the calculation is completed, the system generates a membership matrix, in which the rows represent different quality assessment indicators, the columns represent different evaluation levels, and the matrix elements are the corresponding membership values.

[0058] Step 303: Determine the weight of each quality assessment indicator using the analytic hierarchy process.

[0059] Step 304: Calculate the quality score of the target data based on the membership degree and the weight.

[0060] Specifically, based on the hierarchical analysis decision system, a hierarchical model is first constructed, with the data quality assessment target as the top layer and the five indicators of completeness, accuracy, consistency, timeliness and availability as the second layer. The system automatically generates a judgment matrix for the characteristics of the current project based on the judgment matrix template in the expert knowledge base. During the generation process, the system will consider the specific needs and data characteristics of the project and dynamically adjust the comparison values ​​in the judgment matrix. For example, for projects with high real-time requirements, the system will automatically increase the importance of the timeliness indicator relative to other indicators. After generating the judgment matrix, the system uses the eigenvalue method to calculate the weight vector of each indicator. During the calculation process, the system will automatically perform a consistency check. If the consistency ratio exceeds the preset threshold, the system will automatically adjust the judgment matrix and recalculate until the consistency requirements are met. After obtaining the weight vector, the weight information is combined with the membership matrix calculated in the previous step, and the weighted average method is used to calculate the final quality score of the target data.

[0061] Step 305: According to a preset quality grade classification standard, the quality score is mapped to the quality grade, and a first evaluation result including the quality score and the quality grade is generated.

[0062] Specifically, the preset quality grade classification standard is read, including four grades of "excellent", "good", "average", and "poor", and each grade corresponds to a score range. For example, 90 points or more is "excellent", 80-90 points is "good", 60-80 points is "average", and below 60 points is "poor". The quality score is mapped to the quality grade, and the quality score, quality grade, membership of each grade, and confidence are integrated into a structured first evaluation result.

[0063] Step 40: Evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data.

[0064] Specifically, a quality impact map of the target data and the associated data is constructed, and the associated data nodes directly related to the target data are extracted from the data association map. For each associated data node, the system queries its corresponding first evaluation result, including the quality score and quality level. Then, the system calculates the degree of influence of each associated data on the quality of the target data according to the preset association influence rules. These rules take into account factors such as association strength, data flow direction (such as upstream data or downstream data), and association type (such as direct reference or indirect dependency). For example, for strongly associated upstream data, its quality problems may have a greater impact on the target data, and the system will give it a higher impact weight. The system uses a weighted average method or a more complex machine learning model (such as a random forest or a neural network) to comprehensively calculate the overall quality impact of the associated data. In this process, the system considers the possible interaction effects between the associated data and captures complex association patterns through graph network analysis technology. After the calculation is completed, the system merges the quality impact of the associated data with the first evaluation result of the target data to generate a second evaluation result.

[0065] Based on the above embodiment, as an optional embodiment, the step of evaluating the associated data based on the first evaluation result to obtain the second evaluation result of the target data further includes the following steps: Step 401: Construct a quality impact map of target data and associated data.

[0066] Specifically, all data nodes directly and indirectly related to the target data are extracted from the data association graph to form a preliminary node set. Subsequently, the system analyzes the relationship type between these nodes, including data flow direction (such as upstream data, downstream data), degree of dependence (such as strong dependence, weak dependence) and influence mode (such as direct influence, indirect influence). Based on these analysis results, the system constructs connecting edges between nodes, where the direction of the edge represents the propagation direction of the quality impact, and the weight of the edge represents the impact strength. The quality score and quality grade information in the first evaluation result are used to assign an initial quality state to each node in the graph. Next, the system applies network analysis techniques, such as centrality analysis algorithms, to calculate the importance weight of each node in the entire graph. This step takes into account factors such as node connectivity and influencing path length, and can identify key influencing nodes and paths. In order to capture the dynamic characteristics of quality impact, the system also introduces the time dimension. By analyzing historical data, time attributes are added to the edges in the graph to reflect the time delay of the impact propagation. The system also applies machine learning algorithms, such as graph neural networks, to learn and predict potential impact relationships between nodes and enrich the connection structure of the graph. Finally, the system generates a visual quality impact graph.

[0067] Step 402: Based on the first evaluation result and the quality impact map, calculate the quality impact value of the target data on each associated data.

[0068] Specifically, based on the constructed quality impact map, the system begins to calculate the quality impact value of the target data on each associated data. The calculation process uses a probabilistic reasoning model based on the Bayesian network, taking into account multiple factors: 1) the quality status of the target data; 2) the impact path from the target data to the associated data; 3) the quality status of the intermediate nodes on the path; 4) the impact strength between each node. The system simulates multiple scenarios of target data quality changes through the Monte Carlo simulation method, and calculates the probability distribution of the potential impact of these changes on the quality of each associated data. Finally, the system comprehensively analyzes these probability distributions to derive the quality impact value of the target data on each associated data.

[0069] Step 403: Perform quality assessment on each associated data to obtain a quality assessment result of each associated data.

[0070] Step 404: Calculate the consistency index between the quality impact value and the quality assessment result.

[0071] Specifically, the same quality assessment method as the target data is applied to each associated data node in the quality impact map. This includes using the weights determined by the hierarchical analysis method and the fuzzy comprehensive evaluation method to calculate the quality score and quality level of each associated data. The evaluation process takes into account multiple dimensions such as completeness, accuracy, consistency, timeliness and availability to ensure the comprehensiveness and comparability of the evaluation results. After completing the quality assessment of the associated data, the system compares these assessment results with the previously calculated quality impact values, and uses the weighted Kappa coefficient calculation method to calculate the consistency index between the quality impact values ​​and the actual quality assessment results. During the calculation process, the system takes into account the ordinal relationship between different quality levels and uses a weighted scheme to more accurately reflect the degree of inconsistency. The consistency index is a quantitative indicator used to measure the degree of consistency between the quality impact value and the actual quality assessment result.

[0072] Step 405: Based on the consistency index, the quality score in the first evaluation result is corrected to generate a second evaluation result including the corrected quality score and quality impact analysis.

[0073] Specifically, the necessity and extent of correction are first determined based on the value of the consistency index. If the consistency index is high (for example, greater than 0.8), the system may only make slight adjustments; if the consistency index is low, a larger correction is required. The correction process uses the Bayesian update method, taking the first evaluation result as the prior probability, the consistency index and its confidence interval as new evidence, and calculating the posterior probability distribution. This method can smoothly integrate new information and avoid over-correction. At the same time, the system considers the association relationship in the quality impact map and uses the graph convolutional neural network (GCN) model to capture the mutual influence between data items. The GCN model can learn and utilize the topological structure between data items to improve the accuracy of correction. During the correction process, the system also applies an adaptive learning rate strategy to dynamically adjust the correction amplitude according to the characteristics and importance of different data items. After the correction is completed, the system generates a second evaluation result, which includes the corrected quality score, quality level, and a detailed quality impact analysis report. The report not only includes the updated quality status, but also includes quality impact propagation path analysis, key impact node identification, potential risk warning, etc.

[0074] Step 50: Generate a data quality report based on the first evaluation result and the second evaluation result, and visualize the data quality report.

[0075] Specifically, the first evaluation results and the second evaluation results are compared and analyzed to identify key change points and trends. The system uses natural language generation (NLG) technology to automatically generate descriptive text to explain the reasons for changes in the evaluation results and the potential impact. The report content includes but is not limited to: overall data quality scores and grades, detailed scores for each dimension (completeness, accuracy, consistency, timeliness and availability), a list of quality issues, quality impact analysis results, consistency index analysis, identification of key data nodes, and suggestions for improvement. Then, a variety of visualization techniques, such as dashboards, heat maps, network diagrams, and trend diagrams, are used to transform complex data quality information into intuitive graphical representations. For example, a radar chart is used to show the score changes of each quality dimension, a Sankey diagram is used to show the propagation path of quality impacts, and a heat map is used to show the degree of consistency between data items. The system also applies interactive visualization technology, allowing users to dynamically adjust views and explore in depth the data quality details of interest.

[0076] Based on the above embodiment, as an optional embodiment, the step of generating a data quality report based on the first evaluation result and the second evaluation result further includes the following steps: Step 501: construct a multi-level data quality assessment matrix, where the data quality assessment matrix includes a first quality dimension of target data and a second quality dimension of associated data.

[0077] Step 502: Map the first evaluation result to a first quality dimension of target data of the matrix, and map the second evaluation result to a second quality dimension of associated data of the matrix.

[0078] Specifically, the structure of the data quality assessment matrix is ​​first predefined, where rows represent data items (including target data and associated data) and columns represent quality dimensions. The first quality dimension includes completeness, accuracy, consistency, timeliness, and availability, which are direct assessment indicators for target data. The second quality dimension includes influence, dependency, propagation, and criticality, which are assessment indicators for the impact of associated data on target data quality. The first assessment result is then mapped to the first quality dimension of the target data of the matrix, and the second assessment result is mapped to the second quality dimension of the associated data of the matrix.

[0079] Step 503: Calculate the comprehensive quality scores of the first quality dimension and the second quality dimension, and determine the comprehensive quality level based on the comprehensive quality scores.

[0080] Step 504: Generate a data quality report including an overall quality score, an overall quality grade, and an analysis of the first quality dimension and the second quality dimension.

[0081] Specifically, each indicator in the multi-level data quality assessment matrix is ​​standardized, and the fuzzy membership function is used to map the indicator values ​​of different scales to the [0,1] interval. Then the analytic hierarchy process (AHP) is applied to determine the weights of each indicator in the first quality dimension (including completeness, accuracy, consistency, timeliness and availability). At the same time, the system uses the analytic network process (ANP) to determine the weights of each indicator in the second quality dimension (including influence, dependency, propagation and criticality), which can consider the mutual influence between related data. The weight determination process takes into account the preset expert knowledge base and project characteristics, and realizes dynamic adjustment through machine learning algorithms. The weighted summation method is used to calculate the comprehensive score of the first quality dimension to reflect the quality status of the target data itself. For the second quality dimension, the system applies a deep learning model based on graph neural network, which can make full use of the structural information in the quality impact map to more accurately evaluate the overall impact of related data on the quality of target data. A multi-objective optimization method is used to construct an objective function that considers the trade-offs of the two dimensions, and the final comprehensive quality score is obtained through fuzzy synthesis operation. Then, the preset quality grade classification standard is applied to map the score to the corresponding quality grade. Finally, natural language generation (NLG) technology is used to create the text content of the report. The report includes a summary of the overall quality score and grade, a detailed analysis of the first quality dimension and the second quality dimension, an analysis of the contribution of each indicator to the overall score, and a quality trend analysis.

[0082] Based on the above embodiment, as an optional embodiment, a data quality management method may further include the following process: Specifically, the data quality report is deeply semantically analyzed, and the key information in the report is extracted using natural language processing technology, including the description of the quality problem, the degree of impact, the data items involved, and the quality dimensions. Then, the system associates this information with the quality impact map and uses the graph pattern matching algorithm to identify the potential factors affecting data quality. The system applies causal reasoning models, such as structural equation models (SEM) or Bayesian networks, to analyze the causal relationship between these factors and construct a causal chain network. In order to determine the key factors, the system uses a multi-criteria decision-making method to comprehensively consider the following aspects: 1) the degree of impact of the factors on data quality, which is quantified through sensitivity analysis; 2) the controllability of the factors, which is evaluated based on the expert knowledge base; 3) the cost of improvement, which is estimated using historical project data; 4) the sustainability of the improvement effect, which is predicted through time series analysis. The system applies a fuzzy comprehensive evaluation method to calculate a comprehensive importance score for each potential factor, and determines the key factors based on the score ranking. After determining the key factors, a recommended solution is generated for each key sub-project. First, a knowledge map is constructed to integrate historical project experience, industry best practices, and expert knowledge. Using a graph neural network model, the system is able to learn from this knowledge graph and reason out solutions that are appropriate for the current situation. To generate personalized recommendations, the system considers the characteristics of each key sub-project, such as project size, technical field, resource constraints, etc., uses a case-based reasoning approach to match the most similar historical cases, and makes appropriate adjustments. A multi-objective optimization algorithm is then used to generate a series of possible recommendations that seek a balance between improvement effects, implementation costs, and risks.

[0083] See also Figure 3 , is a module diagram of a data quality management system provided in an embodiment of the present application, the data quality management system may include: a sub-project determination module, an associated data determination module, a first evaluation module, a second evaluation module and a quality report generation module, wherein: A sub-project determination module is used to obtain the target plan of the military-civilian integration project and determine the key sub-projects in the target plan; A related data determination module, used for collecting target data of each of the key sub-items and determining related data of the target data; A first evaluation module, used to verify and evaluate each of the target data to obtain a first evaluation result; A second evaluation module, configured to evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data; A quality report generation module is used to generate a data quality report based on the first evaluation result and the second evaluation result, and to visually display the data quality report.

[0084] Optionally, the sub-project determination module is also used to perform a hierarchical analysis on the target plan and establish a project hierarchy model; based on the project hierarchy model, calculate the contribution of each sub-project to the overall target; and define the sub-projects whose contribution is higher than a contribution threshold as key sub-projects.

[0085] Optionally, the associated data determination module is also used to establish a data collection template for each key sub-project, the data collection template including a data type, source, format and collection cycle; according to the data collection template, collect target data at the data source interface of each key sub-project; construct a data association map of the military-civilian integration project, and identify data associated with the target data as associated data based on the data association map.

[0086] Optionally, the first evaluation module is also used to construct quality evaluation indicators for multidimensional data, and the indicator system includes completeness, accuracy, consistency, timeliness and availability; based on the target data, a fuzzy comprehensive evaluation method is applied to calculate the membership of each quality evaluation indicator; the weight of each quality evaluation indicator is determined using the hierarchical analysis method; based on the membership and weight, the quality score of the target data is calculated; according to a preset quality grade classification standard, the quality score is mapped to a quality grade, and a first evaluation result including the quality score and quality grade is generated.

[0087] Optionally, the second evaluation module is also used to construct a quality impact map of the target data and the associated data; based on the first evaluation result and the quality impact map, calculate the quality impact value of the target data on each associated data; perform quality evaluation on each associated data to obtain a quality evaluation result of each associated data; calculate a consistency index between the quality impact value and the quality evaluation result; based on the consistency index, correct the quality score in the first evaluation result to generate a second evaluation result including a corrected quality score and a quality impact analysis.

[0088] Optionally, the quality report generation module is also used to construct a multi-level data quality assessment matrix, which includes a first quality dimension of the target data and a second quality dimension of the associated data; map the first assessment result to the first quality dimension of the target data of the matrix, and map the second assessment result to the second quality dimension of the associated data of the matrix; calculate the comprehensive quality score of the first quality dimension and the second quality dimension, and determine the comprehensive quality grade based on the comprehensive quality score; generate a data quality report including the comprehensive quality score, comprehensive quality grade, first quality dimension and second quality dimension analysis.

[0089] Optionally, a data quality management system further includes a proposal generation module, which is used to determine key factors affecting data quality in each of the key sub-projects based on the data quality report; and generate a proposal for each of the key sub-projects based on the key factors.

[0090] It should be noted that: when the system provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0091] An embodiment of the present application also provides a computer storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing a data quality management method of the above embodiment. The specific execution process can be found in the specific description of the above embodiment, which will not be repeated here.

[0092] Please refer to Figure 3 The application also discloses an electronic device. Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0093] The communication bus 302 is used to realize the connection and communication between these components.

[0094] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0095] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0096] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0097] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a data quality management method.

[0098] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a data quality management method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0099] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0104] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0105] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A data quality management method, characterized in that: include: Obtain the target plan of the military-civilian integration project and identify the key sub-projects in the target plan; Collecting target data of each of the key sub-items and determining associated data of the target data; Verifying and evaluating each of the target data to obtain a first evaluation result; Evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data; A data quality report is generated based on the first evaluation result and the second evaluation result, and the data quality report is visually displayed.

2. The data quality management method according to claim 1, characterized in that: Determining the key sub-items in the target plan includes: Conduct hierarchical analysis on the target plan and establish a project hierarchical structure model; Based on the project hierarchy model, calculating the contribution of each of the sub-projects to the overall goal; The sub-items whose contribution is higher than the contribution threshold are regarded as key sub-items.

3. The data quality management method according to claim 1, characterized in that: The collecting of target data of each of the key sub-items and determining associated data of the target data includes: Establishing a data collection template for each key sub-project, wherein the data collection template includes data type, source, format and collection cycle; According to the data collection template, collecting target data at the data source interface of each of the key sub-projects; A data association map of a military-civilian integration project is constructed, and data associated with the target data is identified as associated data based on the data association map.

4. The data quality management method according to claim 1, characterized in that: The verifying and evaluating each of the target data to obtain a first evaluation result includes: Constructing quality assessment indicators for multi-dimensional data, the indicator system includes completeness, accuracy, consistency, timeliness and availability; Calculate the degree of membership of each quality assessment indicator based on a fuzzy comprehensive evaluation method of target data; Determine the weight of each quality assessment indicator by using the analytic hierarchy process; Calculating a quality score of the target data based on the membership degree and the weight; According to a preset quality grade classification standard, the quality score is mapped to a quality grade, and a first evaluation result including the quality score and the quality grade is generated.

5. The data quality management method according to claim 1, characterized in that: The evaluating the associated data based on the first evaluation result to obtain a second evaluation result of the target data includes: Construct a quality impact map of target data and associated data; Calculating the quality impact value of the target data on each associated data based on the first evaluation result and the quality impact map; Performing quality assessment on each associated data to obtain a quality assessment result of each associated data; Calculating a consistency index between the quality impact value and the quality assessment result; Based on the consistency index, the quality score in the first evaluation result is corrected to generate a second evaluation result including the corrected quality score and quality impact analysis.

6. The data quality management method according to claim 1, characterized in that: The generating a data quality report based on the first evaluation result and the second evaluation result includes: Constructing a multi-level data quality assessment matrix, wherein the data quality assessment matrix includes a first quality dimension of the target data and a second quality dimension of the associated data; Mapping the first evaluation result to a first quality dimension of target data of the matrix, and mapping the second evaluation result to a second quality dimension of associated data of the matrix; Calculating a comprehensive quality score of the first quality dimension and the second quality dimension, and determining a comprehensive quality grade based on the comprehensive quality score; Generate a data quality report containing comprehensive quality score, comprehensive quality grade, first quality dimension and second quality dimension analysis.

7. The data quality management method according to claim 1, characterized in that: The method further comprises: Determine the key factors affecting data quality in each of the key sub-projects based on the data quality report; Generate a proposal for each of the key sub-projects based on the key factors.

8. A data quality management system, characterized in that: The system comprises: A sub-project determination module is used to obtain the target plan of the military-civilian integration project and determine the key sub-projects in the target plan; A related data determination module, used for collecting target data of each of the key sub-items and determining related data of the target data; A first evaluation module, used to verify and evaluate each of the target data to obtain a first evaluation result; A second evaluation module, configured to evaluate the associated data based on the first evaluation result to obtain a second evaluation result of the target data; A quality report generation module is used to generate a data quality report based on the first evaluation result and the second evaluation result, and to visually display the data quality report.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

Images