Big model-based data quality evaluation result generation method and device, equipment and storage medium

Abstract

The application discloses a large model-based data quality evaluation result generation method and device, equipment and a storage medium, relates to the technical field of resource evaluation, and comprises the following steps: by fusing structured and unstructured data, performing deep semantic feature extraction by using a preset quality evaluation model, then realizing comprehensive and accurate evaluation of data quality by a feature screening and dynamic weight scoring mechanism, supporting defect diagnosis and optimization suggestion generation, and improving the accuracy and efficiency of data quality evaluation.

Description

Technical Field

[0001] This invention relates to the field of resource assessment technology, and in particular to a method, apparatus, device, and storage medium for generating data quality assessment results based on a large model. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, data has become a crucial foundational resource for enterprise operations. In many key industries such as finance, healthcare, government affairs, and transportation, data quality directly impacts the stability of business systems, the accuracy of analytical models, and the effectiveness of decision support.

[0003] Traditional data quality assessment methods primarily rely on manually defined rules or statistical approaches, making it difficult to capture semantic relationships between fields. This results in inaccurate assessments of dimensions such as consistency and timeliness. Furthermore, most existing methods focus on structured data tables and cannot effectively handle unstructured data such as text, logs, and documents, leading to biased assessment results that are ill-suited to the challenges of diverse data sources, complex structures, and frequent changes. Therefore, there is an urgent need for a data quality assessment method to improve the accuracy and efficiency of data quality assessment and to optimize it accordingly. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, device, and storage medium for generating data quality assessment results based on a large model, aiming to solve the technical problem of how to improve the accuracy and efficiency of data quality assessment.

[0005] To achieve the above objectives, this application proposes a method for generating data quality assessment results based on a large model, comprising:

[0006] Obtain the quality dataset to be evaluated, which includes structured data tables and unstructured text data;

[0007] The quality dataset is subjected to semantic feature extraction through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module.

[0008] The key quality features are filtered based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness.

[0009] The target quality characteristics are calculated based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data.

[0010] In one embodiment, the step of extracting key quality features from the quality dataset using a preset quality assessment model includes:

[0011] The quality data is vectorized through the input embedding layer of the preset quality assessment model to obtain dense vector features;

[0012] The dense vector features are weighted using a multi-head self-attention layer to generate attention features;

[0013] The attention features are input into a feedforward neural network layer for nonlinear transformation and feature enhancement to obtain deep semantic features.

[0014] The deep semantic features are dimensionality reduced by the feature dimensionality reduction module of the output layer to obtain key quality features.

[0015] In one embodiment, the step of filtering the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, wherein the dimension-feature mapping relationship library includes three quality dimensions of data quality: integrity, consistency, and timeliness, includes:

[0016] A feature importance evaluation model is established for each quality dimension, and the feature importance evaluation model is trained based on the random forest algorithm;

[0017] The key quality features are input into the feature importance evaluation model corresponding to each dimension to generate importance scores for the corresponding dimension.

[0018] The overall score is obtained by summing the weights of the importance scores.

[0019] When the overall score exceeds a preset score threshold, the corresponding quality feature among the key quality features is marked as a candidate feature;

[0020] Redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between the features is calculated;

[0021] The candidate features corresponding to the Pearson correlation coefficient are screened according to preset rules to obtain the target quality features.

[0022] In one embodiment, the step of calculating the target quality features based on a preset evaluation rule base to obtain a comprehensive quality score includes:

[0023] A three-dimensional weight space model is constructed, in which the three coordinate axes correspond to the dimensions of completeness, consistency and timeliness of quality data, respectively, and the benchmark weight vector of the model is dynamically adjusted based on the business scenario.

[0024] The target quality features are mapped to the three-dimensional weight space model using the feature projection method, generating the spatial distance between feature points and each benchmark weight vector;

[0025] The initial dimensional weights are generated based on the spatial distance.

[0026] The initial dimension weights are corrected using the entropy weighting method to obtain the corrected weight coefficients.

[0027] The modified weighting coefficients are multiplied by a matrix with the scoring function in the preset evaluation rule base to obtain a preliminary quality score.

[0028] The preliminary quality score is normalized using the Sigmoid activation function to generate a comprehensive quality score.

[0029] In one embodiment, the step of mapping the target quality features to the three-dimensional weight space model using the feature projection method to generate the spatial distance between feature points and each reference weight vector includes:

[0030] The optimal projection direction of the target quality feature in three-dimensional space is determined by principal component analysis.

[0031] The target quality features are mapped onto a three-dimensional space along the optimal projection direction to generate a feature point cloud distribution.

[0032] The Euclidean distance from each feature point to the three reference weight vectors is calculated based on the feature point cloud distribution, where each reference vector represents the ideal state of a quality dimension.

[0033] The Euclidean distance is corrected using the Mahalanobis distance algorithm to obtain the corrected distance;

[0034] The corrected distances are normalized to generate spatial distances between feature points and each baseline weight vector.

[0035] In one embodiment, after the step of calculating the target quality features based on a preset evaluation rule base to obtain a comprehensive quality score, the method further includes:

[0036] When the overall quality score is lower than the preset quality score threshold, the overall quality score is input into the preset quality defect diagnosis tree model, and a quality optimization report is output. The preset quality defect diagnosis tree model includes three diagnostic branches: integrity defects, consistency defects, and timeliness defects.

[0037] In one embodiment, the step of inputting the comprehensive quality score into a preset quality defect diagnosis tree model and outputting a quality optimization report when the comprehensive quality score is lower than a preset quality score threshold includes:

[0038] The comprehensive quality score is input into a preset quality defect diagnosis tree model for dimensional decomposition to identify the severity of defects in each quality dimension. The quality dimensions include the completeness, consistency, and timeliness of the quality data.

[0039] Based on the severity of the defect, an optimized solution is obtained by searching and matching from a preset repair strategy knowledge graph;

[0040] The optimization scheme is then converted into readable suggestion text using natural language generation technology.

[0041] The readable suggestion text is integrated using a multi-document summarization algorithm to generate an optimized suggestion framework;

[0042] The optimization suggestion framework is matched with the historical optimization case library to obtain case reference information. The historical optimization case library includes descriptions of quality defects and implemented optimization measures.

[0043] By combining the aforementioned optimization suggestion framework with case reference information, a quality optimization report is generated.

[0044] Furthermore, to achieve the above objectives, this application also proposes a data quality assessment result generation device based on a large model, the data quality assessment result generation device based on a large model comprising:

[0045] The acquisition module is used to acquire the quality dataset to be evaluated, which includes structured data tables and unstructured text data.

[0046] The processing module is used to extract semantic features from the quality dataset through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module.

[0047] The filtering module is used to filter the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness.

[0048] The results module is used to calculate the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data.

[0049] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data quality assessment result generation method based on a large model as described above.

[0050] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data quality assessment result generation method based on a large model as described above.

[0051] This application integrates structured and unstructured data, uses a pre-defined quality assessment model for deep semantic feature extraction, and then achieves a comprehensive and accurate assessment of data quality through feature filtering and dynamic weight scoring mechanisms. It also supports defect diagnosis and optimization suggestion generation, thereby improving the accuracy and efficiency of data quality assessment. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the first embodiment of the data quality assessment result generation method based on a large model in this application;

[0054] Figure 2 This is a block diagram of the preset quality assessment model structure of the first embodiment of the data quality assessment result generation method based on a large model in this application;

[0055] Figure 3 This is a flowchart illustrating the second embodiment of the data quality assessment result generation method based on a large model in this application;

[0056] Figure 4 This is a flowchart illustrating the third embodiment of the data quality assessment result generation method based on a large model in this application;

[0057] Figure 5 This is a schematic diagram of the module structure of the data quality assessment result generation device based on a large model, which is the first embodiment of the data quality assessment result generation method based on a large model in this application.

[0058] Figure 6 This is a schematic diagram of the hardware operating environment involved in the data quality assessment result generation method based on a large model in the embodiments of this application.

[0059] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0061] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Traditional data quality assessment methods primarily rely on manually set rules or statistically based scoring mechanisms, which struggle to cope with today's complex and ever-changing data environment. Especially when dealing with unstructured text data, traditional methods lack semantic understanding, leading to biased and highly erroneous assessment results. Furthermore, most existing systems only provide quality scores, lacking in-depth diagnosis of the causes of quality problems and actionable optimization suggestions, making it difficult to form an effective feedback loop. Therefore, this application proposes a data quality assessment result generation method based on a large model to address the aforementioned problems. The main solution of this application embodiment is as follows: First, obtain a quality dataset to be evaluated, which includes structured data tables and unstructured text data. Second, extract semantic features from the quality dataset using a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module. Third, filter the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness. Fourth, calculate the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score. The preset evaluation rule library includes scoring functions for evaluating the integrity, consistency, and timeliness of quality data.

[0063] Based on the above, this application also provides a method for generating data quality assessment results based on a large model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data quality assessment result generation method based on a large model according to this application. In this embodiment, the data quality assessment result generation method based on a large model includes steps S10 to S40:

[0064] Step S10: Obtain the quality dataset to be evaluated.

[0065] It should be noted that raw data is collected from multiple data sources and preliminarily cleaned and categorized to form a unified data evaluation input set. The quality dataset includes structured data tables and unstructured text data. Specifically, the quality dataset not only covers structured data tables from databases, data warehouses, or business systems (such as relational tables, field definitions, numerical or categorical attributes), but also unstructured text data from log files, user comments, document reports, web page content, and other sources. Structured data tables typically have clear definitions, including field names, data types, and constraint rules, making them suitable for calculating traditional data quality indicators. Unstructured text data, however, lacks fixed formats and semantic labels, making it difficult to effectively evaluate using traditional methods. Therefore, this application uses unified modeling to process both types of data, leveraging the powerful semantic understanding capabilities of large models to transform unstructured text into interpretable feature vectors. These vectors are then fused and analyzed with structured data in a unified semantic space, thereby achieving a comprehensive quality assessment of multi-source heterogeneous data.

[0066] Step S20: Extract semantic features from the quality dataset using a pre-defined quality assessment model to obtain key quality features.

[0067] It should be noted that, in this embodiment, the aforementioned preset quality assessment model is built based on the Transformer architecture, such as... Figure 2 The diagram shows the structure of the preset quality assessment model. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module.

[0068] Specifically, the input embedding layer transforms structured data tables and unstructured text data from the quality dataset into dense vector representations. This process not only preserves the original information of the data but also provides a unified format for subsequent feature extraction. Next, a multi-head self-attention layer processes these vectors. The multi-head mechanism allows simultaneous attention to different parts of the data, thus capturing semantic information more comprehensively. This mechanism is particularly suitable for processing unstructured text data because it can identify key information and contextual relationships within the text. Subsequently, a feedforward neural network layer further performs nonlinear transformations and enhancements on the extracted features. This layer, through its complex neural network structure, learns deep-level features in the data, further improving the model's ability to perceive data quality features. Finally, the output layer's feature dimensionality reduction module reduces the dimensionality of the extracted deep semantic features to obtain key quality features. This dimensionality reduction process not only reduces the feature dimensions and improves computational efficiency but also removes redundant information, making the final key quality features more focused and effective.

[0069] Through this multi-layered deep learning architecture, the pre-defined quality assessment model can extract high-quality key quality features from structured data tables and unstructured text data. These features not only contain semantic information about the data but also reflect the intrinsic quality and potential problems of the data.

[0070] Step S30: Filter key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features.

[0071] It's important to note that the dimension-feature mapping database encompasses three quality dimensions: completeness, consistency, and timeliness. Specifically, it covers these three core quality dimensions. Completeness focuses on data missing information, such as record integrity and field integrity. Consistency emphasizes whether data format and content conform to predefined rules and whether contradictory information exists. Timeliness considers data update frequency and timestamps to ensure data timeliness. These three dimensions comprehensively cover key aspects of data quality, providing clear guidance for feature selection.

[0072] Furthermore, step S30 also includes: establishing a feature importance evaluation model for each quality dimension. This model is trained using a random forest algorithm. During the screening process, a feature importance evaluation model is first established for each quality dimension. These models, trained using a random forest algorithm, can accurately evaluate the importance of each feature in its corresponding dimension. Let... For the set of key quality characteristics, For a set of quality dimensions, for each dimension Feature Importance Assessment Model Generate features Importance rating The key quality features are input into the feature importance assessment models corresponding to each dimension, generating importance scores for each dimension. These scores quantify the contribution of each feature across different quality dimensions, providing data support for subsequent screening. Next, the importance scores are weighted and summed to obtain a comprehensive score. Let... For dimension Weighting, overall score for:

[0073]

[0074] When the comprehensive score Exceeding the preset score threshold At that time, the corresponding quality characteristics in the key quality characteristics will be... These are marked as candidate features. This process considers not only the feature's performance in a single dimension but also its overall contribution across multiple dimensions, ensuring the comprehensiveness and representativeness of the candidate features. Next, redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between features is calculated. Specifically, let... For a candidate feature set, for each pair of features The Pearson correlation coefficient between the features is calculated as follows:

[0075]

[0076] in, It is a feature and characteristics covariance, Features standard deviation Features The standard deviation of . When (in When a preset relevance threshold is used, the characteristic is identified. and characteristics Redundancy exists. Redundancy detection is performed on candidate features by calculating the Pearson correlation coefficient between features. This detection eliminates highly correlated features, preventing the introduction of redundant information in subsequent evaluations. Finally, candidate features corresponding to the Pearson correlation coefficients are filtered according to preset rules to obtain target quality features. These target quality features cover the key dimensions of data quality while avoiding redundancy, accurately reflecting the actual data quality situation.

[0077] Step S40: Calculate the target quality characteristics based on the preset evaluation rule base to obtain the comprehensive quality score.

[0078] It should be noted that the pre-defined evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data. These scoring functions quantify and assess target quality characteristics based on different quality dimensions, thereby generating a comprehensive quality score. Specifically, the completeness scoring function primarily assesses data missingness, such as whether records are complete or fields are missing. The consistency scoring function focuses on evaluating whether the data format and content conform to predefined rules and whether there is contradictory information. The timeliness scoring function considers the data update frequency, timestamps, etc., to ensure the timeliness of the data. These scoring functions generate an independent score for each quality dimension by comprehensively considering multiple factors.

[0079] Furthermore, in the comprehensive score calculation process, firstly, a corresponding quality score is calculated based on the scoring function of each quality dimension. These scores reflect the quality status of the data across different dimensions. Then, these scores are integrated into a comprehensive quality score through weighted summation. The weight allocation can be adjusted according to specific application scenarios to reflect the importance of different dimensions in data quality assessment. This weighted summation method not only considers the independent contribution of each dimension but also achieves adaptability to different application scenarios through weight adjustment. Through comprehensive score calculation based on a preset evaluation rule base, a comprehensive and quantitative assessment of target quality characteristics can be achieved. The comprehensive quality score not only reflects the overall quality status of the data in the three dimensions of completeness, consistency, and timeliness but also provides a clear quantitative basis for data quality optimization.

[0080] Furthermore, following step S40, the process includes: when the overall quality score is lower than a preset quality score threshold, inputting the overall quality score into a preset quality defect diagnostic tree model and outputting a quality optimization report. The preset quality defect diagnostic tree model includes three diagnostic branches: integrity defects, consistency defects, and timeliness defects. Specifically, by introducing the preset quality defect diagnostic tree model, in-depth analysis of low-scoring data is conducted to generate targeted quality optimization reports, helping users quickly locate and resolve data quality issues. The preset quality defect diagnostic tree model is a decision tree-based diagnostic tool that categorizes data quality defects into three diagnostic branches: integrity defects, consistency defects, and timeliness defects. This classification comprehensively covers the main problem areas of data quality, ensuring the comprehensiveness and accuracy of the diagnosis. The aforementioned quality optimization report not only includes the existing data quality problems and defects but also provides specific optimization measures and implementation steps, as well as relevant case references. Through this report, users can quickly understand the data quality problems and take effective measures to optimize them, thereby improving the overall quality of the data.

[0081] This embodiment integrates structured and unstructured data, uses a preset quality assessment model for deep semantic feature extraction, and then achieves a comprehensive and accurate assessment of data quality through feature filtering and dynamic weight scoring mechanisms. It also supports defect diagnosis and optimization suggestion generation, thereby improving the accuracy and efficiency of data quality assessment.

[0082] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The method for generating data quality assessment results based on large models, step S40, further includes steps S201 to S206:

[0083] Step S201: Construct a three-dimensional weight space model.

[0084] It's important to note that the 3D weight space model is a vector-based evaluation model that maps three key dimensions of data quality into a three-dimensional space. The model's three axes correspond to the dimensions of data integrity, consistency, and timeliness, respectively. The value on each axis represents the quality level of the data in that dimension, with higher values ​​indicating better quality. The model's baseline weight vector is dynamically adjusted based on business scenarios, meaning it can adapt to different business needs and evaluation standards. For example, in the economic field, data accuracy may be more important than timeliness, therefore the consistency dimension in the baseline weight vector would have a higher weight. In the news media field, however, data timeliness may be more important, thus the timeliness dimension in the baseline weight vector would have a higher weight.

[0085] Step S202: The target quality features are mapped to a three-dimensional weight space model using the feature projection method to generate the spatial distance between the feature points and each benchmark weight vector.

[0086] It should be noted that feature projection is a method that projects high-dimensional feature vectors into a low-dimensional space, which can preserve the main information of the feature vectors in the low-dimensional space.

[0087] Furthermore, step S202 also includes: determining the optimal projection direction of the target quality feature in three-dimensional space using Principal Component Analysis (PCA). PCA is a commonly used dimensionality reduction technique that can map high-dimensional data to a low-dimensional space while preserving the main features of the data. In this technical solution, PCA is used to determine the optimal projection direction of the target quality feature in three-dimensional space. Through PCA, complex feature vectors can be simplified into several principal components, which can explain most of the variability of the data.

[0088] Next, the target quality features are mapped onto three-dimensional space along the optimal projection direction to generate a feature point cloud distribution. The position of each feature point in three-dimensional space reflects its performance in three dimensions: completeness, consistency, and timeliness. The feature point cloud distribution provides an intuitive geometric representation for subsequent distance calculations.

[0089] Then, based on the feature point cloud distribution, the Euclidean distance from each feature point to the three baseline weight vectors is calculated, where each baseline vector represents the ideal state of a quality dimension. For the target quality feature vector, Using the baseline weight vector, the feature projection method will... Projected to In the three-dimensional space, the Euclidean distance between the feature points and each reference weight vector is obtained. The Euclidean distance can be expressed as:

[0090]

[0091] in, The length of the vector represents the spatial distance. A smaller spatial distance indicates a closer proximity between the feature vector and the baseline weight vector, meaning better data quality in that dimension. The Euclidean distance is corrected using the Mahalanobis distance algorithm to obtain the corrected distance. Mahalanobis distance considers the covariance structure of the data and more accurately reflects the distance between feature points and the baseline weight vector. The specific formula is:

[0092]

[0093] in, Let be the covariance matrix of the feature point cloud. It is the inverse matrix, and finally the corrected distance is normalized to generate the spatial distance between the feature point and each benchmark weight vector.

[0094] Step S203: Calculate based on spatial distance to generate initial dimension weights.

[0095] It should be noted that initial weights for each dimension can be generated based on spatial distance. These initial dimension weights reflect the relative importance of each dimension in the overall evaluation. First, the spatial distance of each dimension is normalized to the [0, 1] interval for subsequent weight calculation. Then, the initial weight for each dimension is calculated based on the normalized spatial distance. A larger initial weight indicates a higher importance of that dimension in the overall evaluation.

[0096] Step S204: The initial dimension weights are corrected using the entropy weight method to obtain the corrected weight coefficients.

[0097] It's important to note that the entropy weighting method is a weighting method based on information entropy. It measures the dispersion of data by calculating the information entropy of each dimension. The smaller the information entropy, the greater the dispersion of the data, and the higher the importance of that dimension in the evaluation. Conversely, the larger the information entropy, the smaller the dispersion of the data, and the relatively lower the importance of that dimension.

[0098] Specifically, set Given a data matrix containing all target quality features, where each column represents a dimension of data, for the... Each dimension, its information entropy It can be calculated using the following formula:

[0099]

[0100]

[0101] in, It is a normalization factor that ensures the information entropy value is between [0, 1]. The ratio for each eigenvalue For the first The first dimension 1 eigenvalue, This represents the total number of eigenvalues. The corrected weight coefficients are calculated based on the obtained information entropy, using the following formula:

[0102]

[0103] in, The difference coefficient reflects the first... The coefficient of variation indicates the degree of dispersion of the data in each dimension. The larger the coefficient of variation, the greater the dispersion of the data in that dimension, and the higher its importance in the evaluation. It is the sum of the difference coefficients across all dimensions.

[0104] Step S205: Perform matrix multiplication on the corrected weight coefficients and the scoring function in the preset evaluation rule base to obtain a preliminary quality score.

[0105] It should be noted that the scoring functions in the pre-defined evaluation rule base quantify the quality of data by assessing key dimensions such as completeness, consistency, and timeliness. These scoring functions assign scores to target quality characteristics based on established standards, intuitively reflecting the quality status of the data across each dimension. The revised weighting coefficients are derived using scientific methods such as the entropy weighting method, objectively reflecting the actual contribution of each dimension to the overall data quality assessment, avoiding interference from subjective factors, and making the weight allocation more reasonable.

[0106] When the weighting coefficients are multiplied by the scoring function, the results are actually a weighted sum based on the scores of each dimension, according to the importance of each dimension. This process fully considers the differences in the impact of different dimensions on data quality, enabling the preliminary quality score to comprehensively and objectively reflect the overall quality level of the data.

[0107] Step S206: Normalize the preliminary quality score using the Sigmoid activation function to generate a comprehensive quality score.

[0108] It's important to note that the Sigmoid activation function is a commonly used non-linear function that maps input values ​​to the (0,1) interval. This property makes the Sigmoid function particularly suitable for normalization, as it can convert preliminary quality scores from different ranges into a unified, standardized score, facilitating comparisons across datasets and evaluation periods. The scores processed by the Sigmoid function can intuitively reflect the relative level of data quality and are also easy to integrate with other evaluation metrics.

[0109] This embodiment constructs a quality assessment space with integrity, consistency, and timeliness as coordinate axes, dynamically adjusts dimensional weights by combining feature projection and entropy weighting, and generates a normalized comprehensive quality score using a scoring function matrix and a Sigmoid function, thereby achieving intelligent quality assessment and business adaptation for multi-source heterogeneous data.

[0110] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 The data quality assessment result generation step S40, based on the large model, further includes steps S301 to S306:

[0111] Step S301: Input the comprehensive quality score into the preset quality defect diagnosis tree model for dimensional decomposition to identify the severity of defects in each quality dimension.

[0112] It's important to note that the quality dimensions include the completeness, consistency, and timeliness of the data. Specifically, during dimensional decomposition, the diagnostic tree model assesses the severity of defects based on the overall quality score across each dimension. For example, if the data scores low on the completeness dimension, the model will identify issues such as missing data or incomplete fields; on the consistency dimension, the model will check for data format errors and data conflicts; and on the timeliness dimension, the model will evaluate data update frequency and timestamp accuracy. Through this detailed dimensional analysis, the model can generate a comprehensive defect report, specifying the concrete problems and their severity for each dimension.

[0113] Step S302: Based on the severity of the defect, a search and matching operation is performed from the preset repair strategy knowledge graph to obtain an optimized solution.

[0114] It's important to note that the pre-defined remediation strategy knowledge graph is a comprehensive knowledge base containing various data quality defects and their corresponding remediation strategies. These strategies are based on industry best practices, expert experience, and historical data case studies, covering a range of methods from simple data cleaning to complex data reconstruction. The knowledge graph organizes this information in a structured manner, making the retrieval and matching process more efficient and accurate.

[0115] Specifically, once the severity of a defect is identified, the system searches the knowledge graph based on this information. The search process considers the type and severity of the defect, as well as the specific application scenario of the data. For example, for integrity defects, if the defect severity is high, the system may recommend strategies such as data filling or data reconstruction; for consistency defects, it may recommend strategies such as data format standardization or data conflict resolution; and for timeliness defects, it may recommend strategies such as optimizing the data update mechanism or timestamp calibration. After the search and matching are complete, a detailed optimization report is generated. This report not only includes specific remediation strategies but also provides the steps for implementing these strategies, expected results, and potential risk assessments. In this way, users can clearly understand how to optimize for identified defects, thereby improving data quality.

[0116] Step S303: The optimization scheme is converted into readable suggestion text using natural language generation technology.

[0117] It should be noted that Natural Language Generation (NLG) is a technique that converts structured data into natural language text. In this embodiment, NLG is used to convert optimization schemes retrieved from a pre-defined repair strategy knowledge graph into easily understandable suggestion text. These optimization schemes typically contain technical details and operational steps, and NLG can translate this content into plain language, enabling users without a technical background to easily understand and execute them.

[0118] Specifically, NLG technology can generate detailed recommendation texts that include not only the specific optimization steps but also background information and expected results. For example, if the optimization plan recommends data cleaning, NLG technology could generate the following recommendation text: "It is recommended to clean the data to remove duplicate records and correct formatting errors. This will improve data integrity and consistency, thereby enhancing data quality. Specific steps include: 1) Checking for and deleting duplicate records in the data; 2) Correcting the data format to ensure all fields conform to predefined formatting standards."

[0119] Furthermore, NLG technology can generate personalized suggestion text based on users' specific needs and backgrounds. For example, for different business scenarios or data types, the suggestion text can include specific considerations and optimization strategies. This personalized content generation not only improves the relevance of the suggestions but also enhances user acceptance and willingness to implement them.

[0120] Step S304: Integrate the readability suggestion text using a multi-document summarization algorithm to generate an optimized suggestion framework.

[0121] It should be noted that multi-document summarization (MLS) is a text processing technique that extracts key information from multiple documents and generates a concise summary. In this technical solution, this algorithm is used to integrate readable suggestion texts from different optimization schemes. These texts may contain repetitive information or suggestions from different perspectives. Through MLS, the most essential content can be extracted to generate a comprehensive optimization suggestion framework.

[0122] Specifically, first, the system collects all generated readable suggestion texts and preprocesses them, including removing duplicate content and correcting grammatical errors. This step ensures the quality and consistency of the input text. Next, a multi-document summarization algorithm analyzes the preprocessed text, extracting key information from each text. This key information typically includes specific optimization steps, expected results, and precautions. The algorithm integrates the extracted key information to generate a comprehensive optimization suggestion framework. This framework not only includes all important optimization suggestions but also optimizes the logical structure to make the content more coherent and easier to understand. For example, similar suggestions are grouped together, and steps are arranged according to the order of implementation. Finally, the system generates a clear and concise optimization suggestion framework. This framework is presented in an easy-to-read and easy-to-use format, helping users quickly understand and implement optimization measures.

[0123] Step S305: Perform similarity matching between the optimization suggestion framework and the historical optimization case library to obtain case reference information.

[0124] It should be noted that the historical optimization case library is a database that stores a large number of past data quality optimization cases. The historical optimization case library includes descriptions of quality defects and optimization measures that have been implemented. These cases are based on actual application scenarios and experience summaries and have reference value.

[0125] Specifically, first, key information, including defect types, optimization goals, and main measures, is extracted from the optimization suggestion framework. This information serves as the matching benchmark. Next, cases similar to the optimization suggestion framework are retrieved from the historical optimization case library. The retrieval process is based on a text similarity algorithm, comparing the key information in the optimization suggestion framework with the defect descriptions and optimization measures recorded in the case library. Then, the similarity between the retrieved cases and the optimization suggestion framework is evaluated, and the most closely matched case is selected as a reference. Cases with high similarity indicate that optimization measures have been successfully implemented in similar scenarios, and these measures have high reference value for current optimization work. Finally, detailed information is extracted from the matched cases, including specific optimization steps, precautions during implementation, problems encountered, and solutions. This information supplements the optimization suggestion framework, providing users with more comprehensive guidance.

[0126] Step S306: Combine the optimization suggestion framework with case reference information to output a quality optimization report.

[0127] It should be noted that the optimization recommendation framework provides optimization directions and measures based on the current data quality assessment results, while the case reference information extracts successful experiences and precautions from a historical optimization case library. Combining the two ensures that the optimization report not only has theoretical guidance but also practical feasibility and effectiveness.

[0128] This embodiment identifies the types and severity of data quality defects by constructing a diagnostic tree model, and combines it with knowledge graph matching and repair strategies. It also uses natural language generation and summarization technology to output structured optimization suggestions, and finally integrates historical case information to form a complete quality optimization report. This achieves closed-loop management of the entire process from quality assessment to problem diagnosis to optimization suggestions, thereby improving the intelligence and automation level of data quality problem handling.

[0129] Based on the first embodiment of this application, this application also provides a data quality assessment result generation device based on a large model. Please refer to... Figure 5 The device includes:

[0130] Module 10 is used to acquire the quality dataset to be evaluated, which includes structured data tables and unstructured text data.

[0131] Processing module 20 is used to extract semantic features from the quality dataset through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module.

[0132] The filtering module 30 is used to filter key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness.

[0133] The results module 40 is used to calculate the target quality characteristics based on the preset evaluation rule base to obtain the comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency and timeliness of quality data.

[0134] The data quality assessment result generation apparatus based on a large model provided in this application employs the data quality assessment result generation method based on a large model in the above embodiments, and can solve the technical problem of how to improve the accuracy and efficiency of data quality assessment. Compared with the prior art, the beneficial effects of the data quality assessment result generation apparatus based on a large model provided in this application are the same as the beneficial effects of the data quality assessment result generation method based on a large model provided in the above embodiments, and other technical features in the data quality assessment result generation apparatus based on a large model are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0135] In one embodiment, the processing module 20 is further configured to vectorize the quality data through the input embedding layer of the preset quality assessment model to obtain dense vector features; to assign weights to the dense vector features through a multi-head self-attention layer to generate attention features; to input the attention features into a feedforward neural network layer for nonlinear transformation and feature enhancement to obtain deep semantic features; and to perform dimensionality reduction processing on the deep semantic features through the feature dimensionality reduction module of the output layer to obtain key quality features.

[0136] In one embodiment, the screening module 30 is further configured to establish a feature importance assessment model for each quality dimension, the feature importance assessment model being trained based on a random forest algorithm; input key quality features into the feature importance assessment model corresponding to each dimension respectively to generate an importance score for the corresponding dimension; sum the weights based on the importance scores to obtain a comprehensive score; when the comprehensive score exceeds a preset score threshold, mark the corresponding quality feature among the key quality features as a candidate feature; perform redundancy detection on the candidate features and calculate the Pearson correlation coefficient between features; and screen the candidate features corresponding to the Pearson correlation coefficient according to preset rules to obtain the target quality feature.

[0137] In one embodiment, the result module 40 is further used to construct a three-dimensional weight space model. The three coordinate axes of the model correspond to the integrity, consistency, and timeliness dimensions of the quality data, respectively. The benchmark weight vector of the model is dynamically adjusted based on the business scenario. The target quality features are mapped to the three-dimensional weight space model using the feature projection method to generate the spatial distance between the feature points and each benchmark weight vector. The initial dimension weights are generated based on the spatial distances. The initial dimension weights are corrected using the entropy weight method to obtain the corrected weight coefficients. The corrected weight coefficients are multiplied by a matrix with the scoring function in the preset evaluation rule base to obtain a preliminary quality score. The preliminary quality score is normalized using the Sigmoid activation function to generate a comprehensive quality score.

[0138] In one embodiment, the result module 40 is further configured to determine the optimal projection direction of the target quality feature in three-dimensional space using principal component analysis; map the target quality feature along the optimal projection direction into three-dimensional space to generate a feature point cloud distribution; calculate the Euclidean distance from each feature point to three reference weight vectors based on the feature point cloud distribution, where each reference vector represents the ideal state of a quality dimension; correct the Euclidean distance using the Mahalanobis distance algorithm to obtain the corrected distance; and normalize the corrected distance to generate the spatial distance between the feature point and each reference weight vector.

[0139] In one embodiment, the result module 40 is further configured to input the comprehensive quality score into a preset quality defect diagnosis tree model and output a quality optimization report when the comprehensive quality score is lower than a preset quality score threshold. The preset quality defect diagnosis tree model includes three diagnostic branches: integrity defects, consistency defects, and timeliness defects.

[0140] In one embodiment, the result module 40 is further configured to: input the comprehensive quality score into a preset quality defect diagnosis tree model for dimensional decomposition; identify the severity of defects in each quality dimension, including the completeness, consistency, and timeliness of quality data; perform retrieval and matching from a preset repair strategy knowledge graph based on the severity of defects to obtain an optimization scheme; convert the optimization scheme into readable suggestion text using natural language generation technology; integrate the readable suggestion text using a multi-document summarization algorithm to generate an optimization suggestion framework; perform similarity matching between the optimization suggestion framework and a historical optimization case library to obtain case reference information, the historical optimization case library including quality defect descriptions and implemented optimization measures; and combine the optimization suggestion framework with the case reference information to output a quality optimization report.

[0141] This application provides a data quality assessment result generation device based on a large model. The data quality assessment result generation device based on a large model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data quality assessment result generation method based on a large model in the above embodiment 1.

[0142] The following is for reference. Figure 6This document illustrates a structural diagram of a data quality assessment result generation device based on a large model suitable for implementing embodiments of this application. The data quality assessment result generation device based on a large model in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The data quality assessment result generation device based on a large model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 6 As shown, the data quality assessment result generation device based on a large model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data quality assessment result generation device based on the large model. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the large-model-based data quality assessment result generation device to wirelessly or wiredly communicate with other devices to exchange data. Although various large-model-based data quality assessment result generation devices are shown in the figures, it should be understood that implementation or possession of all of them is not required. More or fewer may be implemented alternatively.

[0144] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0145] The data quality assessment result generation device based on a large model provided in this application, employing the data quality assessment result generation method based on a large model in the above embodiments, can solve the technical problem of how to improve the accuracy and efficiency of data quality assessment. Compared with the prior art, the beneficial effects of the data quality assessment result generation device based on a large model provided in this application are the same as the beneficial effects of the data quality assessment result generation method based on a large model provided in the above embodiments, and other technical features in this data quality assessment result generation device based on a large model are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0148] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data quality assessment result generation method based on a large model in the above embodiments.

[0149] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible storage medium containing or storing a program that can be executed by instructions, used by a device, or used in conjunction with it. The program code contained on the computer-readable storage medium may be transmitted using any suitable storage medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0150] The aforementioned computer-readable storage medium may be included in a device for generating data quality assessment results based on a large model; or it may exist independently and not be assembled into a device for generating data quality assessment results based on a large model.

[0151] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a large-model-based data quality assessment result generation device, enable the device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0154] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described method for generating data quality assessment results based on a large model. This addresses the technical problem of improving the accuracy and efficiency of data quality assessment. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data quality assessment result generation method based on a large model provided in the above embodiments, and will not be elaborated upon here.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for generating data quality assessment results based on a large model.

[0156] The computer program product provided in this application can solve the technical problem of how to improve the accuracy and efficiency of data quality assessment. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data quality assessment result generation method based on large models provided in the above embodiments, and will not be repeated here.

[0157] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for generating data quality assessment results based on a large model, characterized in that, The method includes: Obtain the quality dataset to be evaluated, which includes structured data tables and unstructured text data; The quality dataset is subjected to semantic feature extraction through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module. The key quality features are filtered based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness. The target quality characteristics are calculated based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data. The step of filtering the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, wherein the dimension-feature mapping relationship library includes three quality dimensions of data quality: integrity, consistency, and timeliness, includes: A feature importance evaluation model is established for each quality dimension, and the feature importance evaluation model is trained based on the random forest algorithm; The key quality features are input into the feature importance evaluation model corresponding to each dimension to generate importance scores for the corresponding dimension. The overall score is obtained by summing the weights of the importance scores. When the overall score exceeds a preset score threshold, the corresponding quality feature among the key quality features is marked as a candidate feature; Redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between the features is calculated; The candidate features corresponding to the Pearson correlation coefficient are screened according to preset rules to obtain the target quality features; The step of calculating the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score includes: A three-dimensional weight space model is constructed, in which the three coordinate axes correspond to the dimensions of completeness, consistency and timeliness of quality data, respectively, and the benchmark weight vector of the model is dynamically adjusted based on the business scenario. The target quality features are mapped to the three-dimensional weight space model using the feature projection method, generating the spatial distance between feature points and each benchmark weight vector; The initial dimensional weights are generated based on the spatial distance. The initial dimension weights are corrected using the entropy weighting method to obtain the corrected weight coefficients. The modified weighting coefficients are multiplied by a matrix with the scoring function in the preset evaluation rule base to obtain a preliminary quality score. The preliminary quality score is normalized using the Sigmoid activation function to generate a comprehensive quality score.

2. The method as described in claim 1, characterized in that, The step of extracting key quality features from the quality dataset using a preset quality assessment model includes: The quality data is vectorized through the input embedding layer of the preset quality assessment model to obtain dense vector features; The dense vector features are weighted using a multi-head self-attention layer to generate attention features; The attention features are input into a feedforward neural network layer for nonlinear transformation and feature enhancement to obtain deep semantic features. The deep semantic features are dimensionality reduced by the feature dimensionality reduction module of the output layer to obtain key quality features.

3. The method as described in claim 1, characterized in that, The step of mapping the target quality features to the three-dimensional weight space model using the feature projection method to generate the spatial distance between feature points and each reference weight vector includes: The optimal projection direction of the target quality feature in three-dimensional space is determined by principal component analysis. The target quality features are mapped onto a three-dimensional space along the optimal projection direction to generate a feature point cloud distribution. The Euclidean distance from each feature point to the three reference weight vectors is calculated based on the feature point cloud distribution, where each reference vector represents the ideal state of a quality dimension. The Euclidean distance is corrected using the Mahalanobis distance algorithm to obtain the corrected distance; The corrected distances are normalized to generate spatial distances between feature points and each baseline weight vector.

4. The method as described in claim 1, characterized in that, After the step of calculating the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score, the method further includes: When the overall quality score is lower than the preset quality score threshold, the overall quality score is input into the preset quality defect diagnosis tree model, and a quality optimization report is output. The preset quality defect diagnosis tree model includes three diagnostic branches: integrity defects, consistency defects, and timeliness defects.

5. The method as described in claim 4, characterized in that, The step of inputting the comprehensive quality score into a preset quality defect diagnosis tree model and outputting a quality optimization report when the comprehensive quality score is lower than a preset quality score threshold includes: The comprehensive quality score is input into a preset quality defect diagnosis tree model for dimensional decomposition to identify the severity of defects in each quality dimension. The quality dimensions include the completeness, consistency, and timeliness of the quality data. Based on the severity of the defect, an optimized solution is obtained by searching and matching from a preset repair strategy knowledge graph; The optimization scheme is then converted into readable suggestion text using natural language generation technology. The readable suggestion text is integrated using a multi-document summarization algorithm to generate an optimized suggestion framework; The optimization suggestion framework is matched with the historical optimization case library to obtain case reference information. The historical optimization case library includes descriptions of quality defects and implemented optimization measures. By combining the aforementioned optimization suggestion framework with case reference information, a quality optimization report is generated.

6. A device for generating data quality assessment results based on a large model, characterized in that, The device includes: The acquisition module is used to acquire the quality dataset to be evaluated, which includes structured data tables and unstructured text data. The processing module is used to extract semantic features from the quality dataset through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module. A filtering module is used to filter the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness. It is also used to establish a feature importance evaluation model for each quality dimension, which is trained based on a random forest algorithm. The key quality features are input into the feature importance evaluation models corresponding to each dimension to generate importance scores for the corresponding dimensions. Weights are summed based on the importance scores to obtain a comprehensive score. When the comprehensive score exceeds a preset score threshold, the corresponding quality features among the key quality features are marked as candidate features. Redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between features is calculated. The candidate features corresponding to the Pearson correlation coefficient are filtered according to preset rules to obtain the target quality features. The results module is used to calculate the target quality features based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data. It is also used to construct a three-dimensional weight space model, where the three coordinate axes correspond to the dimensions of completeness, consistency, and timeliness of the quality data, respectively. The baseline weight vector of the model is dynamically adjusted based on the business scenario. The target quality features are mapped to the three-dimensional weight space model using a feature projection method to generate spatial distances between feature points and each baseline weight vector. Initial dimension weights are generated based on these spatial distances. The initial dimension weights are corrected using an entropy weight method to obtain corrected weight coefficients. The corrected weight coefficients are multiplied by a matrix multiplication operation with the scoring functions in the preset evaluation rule base to obtain a preliminary quality score. The preliminary quality score is normalized using a sigmoid activation function to generate a comprehensive quality score.

7. A device for generating data quality assessment results based on a large model, characterized in that, The device includes: a memory, a processor, and a large-model-based data quality assessment result generation program stored on the memory and running on the processor, the large-model-based data quality assessment result generation program being configured to implement the steps of the large-model-based data quality assessment result generation method as described in any one of claims 1-5.

8. A storage medium, characterized in that, The storage medium stores a data quality assessment result generation program based on a large model, which, when executed by a processor, implements the steps of the data quality assessment result generation method based on a large model as described in any one of claims 1-5.

Images