Large-model-based data quality evaluation result generation method and device, equipment and storage medium

Abstract

The invention discloses a data quality evaluation result generation method and device based on a large model, equipment and a storage medium, and relates to the technical field of resource evaluation, and the method comprises the steps: fusing structured and unstructured data, carrying out the deep semantic feature extraction through a preset quality evaluation model, and then carrying out the deep semantic feature extraction through a feature screening and dynamic weight scoring mechanism; according to the invention, comprehensive and accurate evaluation of data quality is realized, defect diagnosis and optimization suggestion generation are supported, and the accuracy and efficiency of data quality evaluation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource evaluation, and in particular to a data quality evaluation result generation method and device based on a large model, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of big data and artificial intelligence technology, data has become an important basic resource for enterprise operation. In many key industries such as finance, medical treatment, government affairs and transportation, data quality directly affects the stability of business systems, the accuracy of analysis models and the effectiveness of decision support.

[0003] Traditional data quality evaluation methods mainly rely on manual rule setting or statistical methods, which are difficult to capture semantic associations between fields, resulting in inaccurate evaluation of consistency, timeliness and other dimensions. At the same time, most existing methods focus on structured data tables and cannot effectively process unstructured data such as text, logs and documents, resulting in one-sided evaluation results and difficulty in adapting to the current challenges of diverse data sources, complex structures and frequent changes. Therefore, there is an urgent need for a data quality evaluation method to improve the accuracy and efficiency of data quality evaluation and targeted optimization. SUMMARY

[0004] The main purpose of the present application is to provide a data quality evaluation result generation method and device based on a large model, equipment and a storage medium, aiming to solve the technical problem of how to improve the accuracy and efficiency of data quality evaluation.

[0005] To achieve the above-mentioned purpose, the present application provides a data quality evaluation result generation method based on a large model, comprising: Obtaining a quality data set to be evaluated, the quality data set comprising structured data tables and unstructured text data; Performing semantic feature extraction on the quality data set through a preset quality evaluation model to obtain key quality features, the preset quality evaluation model comprising an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer and an output layer, the output layer comprising a feature dimension reduction module; Filtering the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, the dimension-feature mapping relationship library comprising three quality dimensions of data quality integrity, consistency and timeliness; Calculating the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score value, the preset evaluation rule library comprising a scoring function for evaluating the integrity, consistency and timeliness of quality data.

[0006] In an embodiment, the step of performing semantic feature extraction on the quality data set by a preset quality evaluation model to obtain key quality features comprises: vectorizing the quality data by an input embedding layer of the preset quality evaluation model to obtain dense vector features; performing weight distribution on the dense vector features by a multi-head self-attention layer to generate attention features; inputting the attention features into a feedforward neural network layer to perform nonlinear transformation and feature enhancement, thereby obtaining deep semantic features; performing dimension reduction on the deep semantic features by a feature dimension reduction module of an output layer to obtain key quality features.

[0007] In an embodiment, the step of screening the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features comprises: establishing a feature importance evaluation model for each quality dimension, wherein the feature importance evaluation model is obtained based on a random forest algorithm; inputting the key quality features into the feature importance evaluation model corresponding to each dimension to generate importance scores under the corresponding dimension; performing weight summation according to the importance scores to obtain a comprehensive score; when the comprehensive score exceeds a preset score threshold, marking the corresponding quality feature in the key quality features as a candidate feature; performing redundancy detection on the candidate features to calculate Pearson correlation coefficients between the features; screening the candidate features corresponding to the Pearson correlation coefficients according to a preset rule to obtain target quality features.

[0008] In an embodiment, the step of calculating the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score value comprises: constructing a three-dimensional weight space model, wherein three coordinate axes of the model correspond to the integrity, consistency and timeliness dimensions of the quality data, and a reference weight vector of the model is dynamically adjusted based on a business scenario; mapping the target quality features to the three-dimensional weight space model by a feature projection method to generate spatial distances between feature points and reference weight vectors; calculating based on the spatial distances to generate initial dimension weights; performing weight correction on the initial dimension weights by an entropy weight method to obtain corrected weight coefficients; The modified weight coefficient is subjected to matrix multiplication operation with a scoring function in a preset evaluation rule library to obtain a preliminary quality score; The preliminary quality score is normalized by a Sigmoid activation function to generate a comprehensive quality score value.

[0009] In an embodiment, the step of mapping the target quality feature to the three-dimensional weight space model by the feature projection method to generate spatial distances of feature points from reference weight vectors includes: An optimal projection direction of the target quality feature in three-dimensional space is determined by principal component analysis; The target quality feature is mapped to three-dimensional space along the optimal projection direction to generate a feature point cloud distribution; Based on the feature point cloud distribution, the Euclidean distance of each feature point from three reference weight vectors is calculated, wherein each reference vector represents an ideal state of a quality dimension; The Euclidean distance is corrected by Mahalanobis distance algorithm to obtain a corrected distance; The corrected distance is normalized to generate spatial distances of feature points from reference weight vectors.

[0010] In an embodiment, after the step of calculating the target quality feature based on the preset evaluation rule library to obtain a comprehensive quality score value, the method further includes: When the comprehensive quality score value is lower than a preset quality score threshold, the comprehensive quality score value is input into a preset quality defect diagnosis tree model to output a quality optimization report, the preset quality defect diagnosis tree model including three types of diagnosis branches of integrity defect, consistency defect and timeliness defect.

[0011] In an embodiment, the step of inputting the comprehensive quality score value into the preset quality defect diagnosis tree model to output a quality optimization report when the comprehensive quality score value is lower than a preset quality score threshold includes: The comprehensive quality score value is subjected to dimension decomposition in the preset quality defect diagnosis tree model to identify defect severity of each quality dimension, the quality dimension including integrity, consistency and timeliness of quality data; Based on the defect severity, an optimization scheme is obtained by searching and matching from a preset repair strategy knowledge graph; The optimization scheme is converted into a readability suggestion text by natural language generation technology; The readability suggestion text is integrated by a multi-document summarization algorithm to generate an optimization suggestion framework; The optimization suggestion framework is matched with a historical optimization case library in terms of similarity to obtain case reference information, the historical optimization case library including quality defect descriptions and implemented optimization measures; The optimization suggestion framework is combined with the case reference information to output a quality optimization report.

[0012] In addition, to achieve the above object, the present application also proposes a data quality evaluation result generation device based on a large model, which comprises: An acquisition module is configured to acquire a quality data set to be evaluated, the quality data set comprising a structured data table and unstructured text data; A processing module is configured to perform semantic feature extraction on the quality data set through a preset quality evaluation model to obtain key quality features, the preset quality evaluation model comprising an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer, the output layer comprising a feature dimension reduction module; A screening module is configured to screen the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, the dimension-feature mapping relationship library comprising three quality dimensions of data quality, namely, integrity, consistency, and timeliness; A result module is configured to calculate the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score value, the preset evaluation rule library comprising score functions for evaluating the integrity, consistency, and timeliness of the quality data.

[0013] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, the computer program being executed by a processor to implement the steps of the data quality evaluation result generation method based on a large model as described above.

[0014] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a computer program, the computer program being executed by a processor to implement the steps of the data quality evaluation result generation method based on a large model as described above.

[0015] The present application fuses structured and unstructured data, performs deep semantic feature extraction using a preset quality evaluation model, and then realizes comprehensive and accurate evaluation of data quality through feature screening and a dynamic weight scoring mechanism, supports defect diagnosis and optimization suggestion generation, and improves the accuracy and efficiency of data quality evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can also be obtained based on these drawings without any creative effort.

[0017] Figure 1 A flowchart of the first embodiment of the method for generating a data quality evaluation result based on a large model according to the present application; Figure 2 A pre-set quality evaluation model structure block diagram of the first embodiment of the method for generating a data quality evaluation result based on a large model according to the present application; Figure 3 A flowchart of the second embodiment of the method for generating a data quality evaluation result based on a large model according to the present application; Figure 4 A flowchart of the third embodiment of the method for generating a data quality evaluation result based on a large model according to the present application; Figure 5 A module structure diagram of the data quality evaluation result generation device based on a large model of the first embodiment of the method for generating a data quality evaluation result based on a large model according to the present application; Figure 6 A device structure diagram of the hardware running environment involved in the method for generating a data quality evaluation result based on a large model according to the embodiments of the present application.

[0018] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0020] In order to better understand the technical solutions of the present application, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0021] Traditional data quality evaluation methods mainly rely on manual rule setting or scoring mechanisms based on statistical indicators, which are difficult to cope with the current complex and changing data environment. Especially when facing unstructured text data, traditional methods lack semantic understanding ability, resulting in one-sided evaluation results and large errors. At the same time, existing systems mostly only provide quality scores, lack in-depth diagnosis of the causes of quality problems and executable optimization suggestions, and are difficult to form an effective feedback loop. Therefore, the present application proposes a data quality evaluation result generation method based on a large model to solve the above problems. The main solution of the embodiment of the present application is: obtaining a quality data set to be evaluated, the quality data set including structured data tables and unstructured text data; performing semantic feature extraction on the quality data set through a preset quality evaluation model to obtain key quality features, the preset quality evaluation model including an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer, the output layer including a feature dimension reduction module; filtering the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, the dimension-feature mapping relationship library including three quality dimensions of data quality, namely integrity, consistency, and timeliness; calculating the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score value, the preset evaluation rule library including scoring functions for evaluating the integrity, consistency, and timeliness of the quality data.

[0022] Based on the above, the embodiment of the present application also provides a data quality evaluation result generation method based on a large model, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the data quality evaluation result generation method based on a large model of the present application. In this embodiment, the data quality evaluation result generation method based on a large model includes steps S10-S40: Step S10, obtaining a quality data set to be evaluated.

[0023] It should be noted that the original data is collected from multiple data sources and is preliminarily cleaned and classified in format to form a unified data evaluation input set. The quality data set includes structured data tables and unstructured text data. Specifically, the quality data set not only covers structured data tables such as relational tables, field definitions, numerical or categorical attributes, etc. from databases, data warehouses or business systems, but also includes unstructured text data from log files, user comments, document reports, web page contents and other channels. Structured data tables usually have a clear definition, including field name, data type, constraint rule and other information, which is suitable for the calculation of traditional data quality indicators; while unstructured text data lacks fixed format and semantic tags, and it is difficult to effectively evaluate by traditional methods. Therefore, in this application, by uniformly modeling the two types of data, the powerful semantic understanding ability of the large model is used to convert the unstructured text into an interpretable feature vector, and the structured data is fused and analyzed in a unified semantic space, thereby realizing comprehensive quality evaluation of multi-source heterogeneous data.

[0024] In step S20, the quality data set is subjected to semantic feature extraction by a preset quality evaluation model to obtain key quality features.

[0025] It should be noted that in the present embodiment, the above-mentioned preset quality evaluation model is constructed based on a Transformer architecture, as shown in the preset quality evaluation model structure block diagram Figure 2 The preset quality evaluation model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer, and the output layer includes a feature dimension reduction module.

[0026] Specifically, the input embedding layer converts the structured data tables and unstructured text data in the quality data set into dense vector representations. This process not only preserves the original information of the data, but also provides a unified format for subsequent feature extraction. Then, the multi-head self-attention layer processes these vectors, and through the multi-head mechanism, it can simultaneously focus on different parts of the data, thereby more comprehensively capturing the semantic information in the data. This mechanism is particularly suitable for processing unstructured text data, as it can identify key information and contextual relationships in the text. Subsequently, the feedforward neural network layer further nonlinearly transforms and enhances the extracted features. This layer, through a complex neural network structure, can learn deep features in the data, further improving the model's ability to perceive data quality features. Finally, the feature dimension reduction module of the output layer processes the extracted deep semantic features to obtain key quality features. The dimension reduction process not only reduces the dimension of the features, improves the calculation efficiency, but also removes redundant information, so that the final key quality features are more concentrated and effective.

[0027] Through this multi-layer deep learning architecture, the preset quality assessment model can extract high-quality key quality features from structured data tables and unstructured text data. These features not only contain semantic information of the data, but also reflect the inherent quality and potential problems of the data.

[0028] In step S30, the key quality features are filtered based on the preset dimension-feature mapping relationship library to obtain target quality features.

[0029] It should be noted that the dimension-feature mapping relationship library includes three quality dimensions of data quality, namely integrity, consistency and timeliness. Specifically, the dimension-feature mapping relationship library covers the three core quality dimensions of data quality, namely integrity, consistency and timeliness. The integrity dimension focuses on the missing situation of data, such as whether the record is complete and whether the field is missing. The consistency dimension focuses on whether the format and content of the data meet the predefined rules and whether there is contradictory information. The timeliness dimension considers the update frequency and timestamp of the data to ensure the timeliness of the data. These three dimensions comprehensively cover the key aspects of data quality and provide clear guidance for feature screening.

[0030] Further, step S30 further includes: establishing a feature importance evaluation model for each quality dimension, and the feature importance evaluation model is trained based on a random forest algorithm. In the screening process, a feature importance evaluation model is first established for each quality dimension. These models are trained based on a random forest algorithm and can accurately evaluate the importance of each feature in the corresponding dimension. Let be a set of key quality features, be a set of quality dimensions, for each dimension , the feature importance evaluation model generates the importance score of the feature . The key quality features are input into the feature importance evaluation model corresponding to each dimension to generate the importance score in the corresponding dimension. Through these scores, the contribution of each feature in different quality dimensions can be quantified, thereby providing data support for subsequent screening. Next, the importance scores are summed according to the weights to obtain a comprehensive score. Let be the weight of dimension , and the comprehensive score is: When the comprehensive score exceeds a preset score threshold , the corresponding quality feature The features are marked as candidate features. This process not only considers the performance of the features in a single dimension, but also integrates the overall contribution of the features in multiple dimensions to ensure the comprehensiveness and representativeness of the candidate features. Then, redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between the features is calculated. Specifically, for a candidate feature set, the Pearson correlation coefficient between each pair of features is calculated as follows: wherein, is the covariance of feature and feature , is the standard deviation of feature , is the standard deviation of feature . When (wherein is a preset correlation threshold), it is determined that feature and feature are redundant. The Pearson correlation coefficient between the features is calculated through the redundancy detection of the candidate features. Through this detection, highly correlated features can be removed to avoid introducing redundant information in subsequent evaluation. Finally, the candidate features corresponding to the Pearson correlation coefficient are screened according to a preset rule to obtain target quality features. These target quality features not only cover the key dimensions of data quality, but also avoid redundancy, and can accurately reflect the actual situation of data quality.

[0031] Step S40, based on a preset evaluation rule library, the target quality features are calculated to obtain a comprehensive quality score value.

[0032] It should be noted that the preset evaluation rule library includes score functions for evaluating the integrity, consistency, and timeliness of the quality data. These score functions quantitatively evaluate the target quality features according to different quality dimensions to generate a comprehensive quality score value. Specifically, the integrity score function mainly evaluates the missing situation of the data, such as whether the record is complete and whether the field is missing. The consistency score function focuses on evaluating whether the format and content of the data meet the predefined rules and whether there is contradictory information. The timeliness score function considers the update frequency and timestamp of the data to ensure the timeliness of the data. These score functions generate an independent score value for each quality dimension by comprehensively considering multiple factors.

[0033] ​​Further, in the comprehensive score calculation process, first, the corresponding quality score is calculated according to the score function of each quality dimension. These score values reflect the quality status of the data in different dimensions. Then, through weighted summation, these score values are integrated into a comprehensive quality score value. The allocation of weights can be adjusted according to specific application scenarios to reflect the importance of different dimensions in data quality evaluation. This weighted summation method not only considers the independent contribution of each dimension, but also realizes the adaptability to different application scenarios through weight adjustment. Through comprehensive score calculation based on the preset evaluation rule library, the target quality characteristics can be comprehensively and quantitatively evaluated. The comprehensive quality score value not only reflects the overall quality status of the data in the integrity, consistency and timeliness dimensions, but also provides a clear quantitative basis for data quality optimization.

[0034] Further, after step S40, it further includes: when the comprehensive quality score value is lower than the preset quality score threshold, inputting the comprehensive quality score value into the preset quality defect diagnosis tree model, and outputting a quality optimization report, the preset quality defect diagnosis tree model including three types of diagnosis branches of integrity defect, consistency defect and timeliness defect. Specifically, by introducing the preset quality defect diagnosis tree model, the low-score data is analyzed in depth to generate a targeted quality optimization report, helping users quickly locate and solve data quality problems. The preset quality defect diagnosis tree model is a diagnosis tool based on decision tree, which divides data quality defects into three types of diagnosis branches of integrity defect, consistency defect and timeliness defect. This classification method comprehensively covers the main problem areas of data quality, ensuring the comprehensiveness and accuracy of diagnosis. The above quality optimization report not only contains the problems and defects of data quality, but also provides specific optimization measures and implementation steps, as well as relevant case references. Through this report, users can quickly understand the problems of data quality and take effective measures to optimize the overall quality of data.

[0035] The embodiment fuses structured and unstructured data, uses a preset quality evaluation model to extract deep semantic features, then realizes comprehensive and accurate evaluation of data quality through feature screening and dynamic weight scoring mechanism, and supports defect diagnosis and optimization suggestion generation, improving the accuracy and efficiency of data quality evaluation.

[0036] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 3 , the method for generating data quality evaluation result based on large model step S40 further includes steps S201-S206: Step S201, a three-dimensional weight space model is constructed. Step S201, a three-dimensional weight space model is constructed.

[0037] It's important to note that the 3D weight space model is a vector-based evaluation model that maps three key dimensions of data quality into a three-dimensional space. The model's three axes correspond to the dimensions of data integrity, consistency, and timeliness, respectively. The value on each axis represents the quality level of the data in that dimension, with higher values ​​indicating better quality. The model's baseline weight vector is dynamically adjusted based on business scenarios, meaning it can adapt to different business needs and evaluation standards. For example, in the economic field, data accuracy may be more important than timeliness, therefore the consistency dimension in the baseline weight vector would have a higher weight. In the news media field, however, data timeliness may be more important, thus the timeliness dimension in the baseline weight vector would have a higher weight.

[0038] Step S202: The target quality features are mapped to a three-dimensional weight space model using the feature projection method to generate the spatial distance between the feature points and each benchmark weight vector.

[0039] It should be noted that feature projection is a method that projects high-dimensional feature vectors into a low-dimensional space, which can preserve the main information of the feature vectors in the low-dimensional space.

[0040] Furthermore, step S202 also includes: determining the optimal projection direction of the target quality feature in three-dimensional space using Principal Component Analysis (PCA). PCA is a commonly used dimensionality reduction technique that can map high-dimensional data to a low-dimensional space while preserving the main features of the data. In this technical solution, PCA is used to determine the optimal projection direction of the target quality feature in three-dimensional space. Through PCA, complex feature vectors can be simplified into several principal components, which can explain most of the variability of the data.

[0041] Next, the target quality features are mapped onto 3D space along the optimal projection direction to generate a feature point cloud distribution. The position of each feature point in 3D space reflects its performance in three dimensions: completeness, consistency, and timeliness. The feature point cloud distribution provides an intuitive geometric representation for subsequent distance calculations.

[0042] Then, based on the feature point cloud distribution, the Euclidean distance from each feature point to the three baseline weight vectors is calculated, where each baseline vector represents the ideal state of a quality dimension. For the target quality feature vector, Using the baseline weight vector, the feature projection method will... Projected to In the three-dimensional space, the Euclidean distance between the feature points and each reference weight vector is obtained. The Euclidean distance can be expressed as: wherein, represents the length of the vector, the smaller the spatial distance, the closer the feature vector to the reference weight vector, that is, the better the quality of the data in this dimension. The Euclidean distance is corrected by the Mahalanobis distance algorithm to obtain the corrected distance. The Mahalanobis distance considers the covariance structure of the data and can more accurately reflect the distance between the feature point and the reference weight vector. The specific formula is: wherein, is the covariance matrix of the feature point cloud, is the inverse matrix thereof, and finally the corrected distance is normalized to generate the spatial distance of the feature point to each reference weight vector.

[0043] Step S203, based on the spatial distance, an initial dimension weight is generated.

[0044] It should be noted that based on the spatial distance, the initial weight of each dimension can be generated. The initial dimension weight reflects the relative importance of each dimension in the comprehensive evaluation. First, the spatial distance of each dimension is normalized to the interval [0, 1] so as to facilitate the subsequent weight calculation. The initial weight of each dimension is calculated based on the normalized spatial distance. The greater the initial weight, the higher the importance of the dimension in the comprehensive evaluation.

[0045] Step S204, the initial dimension weight is corrected by the entropy weight method to obtain the corrected weight coefficient.

[0046] It should be noted that the entropy weight method is a weight determination method based on information entropy, which measures the discrete degree of data by calculating the information entropy of each dimension. The smaller the information entropy, the greater the discrete degree of data, and the higher the importance of the dimension in the evaluation. On the contrary, the greater the information entropy, the smaller the discrete degree of data, and the relatively lower the importance of the dimension.

[0047] Specifically, let be a data matrix containing all target quality characteristics, wherein each column represents the data of one dimension. For the i-th dimension, its information entropy can be calculated by the following formula: wherein, is a normalization factor to ensure that the value of the information entropy is between 0 and 1, is the ratio of each feature value, is the i-th feature value of the i-th dimension, is the i-th feature value of the i-th dimension, is the i-th feature value of the i-th dimension, ​is the total number of feature values. According to the calculated information entropy, the modified weight coefficient is obtained, and the specific formula is: wherein, is the difference coefficient, which reflects the dispersion degree of the first dimension, the greater the difference coefficient, the greater the dispersion degree of the data in this dimension, and the higher the importance of the dimension in the evaluation, is the sum of the difference coefficients of all dimensions.

[0048] In step S205, the modified weight coefficient is multiplied by the scoring function in the preset evaluation rule library to obtain a preliminary quality score.

[0049] It should be noted that the scoring function in the preset evaluation rule library quantitatively evaluates the quality of the data according to the integrity, consistency and timeliness of the data quality. These scoring functions score the target quality characteristics according to the established standards, and intuitively reflect the quality of the data in each dimension. The modified weight coefficient is obtained based on the entropy weight method, which objectively reflects the actual contribution of each dimension in the overall data quality evaluation, avoids subjective interference, and makes the weight distribution more reasonable.

[0050] When the weight coefficient is multiplied by the scoring function, it is actually a weighted sum based on the scores of each dimension according to the importance of each dimension. This process fully considers the differences in the impact of different dimensions on data quality, so that the preliminary quality score can fully and objectively reflect the overall quality level of the data.

[0051] In step S206, the preliminary quality score is normalized by the Sigmoid activation function to generate a comprehensive quality score value.

[0052] It should be noted that the Sigmoid activation function is a commonly used nonlinear function that maps input values to the interval (0, 1). This characteristic makes the Sigmoid function particularly suitable for normalization, as it can convert preliminary quality scores of different ranges into a unified standardized score, facilitating cross-dataset and cross-evaluation period comparisons. The score value processed by the Sigmoid function can intuitively reflect the relative level of data quality, and also facilitates integration with other evaluation indicators.

[0053] This embodiment realizes intelligent quality evaluation and business adaptation of multi-source heterogeneous data by constructing a quality evaluation space with integrity, consistency and timeliness as coordinate axes, dynamically adjusting the dimension weight by combining feature projection and entropy weight method, and generating a normalized comprehensive quality score using the scoring function matrix and Sigmoid function.

[0054] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, please refer to Figure 4 , the data quality evaluation result generation step S40 based on the large model still includes steps S301~S306 after that: Step S301, input the comprehensive quality score value into the preset quality defect diagnosis tree model for dimension decomposition, and identify the defect severity of each quality dimension.

[0055] It should be noted that the quality dimension includes the integrity, consistency and timeliness of the quality data. Specifically, when performing dimension decomposition, the diagnosis tree model will evaluate the severity of the defect according to the performance of the comprehensive quality score value in each dimension. For example, if the data score in the integrity dimension is low, the model will identify problems such as data missing and incomplete fields; in the consistency dimension, the model will check for data format errors and data conflicts; and in the timeliness dimension, the model will evaluate data update frequency and timestamp accuracy. Through this detailed dimension analysis, the model can generate a detailed defect report, pointing out the specific problems and severity of each dimension.

[0056] Step S302, based on the defect severity, retrieve and match from the preset repair strategy knowledge graph to obtain an optimization scheme.

[0057] It should be noted that the preset repair strategy knowledge graph is a comprehensive knowledge base that contains various data quality defects and their corresponding repair strategies. These repair strategies are based on industry best practices, expert experience and historical data cases, covering a variety of methods from simple data cleaning to complex data reconstruction. The knowledge graph organizes this information in a structured form, making the retrieval and matching process more efficient and accurate.

[0058] Specifically, when the severity of the defect is identified, the system will retrieve in the knowledge graph according to these information. The retrieval process will consider the type and severity of the defect, as well as the specific application scenario of the data. For example, for integrity defects, if the defect severity is high, the system may recommend using data filling or data reconstruction strategies; for consistency defects, it may recommend data format standardization or data conflict resolution strategies; and for timeliness defects, it may recommend data update mechanism optimization or timestamp calibration strategies. After the retrieval and matching is completed, a detailed optimization scheme report will be generated. This report not only contains specific repair strategies, but also provides steps to implement these strategies, expected effects and possible risk assessment. In this way, users can clearly understand how to optimize the identified defects, thereby improving data quality.

[0059] Step S303, convert the optimization scheme into readable suggestion text using natural language generation technology.

[0060] It should be noted that natural language generation technology is a technology for converting structured data into natural language text. In this embodiment, NLG technology is used to convert the optimization scheme retrieved from the preset repair strategy knowledge graph into easy-to-understand suggestion text. These optimization schemes usually contain technical details and operation steps, and the NLG technology can convert these contents into popular language, so that non-technical users can easily understand and execute.

[0061] Specifically, through the NLG technology, detailed suggestion text can be generated, which not only includes the specific steps of optimization, but also provides background information and expected effects of implementing these steps. For example, if the optimization scheme suggests cleaning the data, the NLG technology can generate the following suggestion text: "It is recommended to clean the data to remove duplicate records and correct format errors. This will improve the integrity and consistency of the data, thereby improving the data quality. The specific steps include: 1) Check for duplicate records in the data and delete them; 2) Correct the data format to ensure that all fields meet the predefined format standards."

[0062] In addition, the NLG technology can also generate personalized suggestion text according to the specific needs and background of the user. For example, for different business scenarios or data types, the suggestion text can contain specific notes and optimization strategies. This personalized content generation not only improves the relevance of the suggestions, but also enhances the user's acceptance and execution willingness.

[0063] Step S304, integrate the readability suggestion text by multi-document summarization algorithm to generate an optimization suggestion framework.

[0064] It should be noted that the multi-document summarization algorithm is a text processing technology that can extract key information from multiple documents and generate a concise summary. In this technical solution, the algorithm is used to integrate the readability suggestion text from different optimization schemes. These texts may contain repeated information or different angle suggestions, and through the multi-document summarization algorithm, the most core content can be extracted to generate a comprehensive optimization suggestion framework.

[0065] Specifically, first, the system collects all generated readability suggestion texts and pre-processes them, including removing repetitive content, correcting grammatical errors, etc. This step ensures the quality and consistency of the input texts. Next, the multi-document summarization algorithm analyzes the pre-processed texts and extracts key information from each text. These key information usually includes optimized specific steps, expected effects, and precautions. The algorithm integrates the extracted key information to generate a comprehensive optimization suggestion framework. This framework not only contains all important optimization suggestions, but also optimizes the logical structure to make the content more coherent and easy to understand. For example, similar suggestions are categorized together, and steps are arranged in order of implementation. Finally, the system generates a clear and concise optimization suggestion framework. This framework is presented in a format that is easy to read and operate, helping users quickly understand and implement optimization measures.

[0066] Step S305, the optimization suggestion framework is matched with the historical optimization case library to obtain case reference information.

[0067] It should be noted that the historical optimization case library is a database that stores a large number of past data quality optimization cases. The historical optimization case library includes quality defect descriptions and implemented optimization measures. These cases are based on actual application scenarios and experience summaries, and have reference value.

[0068] Specifically, first, extract key information from the optimization suggestion framework, including defect types, optimization goals, and main measures, etc. These information will be used as the basis for matching. Then, search for similar cases in the historical optimization case library based on the optimization suggestion framework. The retrieval process is based on a text similarity algorithm that compares the key information in the optimization suggestion framework with the defect descriptions and optimization measures recorded in the case library. Next, evaluate the similarity of the retrieved cases with the optimization suggestion framework and select the most matching case as a reference. Cases with high similarity mean that optimization measures have been successfully implemented in similar scenarios, and these measures have high reference value for current optimization work. Finally, extract detailed information from the matched cases, including specific optimization steps, precautions during implementation, problems encountered and solutions, etc. These information will be used as a supplement to the optimization suggestion framework to provide more comprehensive guidance for users.

[0069] Step S306, the optimization suggestion framework is combined with the case reference information to output a quality optimization report.

[0070] It should be noted that the optimization suggestion framework provides optimization directions and measures based on the current data quality evaluation results, while the case reference information extracts successful experience and precautions in actual operation from the historical optimization case library. Combining the two can ensure that the optimization report not only has theoretical guidance, but also has practical feasibility and effectiveness.

[0071] The embodiment identifies the data quality defect type and severity by constructing a diagnosis tree model, matches and repairs strategies combined with a knowledge graph, outputs structured optimization suggestions using natural language generation and summary technology, and finally forms a complete quality optimization report by fusing historical case information, realizes the whole-process closed-loop management from quality evaluation to problem diagnosis to optimization suggestion, and improves the intelligent and automatic level of data quality problem processing.

[0072] Based on the first embodiment of the present application, the present application also provides a data quality evaluation result generation device based on a large model, please refer to Figure 5 , the device comprises: The acquisition module 10 is configured to acquire a quality data set to be evaluated, and the quality data set comprises a structured data table and unstructured text data.

[0073] The processing module 20 is configured to extract semantic features of the quality data set by a preset quality evaluation model to obtain key quality features, wherein the preset quality evaluation model comprises an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer, and the output layer comprises a feature dimension reduction module. The screening module 30 is configured to screen the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, and the dimension-feature mapping relationship library comprises three quality dimensions of data quality, namely integrity, consistency and timeliness.

[0074] The result module 40 is configured to calculate the target quality features based on a preset evaluation rule library to obtain a comprehensive quality score value, and the preset evaluation rule library comprises a scoring function for evaluating the integrity, consistency and timeliness of the quality data.

[0075] The data quality evaluation result generation device based on a large model provided by the present application adopts the data quality evaluation result generation method based on a large model in the above embodiment, which can solve the technical problem of how to improve the accuracy and efficiency of data quality evaluation. Compared with the prior art, the beneficial effects of the data quality evaluation result generation device based on a large model provided by the present application are the same as those of the data quality evaluation result generation method based on a large model provided by the above embodiment, and other technical features of the data quality evaluation result generation device based on a large model are the same as those disclosed in the above embodiment method. Here, it is not repeated.

[0076] In an embodiment, the processing module 20 is further configured to perform vectorization processing on the quality data by an input embedding layer of a preset quality evaluation model to obtain dense vector features; perform weight distribution on the dense vector features by a multi-head self-attention layer to generate attention features; input the attention features into a feedforward neural network layer to perform nonlinear transformation and feature enhancement to obtain deep semantic features; and perform dimension reduction processing on the deep semantic features by a feature dimension reduction module of an output layer to obtain key quality features.

[0077] In an embodiment, the screening module 30 is further configured to establish a feature importance evaluation model for each quality dimension, wherein the feature importance evaluation model is trained based on a random forest algorithm; input the key quality features into the feature importance evaluation model corresponding to each dimension to generate importance scores in the corresponding dimensions; perform weight summation according to the importance scores to obtain a comprehensive score; when the comprehensive score exceeds a preset score threshold, mark the corresponding quality features in the key quality features as candidate features; perform redundancy detection on the candidate features to calculate Pearson correlation coefficients between the features; and screen the candidate features corresponding to the Pearson correlation coefficients according to a preset rule to obtain target quality features.

[0078] In an embodiment, the result module 40 is further configured to construct a three-dimensional weight space model, wherein three coordinate axes of the model correspond to the integrity, consistency and timeliness dimensions of the quality data, and a reference weight vector of the model is dynamically adjusted based on a business scenario; map the target quality features to the three-dimensional weight space model by a feature projection method to generate spatial distances between feature points and the reference weight vectors; calculate based on the spatial distances to generate initial dimension weights; perform weight correction on the initial dimension weights by an entropy weight method to obtain corrected weight coefficients; perform matrix multiplication operation on the corrected weight coefficients and scoring functions in a preset evaluation rule library to obtain preliminary quality scores; and perform normalization processing on the preliminary quality scores by a Sigmoid activation function to generate a comprehensive quality score value.

[0079] In an embodiment, the result module 40 is further configured to determine an optimal projection direction of the target quality features in the three-dimensional space by a principal component analysis method; map the target quality features to the three-dimensional space along the optimal projection direction to generate a feature point cloud distribution; calculate based on the feature point cloud distribution to obtain Euclidean distances from each feature point to three reference weight vectors, wherein each reference vector represents an ideal state of a quality dimension; correct the Euclidean distances by a Mahalanobis distance algorithm to obtain corrected distances; and perform normalization processing on the corrected distances to generate spatial distances between the feature points and the reference weight vectors.

[0080] In an embodiment, the result module 40 is further configured to input the comprehensive quality score into a preset quality defect diagnosis tree model when the comprehensive quality score is lower than a preset quality score threshold, and output a quality optimization report, the preset quality defect diagnosis tree model including three types of diagnosis branches of integrity defects, consistency defects and timeliness defects.

[0081] In an embodiment, the result module 40 is further configured to perform dimension decomposition on the comprehensive quality score input into the preset quality defect diagnosis tree model, identify defect severity of each quality dimension, the quality dimension including integrity, consistency and timeliness of the quality data; perform retrieval and matching based on the defect severity from a preset repair strategy knowledge graph to obtain an optimization scheme; convert the optimization scheme into a readability suggestion text by using a natural language generation technology; integrate the readability suggestion text by using a multi-document summarization algorithm to generate an optimization suggestion framework; perform similarity matching of the optimization suggestion framework with a historical optimization case library to obtain case reference information, the historical optimization case library including quality defect descriptions and implemented optimization measures; output a quality optimization report by combining the optimization suggestion framework with the case reference information.

[0082] The present application provides a large model-based data quality evaluation result generation device, which comprises at least one processor and a memory in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the large model-based data quality evaluation result generation method in Embodiment I.

[0083] Reference will now be made to the drawings, and specific examples relating to the large model-based data quality evaluation result generation device will be illustrated in the following embodiments with reference to the accompanying drawings. Figure 6 The large model-based data quality evaluation result generation device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The large model-based data quality evaluation result generation device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0084] As shown in Figure 6As shown, the large model-based data quality evaluation result generation device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the large model-based data quality evaluation result generation device to operate are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the large model-based data quality evaluation result generation device to communicate with other devices wirelessly or by wire to exchange data. Although a large model-based data quality evaluation result generation device with various components is shown in the figure, it should be understood that not all of the shown components are required to be implemented or possessed. More or less can be alternatively implemented or possessed.

[0085] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carrying out the program codes for performing the methods shown in the flowcharts carried on a computer readable storage medium. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0086] The device for generating a data quality evaluation result based on a large model provided in the application adopts the method for generating a data quality evaluation result based on a large model in the above embodiments, and can solve the technical problem of how to improve the accuracy and efficiency of data quality evaluation. Compared with the prior art, the device for generating a data quality evaluation result based on a large model provided in the application has the same beneficial effects as the method for generating a data quality evaluation result based on a large model provided in the above embodiments, and other technical features in the device for generating a data quality evaluation result based on a large model are the same as the features disclosed in the above method, which will not be repeated here.

[0087] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0088] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0089] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for executing the method for generating a data quality evaluation result based on a large model in the above embodiments.

[0090] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible storage medium containing or storing a program that can be executed by or used in conjunction with an instruction execution device. The program code contained on the computer readable storage medium can be transmitted in any suitable storage medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0091] The above computer readable storage medium can be included in the large model-based data quality evaluation result generation device, or can exist separately and not be assembled into the large model-based data quality evaluation result generation device.

[0092] The above computer readable storage medium carries one or more programs, which, when executed by the large model-based data quality evaluation result generation device, cause the large model-based data quality evaluation result generation device to write computer program code for performing the operations of the present application in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on a user computer, partially on a user computer, as a separate software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the methods and computer program products according to the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0094] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0095] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the above-mentioned big model-based data quality evaluation result generation method, and can solve the technical problem of how to improve the accuracy and efficiency of data quality evaluation. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the big model-based data quality evaluation result generation method provided by the above-mentioned embodiments, and will not be described here.

[0096] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the above-mentioned big model-based data quality evaluation result generation method.

[0097] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy and efficiency of data quality evaluation. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the big model-based data quality evaluation result generation method provided by the above-mentioned embodiments, and will not be described here.

[0098] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for generating data quality assessment results based on a large model, characterized in that, The method includes: Obtain the quality dataset to be evaluated, which includes structured data tables and unstructured text data; The quality dataset is subjected to semantic feature extraction through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module. The key quality features are filtered based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness. The target quality characteristics are calculated based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data.

2. The method as described in claim 1, characterized in that, The step of extracting key quality features from the quality dataset using a preset quality assessment model includes: The quality data is vectorized through the input embedding layer of the preset quality assessment model to obtain dense vector features; The dense vector features are weighted using a multi-head self-attention layer to generate attention features; The attention features are input into a feedforward neural network layer for nonlinear transformation and feature enhancement to obtain deep semantic features. The deep semantic features are dimensionality reduced by the feature dimensionality reduction module of the output layer to obtain key quality features.

3. The method as described in claim 1, characterized in that, The step of filtering the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features, wherein the dimension-feature mapping relationship library includes three quality dimensions of data quality: integrity, consistency, and timeliness, includes: A feature importance evaluation model is established for each quality dimension, and the feature importance evaluation model is trained based on the random forest algorithm; The key quality features are input into the feature importance evaluation model corresponding to each dimension to generate importance scores for the corresponding dimension. The overall score is obtained by summing the weights of the importance scores. When the overall score exceeds a preset score threshold, the corresponding quality feature among the key quality features is marked as a candidate feature; Redundancy detection is performed on the candidate features, and the Pearson correlation coefficient between the features is calculated; The candidate features corresponding to the Pearson correlation coefficient are screened according to preset rules to obtain the target quality features.

4. The method as described in claim 1, characterized in that, The step of calculating the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score includes: A three-dimensional weight space model is constructed, in which the three coordinate axes correspond to the dimensions of completeness, consistency and timeliness of quality data, respectively, and the benchmark weight vector of the model is dynamically adjusted based on the business scenario. The target quality features are mapped to the three-dimensional weight space model using the feature projection method, generating the spatial distance between feature points and each benchmark weight vector; The initial dimensional weights are generated based on the spatial distance. The initial dimension weights are corrected using the entropy weighting method to obtain the corrected weight coefficients. The modified weighting coefficients are multiplied by a matrix with the scoring function in the preset evaluation rule base to obtain a preliminary quality score. The preliminary quality score is normalized using the Sigmoid activation function to generate a comprehensive quality score.

5. The method as described in claim 4, characterized in that, The step of mapping the target quality features to the three-dimensional weight space model using the feature projection method to generate the spatial distance between feature points and each reference weight vector includes: The optimal projection direction of the target quality feature in three-dimensional space is determined by principal component analysis. The target quality features are mapped onto a three-dimensional space along the optimal projection direction to generate a feature point cloud distribution. The Euclidean distance from each feature point to the three reference weight vectors is calculated based on the feature point cloud distribution, where each reference vector represents the ideal state of a quality dimension. The Euclidean distance is corrected using the Mahalanobis distance algorithm to obtain the corrected distance; The corrected distances are normalized to generate spatial distances between feature points and each baseline weight vector.

6. The method as described in claim 1, characterized in that, After the step of calculating the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score, the method further includes: When the overall quality score is lower than the preset quality score threshold, the overall quality score is input into the preset quality defect diagnosis tree model, and a quality optimization report is output. The preset quality defect diagnosis tree model includes three diagnostic branches: integrity defects, consistency defects, and timeliness defects.

7. The method as described in claim 6, characterized in that, The step of inputting the comprehensive quality score into a preset quality defect diagnosis tree model and outputting a quality optimization report when the comprehensive quality score is lower than a preset quality score threshold includes: The comprehensive quality score is input into a preset quality defect diagnosis tree model for dimensional decomposition to identify the severity of defects in each quality dimension. The quality dimensions include the completeness, consistency, and timeliness of the quality data. Based on the severity of the defect, an optimized solution is obtained by searching and matching from a preset repair strategy knowledge graph; The optimization scheme is then converted into readable suggestion text using natural language generation technology. The readable suggestion text is integrated using a multi-document summarization algorithm to generate an optimized suggestion framework; The optimization suggestion framework is matched with the historical optimization case library to obtain case reference information. The historical optimization case library includes descriptions of quality defects and implemented optimization measures. By combining the aforementioned optimization suggestion framework with case reference information, a quality optimization report is generated.

8. A device for generating data quality assessment results based on a large model, characterized in that, The device includes: The acquisition module is used to acquire the quality dataset to be evaluated, which includes structured data tables and unstructured text data. The processing module is used to extract semantic features from the quality dataset through a preset quality assessment model to obtain key quality features. The preset quality assessment model includes an input embedding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output layer. The output layer includes a feature dimensionality reduction module. The filtering module is used to filter the key quality features based on a preset dimension-feature mapping relationship library to obtain target quality features. The dimension-feature mapping relationship library includes three quality dimensions: data quality integrity, consistency, and timeliness. The results module is used to calculate the target quality characteristics based on a preset evaluation rule base to obtain a comprehensive quality score. The preset evaluation rule base includes scoring functions for evaluating the completeness, consistency, and timeliness of quality data.

9. A device for generating data quality assessment results based on a large model, characterized in that, The device includes: a memory, a processor, and a large-model-based data quality assessment result generation program stored on the memory and running on the processor, the large-model-based data quality assessment result generation program being configured to implement the steps of the large-model-based data quality assessment result generation method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a data quality assessment result generation program based on a large model, which, when executed by a processor, implements the steps of the data quality assessment result generation method based on a large model as described in any one of claims 1-7.

Images