Method and apparatus for dataset semantic quality evaluation, and electronic device

Abstract

The application relates to the technical field of data evaluation, and discloses a method and device for semantic quality evaluation of a data set and electronic equipment, wherein the method comprises the following steps: performing deep semantic feature extraction on a plurality of text data in a data set to be evaluated, and obtaining a semantic vector corresponding to each text data; performing multi-dimensional semantic quality evaluation according to the semantic vector, and obtaining a plurality of evaluation scores; the evaluation dimensions of the multi-dimensional semantic quality evaluation include multiple dimensions of semantic consistency evaluation, semantic richness evaluation and semantic bias evaluation; and a comprehensive semantic quality score is determined according to the plurality of evaluation scores, and a visual report is generated. The semantic feature extraction is performed on the plurality of text data, the generated semantic vector can capture complex semantic information of the text and provide rich semantic representation. Then, the data set to be evaluated is evaluated from multiple dimensions, the semantic quality of the data set is comprehensively and objectively evaluated, a deeper quality evaluation is realized, and the quality evaluation effect of the data set is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data evaluation, for example to a method and device for semantic quality evaluation of a data set, and an electronic device. BACKGROUND

[0002] At present, with the rapid development of artificial intelligence technology, especially the wide application of deep learning in the fields of image recognition and natural language processing, the quality of data set, as the basis for training and verifying the model, has an increasingly significant impact on the performance of the model. A high-quality data set can help the model learn and generalize better, thereby improving the accuracy and robustness of the model. On the contrary, a low-quality data set may lead to poor model performance, and even bias and errors.

[0003] In the related art, the quality of the data set is evaluated by evaluating whether the data set contains all necessary information, whether it accurately reflects the actual situation, and whether it is consistent in format, structure and content.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art:

[0005] The data set quality evaluation method in the related art focuses on the surface indicators of the data set, and it is difficult to achieve a deeper quality evaluation of the data set, and the quality evaluation effect of the data set is poor.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not a general review, nor is it intended to determine the key / important components or delineate the scope of protection of these embodiments, but as a prelude to the detailed description below.

[0008] The embodiments of the present disclosure provide a method and device for semantic quality evaluation of a data set, and an electronic device, to achieve a deeper quality evaluation from the semantic level of the data set and improve the quality evaluation effect of the data set.

[0009] In some embodiments, the method for semantic quality evaluation of a data set comprises: performing deep semantic feature extraction on a plurality of text data in the data set to be evaluated to obtain a semantic vector corresponding to each text data; performing multi-dimensional semantic quality evaluation according to the semantic vector to obtain a plurality of evaluation scores; the evaluation dimensions of the multi-dimensional semantic quality evaluation include multiple ones of semantic consistency evaluation, semantic richness evaluation, and semantic bias evaluation; and determining a comprehensive semantic quality score according to the plurality of evaluation scores and generating a visual report.

[0010] Optionally, the deep semantic feature extraction on the plurality of text data in the data set to be evaluated to obtain a semantic vector corresponding to each text data comprises: performing data cleaning and format conversion on the plurality of text data in the data set to be evaluated to obtain a plurality of pre-processed text data; inputting each pre-processed text data into a pre-trained language model respectively to obtain a hidden layer output vector corresponding to all text fields in each text data; and performing aggregation according to all hidden layer output vectors in each text data to determine a semantic vector corresponding to each text data.

[0011] Optionally, the multi-dimensional semantic quality evaluation according to the semantic vector to obtain a plurality of evaluation scores comprises: calculating a semantic similarity distribution of the data set to be evaluated according to the semantic vector to obtain a semantic consistency evaluation score; analyzing the lexical diversity and semantic depth of the data set to be evaluated according to the semantic vector to obtain a semantic richness evaluation score; and analyzing the semantic bias pattern of the data set to be evaluated according to the semantic vector to obtain a semantic bias evaluation score.

[0012] Optionally, the calculation of the semantic similarity distribution of the data set to be evaluated according to the semantic vector to obtain a semantic consistency evaluation score comprises: determining a semantic weight matrix according to the importance of key semantic words in the data set to be evaluated; calculating the semantic similarity between any two text data according to the semantic weight matrix and the semantic vectors corresponding to the any two text data in the data set to be evaluated to obtain a similarity matrix; performing statistical analysis on the similarity matrix to obtain statistical indicators of the similarity matrix; and determining the semantic consistency evaluation score according to the statistical indicators.

[0013] Optionally, the analysis of the lexical diversity and semantic depth of the data set to be evaluated according to the semantic vector to obtain a semantic richness evaluation score comprises: performing word segmentation processing on the plurality of text data in the data set to be evaluated according to the semantic vector, and counting the number of different word categories and the frequency of occurrence according to the word segmentation processing result to obtain a lexical diversity indicator; analyzing the semantic feature distribution of the words in the plurality of text data in the data set to be evaluated according to the semantic vector to obtain a semantic depth indicator; the semantic feature distribution includes the variance of the words in different semantic dimensions, and the semantic roles and relationships of the words in the sentences; and obtaining the semantic richness evaluation score according to the lexical diversity indicator and the semantic depth indicator.

[0014] Optionally, the semantic bias assessment score is obtained by analyzing the semantic bias patterns of the dataset to be evaluated based on semantic vectors, including: identifying biased texts in the dataset to be evaluated based on semantic vectors; identifying structured data related to the biased texts and performing causal analysis based on the structured data to obtain the root cause of the bias, thereby determining the type of bias; and determining the semantic bias assessment score based on the type of bias and the biased texts.

[0015] Optionally, a comprehensive semantic quality score is determined based on multiple assessment scores, and a visualization report is generated, including: weighting the multiple assessment scores to obtain the comprehensive semantic quality score; performing causal analysis based on the multiple assessment scores and the comprehensive semantic quality score to determine the key factors affecting semantic quality; and generating a visualization report based on the multiple assessment scores, the comprehensive semantic quality score, and the key factors affecting semantic quality; wherein the visualization report includes one or more of the following: radar chart display, heat map matrix display, word cloud display, bubble chart display, and time series chart display.

[0016] Optionally, the method for semantic quality assessment of a dataset further includes: determining the evaluation dimensions of multi-dimensional semantic quality assessment based on the task scenario of the dataset to be evaluated; specifically including: in text classification tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment and semantic bias assessment; in natural language generation tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment and semantic richness assessment; in question answering system tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic richness assessment and semantic bias assessment; and in text analysis tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment, semantic richness assessment, and semantic bias assessment.

[0017] In some embodiments, an apparatus for semantic quality assessment of a dataset includes a processor and a memory storing program instructions, the processor being configured to execute the method for semantic quality assessment of a dataset as described above when the program instructions are executed.

[0018] In some embodiments, the electronic device includes: an electronic device body; and the apparatus for semantic quality assessment of a dataset as described above, which is mounted on the electronic device body.

[0019] The method for semantic quality assessment of datasets provided in this disclosure can achieve the following technical effects:

[0020] In this embodiment, deep semantic feature extraction is performed on multiple text data in the dataset to be evaluated. The generated semantic vectors can capture the complex semantic information of the text and provide rich semantic representations. Then, the dataset is quantitatively evaluated from three key dimensions: semantic consistency, semantic richness, and semantic bias. This identifies the semantic consistency of the text in the dataset, assesses whether the dataset contains rich semantic information, and detects the presence of semantic bias, thereby comprehensively and objectively evaluating the semantic quality of the dataset, achieving a deeper level of quality assessment, and improving the effectiveness of dataset quality evaluation. Furthermore, based on multiple evaluation scores, a comprehensive semantic quality score is generated, which can intuitively reflect the overall semantic quality of the dataset to be evaluated. The visualization report can also intuitively display the performance of the dataset to be evaluated on each semantic quality dimension, helping users to understand the semantic quality of the dataset more deeply.

[0021] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0022] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0023] Figure 1 This is a schematic diagram of a method for semantic quality assessment of a dataset provided in an embodiment of this disclosure;

[0024] Figure 2 This is a schematic diagram of another method for semantic quality assessment of datasets provided in this disclosure embodiment;

[0025] Figure 3 This is a schematic diagram of another method for semantic quality assessment of datasets provided in this disclosure embodiment;

[0026] Figure 4 This is a schematic diagram of another method for semantic quality assessment of datasets provided in this disclosure embodiment;

[0027] Figure 5 This is a schematic diagram of an apparatus for semantic quality assessment of a dataset provided in an embodiment of this disclosure;

[0028] Figure 6 This is a schematic diagram of another apparatus for semantic quality assessment of datasets provided in an embodiment of this disclosure. Detailed Implementation

[0029] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0030] The terms "first," "second," etc., used in the technical solutions described in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0031] Unless otherwise stated, the term "multiple" means two or more.

[0032] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0033] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0034] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0035] Combination Figure 1 As shown, this disclosure provides a method for semantic quality assessment of a dataset. The execution entity of this method can be a processor, and the method includes:

[0036] S101, the processor performs deep semantic feature extraction on multiple text data in the dataset to be evaluated, and obtains the semantic vector corresponding to each text data.

[0037] S102, the processor performs multi-dimensional semantic quality evaluation based on semantic vectors and obtains multiple evaluation scores.

[0038] The multidimensional semantic quality assessment includes multiple assessment dimensions such as semantic consistency assessment, semantic richness assessment, and semantic bias assessment.

[0039] S103, the processor determines a comprehensive semantic quality score based on multiple evaluation scores and generates a visual report.

[0040] In this embodiment, deep semantic feature extraction is performed on multiple text data in the dataset to be evaluated. The generated semantic vectors can capture the complex semantic information of the text and provide rich semantic representations. Then, the dataset is quantitatively evaluated from three key dimensions: semantic consistency, semantic richness, and semantic bias. This identifies the semantic consistency of the text in the dataset, assesses whether the dataset contains rich semantic information, and detects the presence of semantic bias, thereby comprehensively and objectively evaluating the semantic quality of the dataset, achieving a deeper level of quality assessment, and improving the effectiveness of dataset quality evaluation. Furthermore, based on multiple evaluation scores, a comprehensive semantic quality score is generated, which can intuitively reflect the overall semantic quality of the dataset to be evaluated. The visualization report can also intuitively display the performance of the dataset to be evaluated on each semantic quality dimension, helping users to understand the semantic quality of the dataset more deeply.

[0041] Optionally, deep semantic feature extraction is performed on multiple text data in the dataset to be evaluated to obtain the semantic vector corresponding to each text data. This includes: cleaning and format conversion of multiple text data in the dataset to be evaluated to obtain multiple preprocessed text data; inputting each preprocessed text data into a pre-trained language model to obtain the hidden layer output vector corresponding to all text fields in each text data; and aggregating all hidden layer output vectors in each text data to determine the semantic vector corresponding to each text data.

[0042] Combination Figure 2 As shown, this disclosure provides another method for semantic quality assessment of datasets, including:

[0043] S201, the processor performs data cleaning and format conversion on multiple text data in the dataset to be evaluated, and obtains multiple preprocessed text data.

[0044] S202, the processor inputs each preprocessed text data into the pre-trained language model to obtain the hidden layer output vector corresponding to all text fields in each text data.

[0045] S203, the processor aggregates all hidden layer output vectors in each text data to determine the semantic vector corresponding to each text data.

[0046] S204, the processor performs multi-dimensional semantic quality evaluation based on semantic vectors and obtains multiple evaluation scores.

[0047] The S205 processor determines a comprehensive semantic quality score based on multiple evaluation scores and generates a visual report.

[0048] In this embodiment, the dataset to be evaluated supports various common dataset formats. Through data cleaning and format conversion, the accuracy and consistency of the input data can be ensured. The preprocessed text data is then input into a pre-trained language model. This pre-trained language model, through a multi-layer neural network structure, can capture the complex semantic information of the text, including vocabulary, grammar, and contextual relationships, thereby generating high-dimensional hidden layer output vectors that more comprehensively represent the semantic information of the text. Finally, by aggregating all hidden layer output vectors, the semantic information of the text is condensed into a single semantic vector, facilitating subsequent semantic quality evaluation.

[0049] Optionally, data cleaning is performed on multiple text data in the dataset to be evaluated, including: removing data noise (such as irrelevant characters, duplicate data, etc.), filling in missing values, and correcting erroneous data.

[0050] Optionally, the format of multiple text data in the dataset to be evaluated may be converted, including converting the dataset to a format acceptable to the pre-trained language model. For example, converting text data to a uniform encoding format (such as UTF-8), or standardizing date formats.

[0051] Optionally, the pre-trained language model includes BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), or a custom model.

[0052] Optionally, obtain the hidden layer output vector corresponding to all text fields in each text data, including: obtaining the hidden layer output vector corresponding to each word or subword.

[0053] In this embodiment, the pre-trained language model first performs word segmentation on the text data, dividing the text into words or subwords. Then, it generates a hidden layer output vector for each word or subword, which contains the semantic information of the word or subword. The hidden layer output vector for each word or subword is high-dimensional, capable of capturing rich semantic information, including vocabulary, grammar, and contextual relationships. The same word will generate different vectors in different contexts, enabling the model to better understand the contextual semantics of words. Similar words or subwords are closer together in the vector space. For example, for the input text data "This phone takes great photos," the word segmentation results can be obtained as: ["this," "phone," "take photos," "effect," "great," "good"]. Then, six different hidden layer output vectors are generated based on these six word segmentation results.

[0054] Optionally, the semantic vector corresponding to each text data is determined by aggregating all hidden layer output vectors in each text data, including: performing average pooling on all hidden layer output vectors or taking the vector corresponding to the [CLS] label to obtain the semantic vector representation of the text data.

[0055] Optionally, the semantic vector is a multi-dimensional real-valued vector, the dimension of which depends on the configuration of the pre-trained language model. For example, the semantic vector generated by the BERT-base model has a dimension of 768, and the semantic vector is stored in floating-point form, with each component representing the weight or intensity of the text data on a specific semantic feature.

[0056] Optionally, semantic vectors possess the characteristics of high dimensionality, semantic condensation, context dependence, and discriminability. High dimensionality refers to the fact that semantic vectors typically have hundreds to thousands of dimensions, enabling them to capture complex semantic information. Semantic condensation refers to the fact that pre-trained language models, through multi-layer neural network operations, condense the vocabulary, grammar, and semantic relationships of text data into numerical vectors, making similar texts appear closer together in the vector space. Context dependence refers to the fact that the same word generates different vectors in different contexts; for example, "apple" corresponds to different semantic representations in the contexts of "fruit" and "technology company." Discriminability refers to the ability to quantify semantic differences in texts, providing a basis for subsequent evaluation. For example, the vector corresponding to "a cat is chasing a mouse" and the vector corresponding to "a dog is chasing a ball" have similar verb-object structure distances in the semantic space, but their subject semantics differ significantly.

[0057] Optionally, a multi-dimensional semantic quality assessment can be performed based on semantic vectors to obtain multiple assessment scores, including: calculating the semantic similarity distribution of the dataset to be assessed based on semantic vectors to obtain a semantic consistency assessment score; analyzing the lexical diversity and semantic depth of the dataset to be assessed based on semantic vectors to obtain a semantic richness assessment score; and analyzing the semantic bias patterns of the dataset to be assessed based on semantic vectors to obtain a semantic bias assessment score.

[0058] Combination Figure 3 As shown, this disclosure provides a method for semantic quality assessment of a dataset, including:

[0059] S301, the processor performs deep semantic feature extraction on multiple text data in the dataset to be evaluated, and obtains the semantic vector corresponding to each text data.

[0060] S302, the processor calculates the semantic similarity distribution of the dataset to be evaluated based on the semantic vectors, and obtains the semantic consistency evaluation score.

[0061] S303: The processor analyzes the lexical diversity and semantic depth of the dataset to be evaluated based on semantic vectors to obtain a semantic richness evaluation score.

[0062] S304, the processor analyzes the semantic bias patterns of the dataset to be evaluated based on semantic vectors to obtain a semantic bias evaluation score.

[0063] S305 The processor determines a comprehensive semantic quality score based on the semantic consistency assessment score, semantic richness assessment score, and semantic bias assessment score, and generates a visual report.

[0064] In this embodiment, by calculating the semantic similarity of the dataset to be evaluated, the semantic consistency of the text in the dataset can be quantified, helping to identify whether there are thematic or semantic biases in the dataset. By analyzing lexical diversity and semantic depth, the semantic richness of the dataset can be quantified, helping to identify whether the dataset contains rich semantic information. By identifying potential semantic biases in the dataset, it helps to discover potential unfairness or inaccuracies in the dataset.

[0065] Optionally, the semantic similarity distribution of the dataset to be evaluated is calculated based on the semantic vectors to obtain a semantic consistency evaluation score, including: determining a semantic weight matrix based on the importance of key semantic words in the dataset to be evaluated; calculating the semantic similarity between any two text data based on the semantic weight matrix and the semantic vectors corresponding to any two text data in the dataset to be evaluated to obtain a similarity matrix; performing statistical analysis on the similarity matrix to obtain statistical indicators of the similarity matrix; and determining the semantic consistency evaluation score based on the statistical indicators.

[0066] In this embodiment, a semantic weight matrix determined based on the importance of key semantic words in the dataset to be evaluated is introduced during the calculation of semantic similarity. This allows key semantic words to contribute more significantly to the semantic similarity calculation. The semantic similarity between two text data can be determined based on cosine similarity. For the semantic vectors A and B of two text data, the formula for calculating semantic similarity is: y = (A·W·B) / (||A||·||B||), where W is the semantic weight matrix, A·W·B represents the dot product of the three vectors, and ||A|| and ||B|| represent the magnitudes of the vectors, respectively. The calculated semantic similarity value ranges from [0, 1], with values ​​closer to 1 indicating higher semantic similarity. By calculating the semantic similarity between any two text data in the dataset to be evaluated, a similarity matrix can be obtained. Then, statistical analysis is performed on the values ​​in the similarity matrix to calculate statistical indicators such as the mean and median of the similarity. Based on these statistical indicators and combined with certain scoring rules (such as setting the score to 90 to 100 for an average similarity of 0.8 or above, and 70 to 89 for 0.6 to 0.8), the semantic consistency evaluation score is quantified.

[0067] In one specific embodiment, after generating semantic vectors for the texts "This phone takes great photos" and "This phone has excellent photography capabilities," a semantic weight matrix is ​​determined for each. In the semantic weight matrix, "photo quality" and "photography capabilities" are assigned higher weights as key semantic terms. Then, the semantic similarity between the two is calculated based on the semantic weight matrix and the semantic vectors. Assuming a result of 0.95, it indicates a high semantic similarity. In another specific embodiment, for an e-commerce user review dataset, the average semantic similarity between all pairwise review texts is calculated to be 0.75. According to the scoring rules, a semantic consistency evaluation score of 85 can be obtained.

[0068] Optionally, a semantic richness assessment score is obtained by analyzing the lexical diversity and semantic depth of the dataset to be evaluated based on semantic vector analysis. This includes: performing word segmentation on multiple text data in the dataset to be evaluated based on semantic vectors, and statistically analyzing the number and frequency of different word categories based on the segmentation results to obtain a lexical diversity index; analyzing the semantic feature distribution of words in multiple text data in the dataset to be evaluated based on semantic vector analysis to obtain a semantic depth index; the semantic feature distribution includes the variance of words in different semantic dimensions, as well as the semantic roles and relationships of words in sentences; and obtaining a semantic richness assessment score based on the lexical diversity index and the semantic depth index.

[0069] In this embodiment, text processing tools (such as NLTK, spaCy, etc.) are used to segment the text, and the frequency and number of different words are statistically analyzed. Simultaneously, a lexical diversity index is calculated based on the number of different word types, such as the ratio of the number of different word types to the total number of words; a higher value indicates a richer and more diverse vocabulary. Then, a pre-trained language model is used to analyze the semantic feature distribution of words in the text to obtain a semantic depth index. For example, principal component analysis is performed on the semantic vectors of words to extract the main semantic dimensions. These semantic dimensions include the different meanings of the words (e.g., "apple" can refer to a fruit or a technology company) and different parts of speech (noun, verb, adjective, etc.). The semantic vectors are projected onto different semantic dimensions, and the variance of the words in each semantic dimension is calculated based on the projection and the projection mean. A larger variance indicates a greater semantic difference in that dimension, indicating a richer semantic depth. For example, for the word "apple," the variance in the semantic dimension "fruit" is 0.25, while the variance in the semantic dimension "technology company" is 0.35, indicating that the word "apple" has a richer semantic expression in the second semantic dimension. Simultaneously, semantic role labeling technology is used to analyze the semantic roles and relationships of words in sentences, further evaluating their semantic depth. Finally, a weighting scheme is formulated by combining lexical diversity and semantic depth indicators. For example, lexical diversity is assigned a weight of 60%, and semantic depth a weight of 40%. Based on their respective performance, scores are calculated separately and then weighted and summed to obtain the semantic richness assessment score.

[0070] In a specific embodiment, for an e-commerce user review dataset, the number of different word categories is 15, and the total number of words is 20. Therefore, the lexical diversity index is 75, and the semantic depth index, determined based on the semantic feature distribution, is 80. According to the weighted allocation, the semantic richness evaluation score is 75 × 0.6 + 80 × 0.4 = 77.

[0071] Optionally, the semantic bias assessment score is obtained by analyzing the semantic bias patterns of the dataset to be evaluated based on semantic vectors, including: identifying biased texts in the dataset to be evaluated based on semantic vectors; identifying structured data related to the biased texts and performing causal analysis based on the structured data to obtain the root cause of the bias, thereby determining the type of bias; and determining the semantic bias assessment score based on the type of bias and the biased texts.

[0072] In this embodiment, a deep learning model (such as a Transformer-based model) can be used to detect semantic bias in text. A pre-trained bias detection model can be used, or fine-tuned training can be performed on a large dataset labeled with bias types (such as gender bias, racial bias, etc.). The bias detection model classifies or predicts text to identify potential semantic bias patterns. Based on the output of the bias detection model, the proportion of biased text in the statistical dataset relative to the total number of texts can be determined. For detected biased text, structured data related to the biased text, such as other text data related to the bias subject, is extracted. This data can help analyze the background of the bias. A causal graph model is constructed based on the structured data to perform counterfactual reasoning to determine the root cause of the bias, thereby determining the bias type. Simultaneously, considering factors such as the severity of the bias type, a rule for calculating the bias score is established. For example, a score of 90 to 100 is set for biased texts below 10%, and a score of 70 to 89 is set for 10% to 20%, etc., and this is adjusted in conjunction with the bias type weights to obtain the final bias score.

[0073] In one specific embodiment, for an e-commerce user review dataset, the text "This brand of mobile phone is more suitable for men" is classified and predicted by a bias detection model to contain gender bias, belonging to a biased statement towards male users. If the proportion of all detected biased texts is 15%, of which 10% belong to a more serious brand bias, the bias score is calculated to be 65 according to the scoring rules.

[0074] Optionally, causal analysis can be performed based on structured data to obtain the root cause of the bias, including: constructing counterfactual questions based on the bias subject in the biased text and obtaining the results of the counterfactual questions; and performing causal analysis based on the results of the counterfactual questions to determine the root cause of the bias.

[0075] In another specific embodiment, for the biased text "The job openings are generally for men," the biased subject is "men." A counterfactual question is constructed: "Suppose a female applicant has the exact same STEM background and professional skills as a male applicant; how much would her probability of being hired change?" By intervening in the gender variable while controlling for the confounding variables "major" and "skills," the probabilities of admission after intervention are estimated. If the counterfactual question's estimation results show that, after controlling for "major" and "skills," the probability difference caused by gender changes becomes insignificant, it indicates that the apparent "gender bias" is actually a confounding effect caused by "major" and "skills," and its true "root cause" is professional bias.

[0076] Optionally, a comprehensive semantic quality score is determined based on multiple assessment scores, and a visualization report is generated, including: weighting the multiple assessment scores to obtain the comprehensive semantic quality score; performing causal analysis based on the multiple assessment scores and the comprehensive semantic quality score to determine the key factors affecting semantic quality; and generating a visualization report based on the multiple assessment scores, the comprehensive semantic quality score, and the key factors affecting semantic quality; wherein the visualization report includes one or more of the following: radar chart display, heat map matrix display, word cloud display, bubble chart display, and time series chart display.

[0077] Combination Figure 4 As shown, this disclosure provides another method for semantic quality assessment of datasets, including:

[0078] S401, the processor performs deep semantic feature extraction on multiple text data in the dataset to be evaluated, and obtains the semantic vector corresponding to each text data.

[0079] S402, the processor performs multi-dimensional semantic quality evaluation based on semantic vectors and obtains multiple evaluation scores.

[0080] S403, the processor performs a weighted calculation on multiple evaluation scores to obtain a comprehensive semantic quality score.

[0081] The S404 processor performs causal analysis based on multiple evaluation scores and a comprehensive semantic quality score to identify key factors affecting semantic quality.

[0082] The S405 processor generates a visual report based on multiple evaluation scores, a comprehensive semantic quality score, and key factors affecting semantic quality.

[0083] In this embodiment, a weighted summation method is used to synthesize the scores of each dimension. Based on the importance of semantic consistency, richness, and bias in semantic quality assessment, corresponding weights are assigned (e.g., consistency weight 40%, richness weight 30%, bias weight 30%). The scores of each dimension are multiplied by their respective weights and then summed to obtain the comprehensive semantic quality score. For example: if the semantic consistency assessment score is 85 with a weight of 40%; the semantic richness assessment score is 78 with a weight of 30%; and the semantic bias assessment score is 65 with a weight of 30%, then the comprehensive semantic quality score is: 85 × 0.4 + 78 × 0.3 + 65 × 0.3 = 75. Then, causal analysis is performed based on the multiple assessment scores and the comprehensive semantic quality score to determine the key factors affecting semantic quality. For example, the key factor causing poor semantic consistency may be the quality of data annotation, and the key factor causing poor semantic richness may be the data source.

[0084] Optionally, the visualization report includes a 3D radar chart. The three dimensions represent semantic consistency, richness, and bias, respectively. Using semantic consistency, semantic richness, and semantic bias as axes, the values ​​range from 0 to 100, visually displaying the scores for each dimension and the overall balance. For example, marking 85 for the consistency dimension, 78 for the richness dimension, and 65 for the bias dimension forms a triangular area, visually displaying the scores for each dimension and the overall balance.

[0085] Optionally, the visualization report includes a heatmap matrix. In semantic consistency assessment, a heatmap is used to display the semantic similarity between pairs of subsets, with each subset representing a row and column, and the color intensity indicating the similarity level. For example, suppose the dataset is divided into 5 subsets. A 5×5 heatmap matrix is ​​generated, where each element represents the semantic similarity of the comment text between the corresponding two subsets. For example, the similarity between subset 1 and subset 2 is 0.85, displayed as a darker red; the similarity between subset 3 and subset 5 is 0.5, displayed as a lighter blue.

[0086] Optionally, the visualization report includes a word cloud of lexical diversity. In the semantic richness assessment, a word cloud of word frequencies in the dataset is generated, with word size reflecting frequency of occurrence, combined with semantic clustering analysis. For example, positive emotion words such as "good," "not bad," and "like" have larger font sizes, while negative emotion words such as "bad" and "not good" have smaller font sizes. Simultaneously, semantic clustering regions of different emotion words are labeled, demonstrating the richness of emotional expression.

[0087] Optionally, the visualization report includes a bubble chart of bias patterns. In semantic bias detection, bias type (such as gender, race) is plotted on the horizontal axis, frequency of occurrence on the vertical axis, bubble size represents the intensity of influence, and different colors distinguish between positive and negative biases. For example, a red bubble with a size of 30 is plotted at the coordinate point (0.2, 0.15) corresponding to gender bias, indicating a frequency of 20% and an intensity of 15%; a blue bubble with a size of 25 is plotted at the coordinate point (0.15, 0.1) corresponding to brand bias, indicating a frequency of 15% and an intensity of 10%.

[0088] Optionally, the visualization report includes interactive time series plots. These plots demonstrate the semantic quality trends of the dataset across different version iterations and support comparison by dimension.

[0089] Optionally, the method for semantic quality assessment of a dataset further includes: determining the evaluation dimensions of multi-dimensional semantic quality assessment based on the task scenario of the dataset to be evaluated; specifically including: in text classification tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment and semantic bias assessment; in natural language generation tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment and semantic richness assessment; in question answering system tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic richness assessment and semantic bias assessment; and in text analysis tasks, determining the evaluation dimensions of multi-dimensional semantic quality assessment to include semantic consistency assessment, semantic richness assessment, and semantic bias assessment.

[0090] In this embodiment, the evaluation dimensions for multi-dimensional semantic quality assessment are determined based on the task scenario of the dataset to be evaluated, ensuring that the dataset is evaluated in accordance with the specific task requirements. Specifically, for text classification tasks, it is necessary to ensure that the model can accurately learn the semantic relationships between text and labels, and to ensure the fairness and impartiality of the dataset; therefore, semantic consistency and semantic bias assessments are required. For natural language generation tasks, it is necessary to ensure that the text generated by the model is semantically coherent and relevant, and that the generated text content is rich; therefore, semantic consistency and semantic richness assessments are required. For question-answering system tasks, it is necessary to ensure that the answers are not only accurate but also rich in content, and to ensure the fairness and impartiality of the question-answering system; therefore, semantic richness and semantic bias assessments are required. For text analysis tasks, it is necessary to ensure the readability and comprehensibility of the text by the model, and to ensure that the text content is rich, covering multiple aspects, and that the text analysis is fair and impartial; therefore, semantic consistency, semantic richness, and semantic bias assessments are required. By evaluating the semantic quality of the dataset under different task scenarios, it is ensured that the dataset meets the requirements of specific tasks.

[0091] The following describes a specific embodiment of the method for semantic quality assessment of datasets provided in this disclosure.

[0092] In one specific implementation, a user submitted an e-commerce user review dataset in CSV (Comma-Separated Values) format. The dataset format was automatically checked and found to be compliant, requiring no conversion. A pre-trained language model was used to encode the review text, generating semantic vectors. Semantic similarity of the review text was calculated, revealing that most reviews revolved around product performance and price, resulting in a semantic consistency score of 85. Lexical diversity and semantic depth analysis showed that the dataset was relatively rich in sentiment expression, achieving a semantic richness score of 78. Semantic bias detection revealed a bias towards certain brands, resulting in a semantic bias score of 65. Combining the scores across all dimensions, the dataset's overall semantic quality score was calculated to be 75. Finally, a visualization report was generated, showcasing the dataset's performance across various semantic quality dimensions, providing users with detailed feedback and optimization suggestions.

[0093] Combination Figure 5 As shown in the embodiments of this disclosure, an apparatus 500 for semantic quality assessment of a dataset is provided, including a semantic feature extraction module 501, a semantic quality assessment module 502, and a scoring and feedback module 503. The semantic feature extraction module 501 is configured to perform deep semantic feature extraction on multiple text data in the dataset to be assessed, obtaining a semantic vector corresponding to each text data. The semantic quality assessment module 502 is configured to perform multi-dimensional semantic quality assessment based on the semantic vectors, obtaining multiple assessment scores; the multi-dimensional semantic quality assessment includes semantic consistency assessment, semantic richness assessment, and semantic bias detection. The scoring and feedback module 503 is configured to determine a comprehensive semantic quality score based on the multiple assessment scores and generate a visual report.

[0094] The apparatus for semantic quality assessment of datasets provided in this disclosure introduces deep learning technology to perform deep semantic analysis of the datasets. This enables it to keenly capture subtle differences and potential problems at the semantic level, opening up new perspectives for dataset quality assessment. By constructing a comprehensive semantic quality assessment index system covering key dimensions such as semantic consistency, richness, and bias, multi-dimensional quantitative assessment is achieved. Finally, a scientifically sound weighted algorithm is used to calculate a comprehensive semantic quality score, and the assessment results are presented in an intuitive visual report, allowing users to quickly understand the semantic quality status of the dataset and providing strong support for optimization and improvement.

[0095] Combination Figure 6As shown, this disclosure provides an apparatus 600 for semantic quality assessment of a dataset, including a processor 601 and a memory 602. Optionally, the apparatus may further include a communication interface 603 and a bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can invoke logical instructions in the memory 602 to execute the method for semantic quality assessment of a dataset as described in the above embodiments.

[0096] Furthermore, the logic instructions in the aforementioned memory 602 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0097] The memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, that is, it implements the method for semantic quality assessment of datasets in the above embodiments.

[0098] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and may also include non-volatile memory.

[0099] This disclosure provides an electronic device, including: an electronic device body, and the aforementioned apparatus for semantic quality assessment of a dataset. The apparatus for semantic quality assessment of a dataset is installed in the electronic device body. The installation relationship described herein is not limited to placement within the electronic device, but also includes installation connections with other components of the electronic device, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the apparatus for semantic quality assessment of a dataset can be adapted to feasible electronic device bodies to achieve other feasible embodiments.

[0100] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method for semantic quality assessment of a dataset.

[0101] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code.

[0102] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the technical solutions described herein. As used in the technical solutions described herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein refers to any and all possible combinations of one or more of the associated listed elements. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for semantic quality assessment of datasets, characterized in that, include: Deep semantic feature extraction is performed on multiple text data in the dataset to be evaluated to obtain the semantic vector corresponding to each text data. Multi-dimensional semantic quality assessment is performed based on semantic vectors to obtain multiple assessment scores; the assessment dimensions of multi-dimensional semantic quality assessment include multiple aspects of semantic consistency assessment, semantic richness assessment, and semantic bias assessment. Based on multiple assessment scores, a comprehensive semantic quality score is determined, and a visual report is generated. Among them, multi-dimensional semantic quality assessment is performed based on semantic vectors to obtain multiple assessment scores, including: The semantic similarity distribution of the dataset to be evaluated is calculated based on semantic vectors to obtain a semantic consistency evaluation score. Specifically, this includes: determining a semantic weight matrix based on the importance of key semantic words in the dataset to be evaluated; calculating the semantic similarity between any two text data based on the semantic weight matrix and the semantic vectors corresponding to any two text data in the dataset to be evaluated to obtain a similarity matrix; performing statistical analysis on the similarity matrix to obtain statistical indicators of the similarity matrix; and determining the semantic consistency evaluation score based on the statistical indicators. The semantic richness assessment score is obtained by analyzing the lexical diversity and semantic depth of the dataset to be evaluated using semantic vector analysis. Specifically, this includes: segmenting multiple text data in the dataset to be evaluated using semantic vectors, and statistically analyzing the number and frequency of different word categories based on the segmentation results to obtain a lexical diversity index; analyzing the semantic feature distribution of words in multiple text data in the dataset to be evaluated using semantic vector analysis to obtain a semantic depth index; the semantic feature distribution includes the variance of words in different semantic dimensions, as well as the semantic roles and relationships of words in sentences; and obtaining a semantic richness assessment score based on the lexical diversity index and the semantic depth index. The semantic bias assessment score is obtained by analyzing the semantic bias patterns of the dataset to be evaluated based on semantic vector analysis. Specifically, this includes: identifying biased texts in the dataset to be evaluated based on semantic vectors; identifying structured data related to the biased texts and performing causal analysis based on the structured data to obtain the root causes of the bias, thereby determining the type of bias; and determining the semantic bias assessment score based on the type of bias and the biased texts.

2. The method according to claim 1, characterized in that, Deep semantic feature extraction is performed on multiple text data in the dataset to be evaluated to obtain the semantic vector corresponding to each text data, including: Data cleaning and format conversion were performed on multiple text data in the dataset to be evaluated to obtain multiple preprocessed text data; Each preprocessed text data is input into a pre-trained language model to obtain the hidden layer output vector corresponding to all text fields in each text data. The semantic vector corresponding to each text data is determined by aggregating all hidden layer output vectors in each text data.

3. The method according to claim 1, characterized in that, Based on multiple assessment scores, a comprehensive semantic quality score is determined, and a visual report is generated, including: A comprehensive semantic quality score is obtained by weighting multiple evaluation scores. Causal analysis was conducted based on multiple assessment scores and a comprehensive semantic quality score to identify key factors affecting semantic quality. A visualization report is generated based on multiple assessment scores, a comprehensive semantic quality score, and key factors affecting semantic quality. The visualization report includes one or more of the following: radar chart, heat map matrix, word cloud, bubble chart, and time series chart.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: Based on the task scenario of the dataset to be evaluated, the evaluation dimensions for multi-dimensional semantic quality assessment are determined; specifically, these include: In text classification tasks, the evaluation dimensions for multi-dimensional semantic quality assessment include semantic consistency assessment and semantic bias assessment. In natural language generation tasks, the evaluation dimensions for multi-dimensional semantic quality assessment include semantic consistency assessment and semantic richness assessment. In the question-answering system task, the evaluation dimensions for multi-dimensional semantic quality assessment include semantic richness assessment and semantic bias assessment. In text analysis tasks, the evaluation dimensions for multi-dimensional semantic quality assessment include semantic consistency assessment, semantic richness assessment, and semantic bias assessment.

5. An apparatus for semantic quality assessment of a dataset, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, execute the method for semantic quality assessment of a dataset as described in any one of claims 1 to 4.

6. An electronic device, characterized in that, include: The electronic device itself; The apparatus for semantic quality assessment of a dataset as described in claim 5 is installed on the electronic device body.

Images