A visual language data quality evaluation method based on structure decomposition theory
By employing a visual language data quality assessment method based on structural decomposition theory, and utilizing feature vectors and entropy calculations to generate comprehensive assessment results, the problems of uniformity and interpretability in visual language data assessment are solved, thereby improving assessment efficiency and accuracy.
Patent Information
- Application Number
- CN202610683136.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-18
- Publication Date
- 2026-08-25
AI Technical Summary
Existing visual language data quality assessment methods cannot provide a unified and systematic assessment framework. Traditional methods suffer from misjudgments and lack interpretability of assessment results, while multimodal large language model assessment methods suffer from bias and waste of computational resources.
We employ a method based on structural decomposition theory, extracting image and text feature vectors through an encoder, evaluating image and text integrity by combining a large language model, introducing cosine similarity and entropy computation for richness, and using a stability-aware fusion strategy to generate comprehensive quality assessment results.
It provides mathematically supported interpretability assessments, which improves assessment efficiency and robustness, reduces computational resource consumption, and ensures the accuracy and consistency of assessment results.
Smart Images
Figure CN122634201A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data quality assessment, and specifically to a method for assessing the quality of visual language data based on structural decomposition theory. Background Technology
[0002] Visual language data, composed of both image and text modalities, is the most common and core data form in multimodal information processing. As the size of visual language datasets expands from millions to billions, data quality issues have received increasing attention. Data quality assessment methods, by identifying high-value data instances from massive heterogeneous datasets, support data filtering in model training processes and fine-grained data governance of visual language corpora, and are important technical means for academia and industry to construct high-quality datasets.
[0003] However, the evaluation complexity of visual language data is significantly higher than that of unimodal data. Its quality assessment not only involves unimodal quality detection of images and text, but also requires measuring the degree of semantic alignment between modalities, and demands that the evaluation results have good interpretability and reliability.
[0004] Common traditional heuristic evaluation methods assess the quality of a single modality or the degree of alignment between modalities in isolation. These methods each have limitations. For example, NIMA, a unimodal metric, focuses on measuring the visual sharpness of an image (Talebi and Milanfar 2018. NIMA: Neural Image Assessment. In IEEE Transactions on Image Processing.), while CCNet focuses on measuring the semantic fluency of text (Wenzek et al. 2020. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In LREC.). Because these methods only measure the quality of a single modality, they are not suitable for cross-modal data and may misclassify data instances with mismatched modalities, such as assigning a high quality score to the incorrect pairing of "blurry image and fluent text." In contrast, alignment metrics such as CLIPScore (Hessel et al. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In EMNLP.) measure the degree of alignment between image and text modalities. They tend to favor simple descriptions with fewer features and give low scores to samples with richer contextual content. This filters out high-information data that is crucial for model generalization.
[0005] Since these traditional methods have failed to provide a unified and systematic evaluation framework, they are unable to deliver consistent and convincing evaluation results in large-scale visual language data governance tasks. More advanced evaluation methods based on multimodal large language models have become the focus of current research.
[0006] However, current state-of-the-art multimodal large language model evaluation methods have certain limitations. While large language models can provide semantic evaluation using large-scale prior knowledge, as black-box models, their generated end-to-end scores lack the mathematical foundation and interpretability required for data governance (Zhang et al. 2024. GPT-4 as a Judge for Multimodal Data Quality; Wu et al. 2023. Early Evaluation of Large Language Models.). Furthermore, due to inherent subjective biases, these methods tend to assign lower scores to high-quality data instances that do not conform to their model preferences, resulting in a loss of diversity in evaluation results. In summary, how to construct a visual language data quality evaluation method that is theoretically supported, highly interpretable, and can balance multi-dimensional indicators is a pressing technical problem to be solved in this field. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a visual language data quality assessment method based on structural decomposition theory.
[0008] The objective of this invention is achieved through the following technical solution: a visual language data quality assessment method based on structural decomposition theory, comprising the following steps: Step 1: When the data to be evaluated is input, the encoder E is used to extract the feature vectors of the image v and the text t in the data, and project them onto the unified representation space to obtain the image feature vector Ev and the text feature vector Et. Step 2: Use a large language model to evaluate and obtain the image integrity score I_img, and combine the text perplexity and semantic coherence evaluations to obtain the text integrity score I_text; Step 3: Calculate the cosine similarity between the image feature vector Ev and the text feature vector Et in the unified representation space, and introduce a length relaxation factor γ for long sequences to obtain the consistency score A of the data; Step 4: Calculate the image richness score V_img using the pixel intensity distribution entropy IH and the image integrity score I_img obtained in Step 2, and calculate the text richness score V_text based on the word-level entropy TH and the text length reward factor B_len; Step 5: Using a stability-aware fusion strategy, integrate the image integrity score I_img and the text integrity score I_text to obtain the data integrity score I, and integrate the image richness score V_img and the text richness score V_text to obtain the data richness score V. Step Six: Combine the integrity score I, richness score V, and consistency score A to calculate the overall quality assessment result Q of the data to be evaluated.
[0009] Furthermore, step two is implemented through the following sub-steps: (2.1) Calculate the text fluency score nromPPL: Calculate the text perplexity PPL of each sentence in the text using a pre-trained large language model, and obtain the normalized text fluency score normPPL by mapping the median operator to the exponent. (2.2) Calculate the local semantic similarity value avgLSS: Calculate the cosine similarity LSS between adjacent sentences using the text feature vector Et, and obtain the local semantic similarity value avgLSS by taking the average value; (2.3) Determine and calculate the text integrity score I_text: If the text contains multiple sentences, the text integrity score I_text is obtained by weighting the text fluency score nromPPL and the local semantic similarity score avgLSS; if the text is a single sentence, the text fluency score nromPPL is directly used as the text integrity score I_text. (2.4) Transfer integrity score: Transfer the image integrity score I_img to step four, and transfer the image integrity score I_img and the text integrity score I_text to step five.
[0010] Furthermore, in step five, the paired input dimension scores S_img and S_text are received as general calculation variables. When calculating the integrity score I, S_img and S_text correspond to the image integrity score I_img and the text integrity score I_text, respectively; when calculating the richness score V, S_img and S_text correspond to the image richness score V_img and the text richness score V_text, respectively.
[0011] Furthermore, step five is implemented through the following sub-steps: (5.1) Input dimension scores and distribute data: Receive the input dimension scores S_img and S_text, and distribute the data to the numerator and denominator paths for parallel processing through parallel routing; (5.2) Perform molecular path calculation: The data flows to the molecular path. First, the score is multiplied by S_img·S_text. Then, it is multiplied to obtain 2·S_img·S_text. Combined with the first stability constant C1, the molecular stability is accumulated. The molecular term is calculated as 2·S_img·S_text + C1. (5.3) Perform denominator path calculation: The data flows to the denominator path, calculate the squares of the two fractions S_img^2 and S_text^2 respectively, and obtain the sum of squares S_img^2+S_text^2. Then, combine the second stability constant C2 to perform denominator stability accumulation, and calculate the denominator term as S_img^2+S_text^2+C2. (5.4) Division operation and output evaluation result: Divide the numerator by the denominator and output the evaluation result.
[0012] Compared with the prior art, the beneficial effects of the present invention include at least the following: Systematically modeling evaluation dimensions based on structural decomposition theory provides mathematical support for evaluating complex cross-modal data, significantly enhancing the interpretability of the evaluation process. By introducing logical consistency checks and automated scoring processes, the entire evaluation task is automated, greatly improving the evaluation efficiency of large-scale datasets. Using a stability-aware fusion strategy, the impact of errors introduced by a single sub-evaluation module is mitigated, improving the robustness of the evaluation results. Combining heuristic methods with the reasoning capabilities of large language models supports fine-grained semantic evaluation while avoiding redundant calls to large language models, reducing the consumption of computational resources. Attached Figure Description
[0013] Figure 1 This is a flowchart of the data processing for the data to be evaluated in this method; Figure 2 This is a flowchart of a stability-aware fusion strategy. Detailed Implementation
[0014] The present invention will now be described in detail with reference to the accompanying drawings.
[0015] See Figure 1 This invention provides a method for assessing the quality of visual language data based on structural decomposition theory, comprising the following steps: Step 1: When the data to be evaluated is input, the encoder E is used to extract the feature vectors of image v and text t from the data, and project them onto a unified representation space to obtain image feature vector Ev and text feature vector Et, and then proceed to steps 2, 3 and 4. Step 2: Use a large language model to evaluate the degree of image distortion to obtain the image integrity score I_img, and scale the original score to the [0, 1] interval using a mapping function. At the same time, combine text perplexity and semantic coherence evaluation to obtain the text integrity score I_text, and proceed to steps 4 and 5; the calculation of I_text is divided into the following sub-steps.
[0016] (2.1) Calculate the text fluency score nromPPL: Calculate the text perplexity PPL of each sentence in the text using a pre-trained large language model, and obtain the normalized text fluency score normPPL by mapping the median operator to the exponent. (2.2) Calculate the local semantic similarity value avgLSS: Calculate the cosine similarity LSS between adjacent sentences using the text feature vector Et, and obtain the local semantic similarity value avgLSS by taking the average value; (2.3) Determine and calculate the text integrity score I_text: If the text contains multiple sentences, the text integrity score I_text is obtained by weighting the text fluency score nromPPL and the local semantic similarity score avgLSS; if the text is a single sentence, the text fluency score nromPPL is directly used as the text integrity score I_text. (2.4) Transfer Integrity Scores: Transfer the image integrity score I_img to step four, and transfer both the image integrity score I_img and the text integrity score I_text to step five. The purpose of setting up the above integrity assessment process is to identify and remove invalid data with fundamental defects in advance, thereby ensuring the basic usability of the data before conducting more complex information richness analysis.
[0017] Step 3: Calculate the cosine similarity between the image feature vector Ev and the text feature vector Et in the unified representation space. Then, scale the similarity using a scaling factor w and truncate negative values. First, use the max(0,·) operation to zero out negatively correlated components, then multiply it by the scaling factor w to amplify the value. Introduce a length relaxation factor γ for long sequences to correct the scaled result, compensating for the global similarity decay caused by long sequences. Calculate the consistency score A of the data and proceed to Step 6. Here, the scaling factor w is set to 2.5, and the length relaxation factor γ is a preset relaxation constant less than 1. Introducing this mechanism effectively ensures that the evaluation index can objectively capture the deep semantic relationships between complex long narratives and images, avoiding quality assessment bias caused by increased text length.
[0018] Step 4: Calculate the normalized information entropy IH of the image pixel intensity distribution, and perform gating processing on it using the image integrity score I_img obtained in Step 2 to obtain the image richness score V_img; calculate the normalized word-level information entropy TH of the text, and introduce a bounded length reward factor B_len based on the text character length l for compensation. The reward coefficient is calculated using the formula Blen(l) = 1 + η(1 - exp (-l / τ)) and multiplied by TH to alleviate the bias of short sequences in entropy estimation to obtain the text richness score V_text, and proceed to Step 5. The B_len is calculated using the saturation exponent form based on the character length l, the maximum gain constant η, and the feature length constant τ.
[0019] Step 5: See Figure 2 Using a stability-aware fusion strategy, the data integrity score I and richness score V are integrated to proceed to step six. In this fusion strategy, the paired input dimension scores S_img and S_text are used as general calculation variables, and their relationship with the specific evaluation dimensions is as follows: when calculating the integrity score I, S_img and S_text correspond to the image integrity score I_img and the text integrity score I_text, respectively; when calculating the richness score V, S_img and S_text correspond to the image richness score V_img and the text richness score V_text, respectively. The specific calculation process is divided into the following sub-steps.
[0020] (5.1) Input dimension scores and distribute data: Receive the input dimension scores S_img and S_text, and distribute the data to the numerator and denominator paths for parallel processing through parallel routing; (5.2) Perform molecular path calculation: The data flows to the molecular path. First, the score is multiplied by S_img · S_text. Then, it is multiplied to obtain 2·S_img·S_text. Combined with the stability constant C1, the molecular stability is accumulated. The molecular term is calculated as 2·S_img·S_text + C1. (5.3) Perform denominator path calculation: The data flows to the denominator path, and the squares of the two fractions S_img^2 and S_text^2 are calculated respectively. After obtaining the sum of squares S_img^2 + S_text^2, the denominator stability is accumulated in combination with the stability constant C2, and the denominator term is calculated as S_img^2 + S_text^2 + C2. (5.4) Division operation and output evaluation result: Divide the numerator by the denominator using division, and pass the output evaluation result to step six. This fusion strategy can effectively identify and penalize samples with significant imbalance between image and text quality.
[0021] Step Six: The integrity score I, richness score V, and consistency score A are directly fused using a product to generate a global comprehensive score ranging from [0, 1]. This score is the overall quality assessment result Q of the data to be evaluated. This fusion process uses a product instead of a weighted sum because the product is more sensitive to single-dimensional quality defects; failure in any dimension can lower the global score, thus more rigorously screening for high-quality samples. Furthermore, it eliminates the need to introduce additional parameters, improving the robustness of the framework.
[0022] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A method for assessing the quality of visual language data based on structural decomposition theory, characterized in that, Includes the following steps: Step 1: When the data to be evaluated is input, the encoder E is used to extract the feature vectors of the image v and the text t in the data, and project them onto the unified representation space to obtain the image feature vector Ev and the text feature vector Et. Step 2: Use a large language model to evaluate and obtain the image integrity score I_img, and combine the text perplexity and semantic coherence evaluations to obtain the text integrity score I_text; Step 3: Calculate the cosine similarity between the image feature vector Ev and the text feature vector Et in the unified representation space, and introduce a length relaxation factor γ for long sequences to obtain the consistency score A of the data; Step 4: Calculate the image richness score V_img using the pixel intensity distribution entropy IH and the image integrity score I_img obtained in Step 2, and calculate the text richness score V_text based on the word-level entropy TH and the text length reward factor B_len; Step 5: Using a stability-aware fusion strategy, integrate the image integrity score I_img and the text integrity score I_text to obtain the data integrity score I, and integrate the image richness score V_img and the text richness score V_text to obtain the data richness score V. Step Six: Combine the integrity score I, richness score V, and consistency score A to calculate the overall quality assessment result Q of the data to be evaluated.
2. The visual language data quality assessment method based on structural decomposition theory according to claim 1, characterized in that, Step two is achieved through the following sub-steps: (2.1) Calculate the text fluency score nromPPL: Calculate the text perplexity PPL of each sentence in the text using a pre-trained large language model, and obtain the normalized text fluency score normPPL by mapping the median operator to the exponent. (2.2) Calculate the local semantic similarity value avgLSS: Calculate the cosine similarity LSS between adjacent sentences using the text feature vector Et, and obtain the local semantic similarity value avgLSS by taking the average value; (2.3) Determine and calculate the text integrity score I_text: If the text contains multiple sentences, the text integrity score I_text is obtained by weighting the text fluency score nromPPL and the local semantic similarity score avgLSS; if the text is a single sentence, the text fluency score nromPPL is directly used as the text integrity score I_text. (2.4) Transfer integrity score: Transfer the image integrity score I_img to step four, and transfer the image integrity score I_img and the text integrity score I_text to step five.
3. The visual language data quality assessment method based on structural decomposition theory according to claim 1, characterized in that, In step five, the paired input dimension scores S_img and S_text are received as general calculation variables. When calculating the integrity score I, S_img and S_text correspond to the image integrity score I_img and the text integrity score I_text, respectively. When calculating the richness score V, S_img and S_text correspond to the image richness score V_img and the text richness score V_text, respectively.
4. The visual language data quality assessment method based on structural decomposition theory according to claim 1, characterized in that, Step five is achieved through the following sub-steps: (5.1) Input dimension scores and distribute data: Receive the input dimension scores S_img and S_text, and distribute the data to the numerator and denominator paths for parallel processing through parallel routing; (5.2) Perform molecular path calculation: The data flows to the molecular path. First, the score is multiplied by S_img·S_text. Then, it is multiplied to obtain 2·S_img·S_text. Combined with the first stability constant C1, the molecular stability is accumulated. The molecular term is calculated as 2·S_img·S_text + C1. (5.3) Perform denominator path calculation: The data flows to the denominator path, calculate the squares of the two fractions S_img^2 and S_text^2 respectively, and after obtaining the sum of squares S_img^2+S_text^2, combine it with the second stability constant C2 to perform denominator stability accumulation, and calculate the denominator term as S_img^2+S_text^2+C2; (5.4) Division operation and output evaluation result: Divide the numerator by the denominator and output the evaluation result.