Method and apparatus for quality assessment of image / video data

Abstract

The application relates to a quality evaluation method and device of image / video data, and relates to the technical field of image quality evaluation. The method comprises the following steps: receiving a media file to be evaluated; obtaining an objective score corresponding to the media file to be evaluated; inputting the media file to be evaluated and the objective score into a target evaluation model which has been trained, to obtain a subjective score corresponding to the media file to be evaluated; the target evaluation model is obtained by training an evaluation model according to sample media files, subjective scores of the sample media files and objective scores of the sample media files; the target evaluation model outputs a corresponding subjective score based on the input media file; the subjective score refers to the score of image quality given by a human observer; the objective score refers to the score of image quality given by a computer algorithm; and the subjective score is output. The application can meet the evaluation requirements of large scale and automation.

Description

Technical Field

[0001] This invention relates to the field of image quality evaluation technology, and in particular to a method and apparatus for quality assessment of image / video data. Background Technology

[0002] In the field of media file (images, videos, etc.) quality assessment, the reliability of the quality assessment results directly determines the effectiveness of image processing algorithm optimization, transmission protocol upgrades, and product quality control.

[0003] Current mainstream media quality assessment systems are divided into two main categories: subjective evaluation and objective evaluation. Subjective evaluation recruits human observers to qualitatively score media quality according to standardized experimental procedures, ultimately forming subjective scores such as Mean Opinion Score (MOS) and Difference Mean Opinion Score (DMOS). These scores directly reflect the human visual and auditory systems' perception of media quality, but they have inherent drawbacks such as high cost, low efficiency, and large subjective bias, making it difficult to meet the needs of large-scale, automated assessment. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for quality assessment of image / video data.

[0005] In a first aspect, this application provides a method for quality assessment of image / video data, the method comprising: Receive media files to be evaluated; Obtain the objective score corresponding to the media file to be evaluated; The media file to be evaluated and the objective score are input into the trained target evaluation model to obtain the subjective score corresponding to the media file to be evaluated. The target evaluation model is obtained by training the evaluation model based on the sample media file, the subjective score of the sample media file, and the objective score of the sample media file. The target evaluation model outputs the corresponding subjective score based on the input media file and the corresponding objective score. The subjective score refers to the score given by a human observer to the image quality. The objective score refers to the score given to the image quality through a computer algorithm. Output the subjective score corresponding to the media file to be evaluated.

[0006] This disclosure automatically calculates the objective score of the media file to be evaluated after receiving it, and then inputs the corresponding subjective score based on the media file to be evaluated and the objective score through the target evaluation model. This eliminates the need for manual subjective acquisition of subjective scores, improves the efficiency of subjective score acquisition, and meets the needs of large-scale and automated evaluation.

[0007] Optionally, the method further includes: A comprehensive score for the media file to be evaluated is obtained based on the objective and subjective scores corresponding to the media file to be evaluated. Output the overall score.

[0008] This disclosure generates a comprehensive score based on subjective scoring, thus organically integrating the advantages of subjective and objective evaluation technologies and improving the perceived accuracy and reliability of the evaluation results.

[0009] Optionally, the method further includes: Obtain a data sample set, which includes: differentiated sample media files and subjective and objective scores of the differentiated sample media files; The evaluation model is trained by using differentiated sample media files and corresponding objective scores as inputs, and corresponding subjective scores as labels for training the evaluation model, thereby obtaining the target evaluation model.

[0010] This disclosure provides a high-quality training data foundation covering multiple types and distortion states for the target evaluation model by constructing a data sample set containing differentiated sample media files and corresponding subjective and objective ratings. This ensures that the model can fully learn the correlation between the objective characteristics of media files and subjective perceptions in different scenarios. The training mode, which uses sample media files and objective ratings as inputs and subjective ratings as labels, enables the model to accurately capture the inherent mapping logic between subjective and objective data, effectively improving the accuracy of the model's output subjective ratings.

[0011] Optional, The subjective ratings include: average opinion score and difference average opinion score; The subjective rating corresponds to at least one of the following rating dimensions: absolute evaluation, relative evaluation, keyboard and mouse sensitivity evaluation, and audio and microphone evaluation.

[0012] Optional, The objective scoring corresponds to at least one of the following scoring dimensions: traditional indicators, structural similarity indicators, perceptual indicators, and audio indicators; The traditional metrics include: peak signal-to-noise ratio, mean square error, and mean absolute error. The structural similarity indicators include: structural similarity index and multi-scale structural similarity index; The perception metrics include: visual information fidelity, video multi-method evaluation fusion, and feature similarity index; The audio metrics include: perceptual objective hearing quality analysis and objective speech quality assessment.

[0013] Optionally, obtaining the data sample set includes: Obtain sample media files of different types; Different types and levels of distortion processing were applied to each sample media file to obtain multiple differentiated sample media files.

[0014] This disclosure constructs a differentiated sample set covering common content types and distortion scenarios in practical applications by acquiring different types of sample media files and performing multi-type and multi-level distortion processing, ensuring that the sample data has sufficient representativeness and diversity.

[0015] Optionally, the method further includes performing the following steps on each of the differentiated sample media files: Collect the raw scores for each subjective rating dimension of the current sample media file; Remove outliers from the original scores; The original scores after removal are normalized; The subjective score of the current media file is obtained based on the normalized original score.

[0016] This disclosure effectively eliminates the interference of abnormal data such as extreme scores and misscored scores on the scoring results by removing outliers from the original subjective scores before calculating the final subjective scores, thus ensuring the statistical validity and reliability of the subjective scores.

[0017] Optionally, the method further includes: The quality level of the media file to be evaluated is obtained based on the comprehensive score. Obtain the proportion of subjective and objective scores in the overall score for the media file to be evaluated; Output the quality level and the percentage performance.

[0018] This disclosure transforms abstract scoring data into intuitive evaluation results by outputting quality levels, correlations between subjective and objective dimensions, and percentage performance. This solves the problems of existing technologies where evaluation results are highly professional, have high interpretation barriers, and are difficult to support business decisions.

[0019] Optionally, obtaining the comprehensive score of the media file to be evaluated based on objective and subjective scores includes: Based on a preset weighting strategy, the comprehensive score of the media file to be evaluated is calculated by combining the objective score and the subjective score.

[0020] This disclosure employs a preset weighting strategy to calculate the comprehensive score. By flexibly adjusting the weight ratio of subjective and objective scores, it achieves a dynamic balance between evaluation efficiency and perception accuracy, overcoming the limitations of existing technologies where fixed weights cannot adapt to different business stages and scenario requirements.

[0021] Secondly, this application provides an image / video data quality assessment device, the device comprising: The receiving module is used to receive the media files to be evaluated; The first acquisition module is used to acquire the objective score corresponding to the media file to be evaluated; The second acquisition module is used to input the media file to be evaluated and the objective score into the trained target evaluation model to obtain the subjective score corresponding to the media file to be evaluated. The target evaluation model is obtained by training the evaluation model based on the sample media file, the subjective score of the sample media file, and the objective score of the sample media file. The target evaluation model outputs the corresponding subjective score based on the input media file and the corresponding objective score. The subjective score refers to the score given by a human observer to the image quality. The objective score refers to the score given to the image quality by a computer algorithm. The first output module is used to output the subjective score corresponding to the media file to be evaluated.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of an image / video data quality assessment method provided in an embodiment of this application Figure 1 ; Figure 2 A flowchart of an image / video data quality assessment method provided in an embodiment of this application Figure 2 ; Figure 3 A schematic diagram illustrating specific parameters of the ITU-R BT.500-13 standard provided in an embodiment of this application; Figure 4 A functional module diagram of an image / video data quality assessment device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0026] Figure 1 A flowchart illustrating a method for quality assessment of image / video data provided in one embodiment of this application is shown below. Figure 1 As shown, the method includes the following steps S101-S104: In step S101, the media file to be evaluated is received.

[0027] The media files to be evaluated here can include still images (such as office documents, medical images, 3D images, etc.) and dynamic videos (such as cloud desktop operation videos, movie clips, VR videos, etc.), and multimedia files containing audio are supported.

[0028] In one embodiment, the file may contain basic format information (such as resolution, encoding format, duration, etc.), and if it is a video file, it may also include complete parameters such as frame rate and bit rate, which facilitates accurate calculation of subsequent objective indicators; it supports files with distortion types such as JPEG compression, Gaussian blur, and network packet loss, adapting to the quality assessment needs in actual application scenarios.

[0029] In step S102, the objective score corresponding to the media file to be evaluated is obtained.

[0030] The computer algorithm automatically calculates objective indicators of the media file to be evaluated across multiple dimensions and integrates them into an objective score.

[0031] In one embodiment, the scoring dimensions corresponding to the objective score include at least one of the following scoring dimensions: traditional indicators, structural similarity indicators, perceptual indicators, and audio indicators; Traditional metrics include: Peak Signal-to-Noise Ratio (PSNR) (a traditional objective metric for image quality, which measures the degree of image distortion by calculating the pixel error between the original image and the processed image; a higher value indicates better quality), Mean Square Error (MSE) (a traditional image error metric, which calculates the average of the squared differences between corresponding pixels in the original image and the processed image; a lower value indicates less image distortion), and Mean Absolute Error (MAE) (a traditional image error metric, which calculates the average of the absolute values ​​of the differences between corresponding pixels in the original image and the processed image, reflecting the overall error level of the image). Structural similarity metrics include: Structural Similarity Index (SSIM) (based on the human visual sensitivity to image structure, it measures image similarity from three dimensions: brightness, contrast, and structure; the closer the value is to 1, the better the quality) and Multi-Scale Structural Similarity Index (MS-SSIM) (an extension of SSIM, which analyzes image structure at different scales to better reflect the human eye's perception of images at different resolutions). Perceptual metrics include: Visual Information Fidelity (VIF) (an objective perceptual metric based on information theory, measuring the degree to which a processed image retains the visual information of the original image; a higher value indicates better fidelity), Visual Multimethod Assessment Fusion (VMAF) (a mainstream objective video quality metric that integrates multiple image quality assessment models, simulates human visual perception, and is suitable for video quality evaluation in multiple scenarios), and Feature Similarity Index (FSIM) (an objective perceptual metric that extracts features such as phase consistency and gradient magnitude from images to measure the feature similarity between two images). Audio metrics include: Perceptual Objective Listening Quality Analysis (POLQA) (an objective audio quality metric that simulates the human auditory system, quantifies sound quality loss in voice communication, and is applicable to scenarios such as network voice and cloud desktop audio) and Visual Speech Quality Objective Listener (ViSQOL) (an objective audio quality metric that is particularly good at quality evaluation in scenarios where music and speech are mixed, and is open source and adapted for academic research and product development).

[0032] For example, PSNR, MSE, and MAE can be calculated using tools such as OpenCV to measure pixel-level errors; SSIM and MS-SSIM can be calculated using tools such as OpenCV; VIF, VMAF, and FSIM can be calculated using libraries such as Netflix VMAF to simulate the visual perception characteristics of the human eye; and POLQA and ViSQOL can be calculated using tools such as POLQA SDK and Librosa to evaluate media files containing audio.

[0033] The objective scores of multiple dimensions are "standardized (eliminating differences in units) and deduplicated (e.g., removing redundant indicators with a correlation > 0.9 with VMAF)" to obtain the objective score (O-Score) input into the target evaluation model.

[0034] For example: The calculation formula is: O-Score = ×2×20%+E×100×30%+F×40%+G×10%, where, E represents the normalized score of traditional metrics (PSNR / MSE / MAE), E represents the score of structural similarity metrics (SSIM / MS-SSIM), F represents the score of perceptual metrics (VIF / VMAF / FSIM), and G represents the score of audio metrics (POLQA / ViSQOL).

[0035] in, The role of the coefficient 2 in ×2×20% is: It is the normalized score of the traditional indicator: Traditional metrics have significantly different native ranges (e.g., PSNR is typically 20-50 dB, while MSE can range from 0 to several hundred dB). After "standardization," The range of values ​​is normalized to 0-1 (highest score 1, lowest score 0).

[0036] The purpose of multiplying by "2": The normalized score of 0-1 is mapped to the range of 0-2, and then multiplied by 20% weight (2×20%=0.4). The final contribution range of the traditional index in the O-Score is 0-0.4, ensuring that its weight ratio is consistent with the design (20%) and will not be ignored due to its small original magnitude.

[0037] The function of the coefficient 100 in E×100×30% is: E is the score of the structural similarity index (SSIM / MS-SSIM): The native value range of SSIM / MS-SSIM is 0-1 (the closer to 1, the more similar the structure and the better the quality), and no additional normalization is required.

[0038] The purpose of multiplying by "100": The original score of 0-1 is mapped to the range of 0-100, and then multiplied by 30% weight (100×30%=30). The final contribution range of the structural similarity index in the O-Score is 0-30. The magnitude of the mapping is matched with that of other indicators, ensuring that the weight ratio of each indicator is accurately applied when the weighted sum is calculated.

[0039] In this disclosure, since the objective scores are obtained from multiple dimensions, the objective scores are also weighted and summed based on the weights corresponding to each preset scoring dimension. The objective scores input into the target evaluation model are the weighted sum values.

[0040] In step S103, the media file to be evaluated and the multi-dimensional objective scores are input into the trained target evaluation model to obtain the subjective scores corresponding to the media file to be evaluated. The target evaluation model is obtained by training the evaluation model based on the sample media file, the subjective scores of the sample media file, and the objective scores of the sample media file. The target evaluation model outputs the corresponding subjective scores based on the input media file and the corresponding objective scores. Subjective scores refer to the scores given by human observers to image quality. Objective scores refer to the scores given to image quality by computer algorithms.

[0041] In this disclosure, the subjective score output by the target evaluation model is a comprehensive subjective score, namely the S-Score in the following embodiments, rather than the subjective scores corresponding to each scoring dimension.

[0042] In one embodiment, this disclosure also provides a training process for the evaluation model, such as... Figure 2 As shown, the training process includes the following sub-steps S201-S202: S201. Obtain a data sample set, which includes: differentiated sample media files and subjective and objective scores of the differentiated sample media files.

[0043] In one embodiment, subjective ratings include: Mean Opinion Score (MOS) and Differential Mean Opinion Score (DMOS); MOS is the core subjective evaluation metric, which quantifies the subjective perceived quality of an image / audio by statistically averaging the ratings of multiple observers, and is mostly used for relative evaluation; DMOS is a subjective evaluation metric, which quantifies the degree of image quality loss by comparing the rating differences between the original image and the processed image, and is mostly used for absolute evaluation.

[0044] The subjective rating should include at least one of the following rating dimensions: absolute evaluation, relative evaluation, keyboard and mouse sensitivity evaluation, and audio and microphone evaluation.

[0045] Subjective scoring methods include: The Double Stimulus Impairment Scale (DSIS) is a subjective assessment method that compares the original image with the damaged image, allowing observers to rate the degree of image damage. The Double Stimulus Continuous Quality Scale (DSCOS) is a subjective evaluation method that uses a continuous scale to allow observers to continuously evaluate the quality differences between two sets of images. Single Stimulus Continuous Quality Evaluation (SSCQE) is a subjective evaluation method that does not require the original image as a reference, allowing the observer to directly evaluate the quality using only a single image to be tested. The Double Stimulus Continuous Quality Scale (DSCQS) is often used for subjective absolute evaluation. It alternates between playing the original image and the image to be tested, and the observer scores the image according to a fixed scale.

[0046] In one embodiment, the method further includes performing the following steps A1-A4 on each of the differentiated sample media files: A1. Collect the raw scores for each subjective rating dimension of the current sample media file.

[0047] A2. Remove outliers from the original scores.

[0048] A3. Normalize the original scores after removal; A4. Obtain the subjective score of the current media file based on the normalized raw score.

[0049] For example, obtaining a subjective rating for the current media file based on the normalized raw rating may include: The normalized raw scores are statistically averaged to obtain the subjective score of the current media file.

[0050] Specifically, the original scores undergo "outlier removal (e.g., 3σ principle), normalization (mapping to 0-100 points), and statistical averaging" to obtain the subjective score (S-Score). The calculation formula is: S-Score = A×10×20% + B×10×20% + C×10×30% + D×10×20%, where A is the relative rating (SSCQE) score, B is the absolute rating (DSCQS) score, C is the keyboard and mouse sensitivity rating score, and D is the audio and microphone rating score. Each item uses a 5-point scale (corresponding to 9-10, 7-8). The formula uses a coefficient of 10 to map the original 5-point scores of each subjective rating dimension to a unified quantification range compatible with the percentage system (0-100 points). (Multiplying the 5-point score (1-5 points) by 10 will linearly map it to the range of 10-50 points (or the broader range of 0-100 points)) to ensure the accuracy of the weights of each dimension when performing subsequent weighted summation. For example: 5 points (highest level) × 10 = 50 points, 4 points (medium to high level) × 10 = 40 points, 3 points (medium level) × 10 = 30 points.

[0051] When training the evaluation model, the input to the model is the O-Score corresponding to the sample media file and the sample media file itself, and the output of the evaluation model is the predicted S-Score.

[0052] In one embodiment, to increase the variability of the samples, a data sample set is obtained, including B1-B2: B1. Obtain sample media files of different types; B2. Perform different types and levels of distortion processing on each sample media sample file to obtain multiple differentiated sample media files.

[0053] Specifically, images / videos covering more than 8 categories, including "faces, natural scenery, office documents (Word / Excel / PPT), architecture, 3D models, maps, dynamic line drawings, and medical images," can be collected. For each category, 5 types of distortion can be added: "JPEG compression (quality factor 10-90), Gaussian blur (standard deviation 0.5-5.0), salt and pepper noise (density 0.01-0.1), contrast change (0.2-2.0 times), and motion blur (kernel size 3-15)." Each type of distortion can be set with 5-8 intensity levels. The total sample size should be ≥1500 images or ≥500 video clips (each clip is 10-30 seconds long).

[0054] S202. Using differentiated sample media files and corresponding objective scores as inputs to the evaluation model, and using corresponding subjective scores as labels when training the evaluation model, the evaluation model is trained to obtain the target evaluation model.

[0055] In one embodiment, the evaluation model disclosed herein may include random forest, gradient boosting tree (XGBoost), or lightweight neural network (such as MLP) to balance prediction accuracy and computational efficiency.

[0056] During training, 70% of the data is divided into a training set, 20% into a validation set, and 10% into a test set. Model parameters (such as tree depth and learning rate) are optimized through "cross-validation".

[0057] When the evaluation model is a random forest, the specific training process is as follows: 1. Initialize hyperparameters: Set the number of trees (50–500), maximum depth (3–20), etc. Adjust the tree depth to capture feature associations for the non-linear relationship between objective and subjective scores of different media files (such as video / audio / text and images), and control the number of trees to balance "adaptability to multiple types of media files" and "computational efficiency". 2. Single tree training: Perform sampling with replacement on the media file training set and randomly select nodes to obtain objective rating features. The MSE minimum split is used to avoid the model overfitting to extreme subjective ratings of a certain type of media file. The MSE split criterion directly aims to "reduce the prediction error (RMSE) of subjective ratings of media files". 3. Parallel construction of multiple trees: Independently train multiple trees and integrate the results (average output) to reduce the randomness of a single tree's subjective rating prediction for a certain type of media file and improve the Pearson correlation coefficient between objective and subjective ratings. 4. Cross-validation parameter tuning: Select the parameters with the highest Pearson score and RMSE≤5 in the validation set, directly align with the evaluation requirements of media file rating prediction, and screen out the parameter combination that can best map the objective score to the subjective score. 5. Full training set training + validation: Train with optimal parameters, and evaluate the prediction effect of different media files with the validation set to ensure that the model meets the subjective rating prediction standards for various differentiated media files, thus forming the basic version of the target evaluation model.

[0058] When evaluating the XGBoost model, the specific training process is as follows: 1. Initialize hyperparameters: Set the learning rate (0.01–0.3), regularization coefficient, etc. The learning rate controls the correction magnitude of each tree to the "media file subjective rating residual" to avoid overfitting to a certain type of media file; the regularization coefficient constrains the weights to prevent the model from being overly sensitive to individual objective ratings (such as image quality) and to ensure the stability of subjective rating predictions for different media files. 2. Initialize the model: Use the mean subjective rating of the media files in the training set as the initial prediction value to give the model a basic baseline. Then, correct the bias tree by tree to quickly reduce the initial RMSE of subjective ratings of different media files. 3. Serial Iterative Training: Fit the subjective rating residuals of media files one by one, split according to gradient + Hessian matrix, accurately capture the interaction relationship of multi-dimensional objective ratings (such as the combined influence of picture quality and smoothness on video subjective ratings), and reduce the prediction error of subjective ratings of different media files in each round to improve the Pearson coefficient. 4. Cross-validation parameter tuning and early stopping: Select parameters with Pearson ≥ 0.85, RMSE ≤ 5 and the fewest number of trees on the validation set. Stop early if the validation set error does not decrease. This ensures the prediction accuracy of subjective ratings for various media files while avoiding excessive model complexity, thus balancing "multi-file type adaptation" and "computational efficiency". 5. Validation and Evaluation: The validation set predicts subjective ratings for different media files, confirming that the model can accurately adapt to the rating patterns of various differentiated media files, thus forming the core version of the target evaluation model.

[0059] When the evaluation model is an MLP, the specific training process is as follows: 1. Network and hyperparameter initialization: Design a lightweight structure (input layer = media objective rating dimension, 1-3 hidden layers, output 1 neuron) + set learning rate / Dropout, etc. The lightweight structure is adapted to the characteristics of media file data volume. Dropout / regularization avoids the model from remembering the rating noise of a certain type of media file and ensures generalization ability. 2. Weight initialization: Xavier initialization ensures gradient stability and avoids extreme bias in subjective rating predictions of special media files in the early stages of training; 3. Forward propagation: Objectively score the input media files in batches, calculate the MSE loss, and train in batches to adapt to the batch processing of a large number of differentiated media files. The MSE loss directly points to "reducing the RMSE of subjective scores of all media files". 4. Backpropagation and parameter update: The Adam optimizer updates weights to efficiently optimize the complex mapping relationship between multi-dimensional objective scores and subjective scores (such as the audio quality and sound field localization jointly determining the subjective score), and improve the Pearson correlation coefficient. 5. Cross-validation and early stopping parameter tuning: Monitor the validation set loss. If it continues to rise, stop the model early to avoid overfitting the model to specific media files in the training set and ensure that subjective scores can be accurately predicted even for unseen media files. 6. Validation and Evaluation: The validation set predicts subjective ratings for various media files, confirming that the model can capture complex relationships and meet the evaluation requirements, thus forming a supplementary target evaluation model.

[0060] In step S104, the subjective score corresponding to the media file to be evaluated is output.

[0061] In one embodiment, the method further includes the following steps C1-C2: C1. Obtain the comprehensive score (C-Score) of the media file to be evaluated based on the objective and subjective scores corresponding to the media file to be evaluated.

[0062] Specifically, a comprehensive score for the media file to be evaluated can be calculated based on a preset weighting strategy, combining objective and subjective scores.

[0063] The formula for calculating the overall score is: C-Score = α × (O-Score) + β × (S-Score), where α and β are weighting coefficients (α + β = 1).

[0064] Initially, α=0.7 and β=0.3 can be set (relying on objective data to ensure efficiency), and later adjusted to α=0.5 and β=0.5 as subjective data accumulates (balancing accuracy and efficiency).

[0065] C2. Output the overall score.

[0066] In one embodiment, this disclosure can also be visualized, in which case the method further includes the following steps D1-D3: D1. Obtain the quality level of the media file to be evaluated based on the comprehensive score.

[0067] For example, the grading criteria (pre-set in the system) are as follows: C-Score ≥ 90 is "Excellent", 70-89 is "Good", 50-69 is "Average", 30-49 is "Poor", and ≤ 29 is "Very Poor".

[0068] In this disclosure, the system automatically matches: if a video's C-Score is 92, it will automatically determine... The score is "Excellent"; C-Score = 58 points, which is judged as "Average".

[0069] D2. Obtain the proportion of subjective and objective scores in the overall score for the media files to be evaluated.

[0070] The calculation combines weighting coefficients (α, β): For example, if the current weights are α=0.6 (objective) and β=0.4 (subjective), and a media file has an O-Score of 80 and an S-Score of 75, the overall score C-Score is 0.6 × 80 + 0.4 × 75 = 78. The objective score contributes 0.6 × 80 = 48 to the C-Score, accounting for approximately 61.5% (48 ÷ 78); the subjective score contributes 0.4 × 75 = 30, accounting for approximately 38.5% (30 ÷ 78).

[0071] D3, Output quality level and percentage performance.

[0072] It directly displays "Excellent / Good / Average / Poor / Very Poor" along with the corresponding C-Score value (e.g., "Quality Level: Excellent (C-Score=92)").

[0073] A "correlation heatmap" can be used to display correlations, with different colors representing the strength of the correlation (e.g., dark red = strong correlation, light blue = weak correlation).

[0074] A "multidimensional quality radar chart" can be used to display the proportion of performance. The radar chart uses axes of different dimensions to represent subjective and objective proportions.

[0075] In one embodiment, the correlation between the rating dimensions of each subjective rating and the rating dimensions of each objective rating in the samples used by the model during training can also be obtained.

[0076] Using Pearson correlation coefficient or Spearman rank correlation coefficient, for example, to calculate the correlation between "structural similarity index (SSIM)" and "subjective absolute evaluation (image quality)", if the result is 0.85, it indicates that the two are strongly correlated, that is, the higher the SSIM index, the better the image quality perceived by humans.

[0077] The final result is an index correlation matrix. For example, the correlation between the perceptual index (VMAF) and the subjective relative evaluation is 0.88, and the correlation between the traditional index (PSNR) and the keyboard and mouse sensitivity evaluation is 0.2 (weak correlation).

[0078] This disclosure can generate a visual report containing all the above information, and also supports integration with ITSM and CMDB systems to use the results for product optimization decisions (such as "VMAF has low correlation with subjective factors in a certain scenario, and objective indicator selection needs to be optimized"), and then integrate and output the final visual dashboard.

[0079] It can also output a C-Score distribution histogram.

[0080] When outputting the C-Score distribution histogram, first create a canvas, then draw the histogram on the canvas: the X-axis represents the C-Score range, the Y-axis represents the number of samples, and finally use different colors to distinguish different distribution ranges.

[0081] When outputting a heatmap of correlation between indicators, first create a canvas, then draw the heatmap, label the values ​​and use color gradients to represent the strength of the correlation.

[0082] When outputting a multi-dimensional quality radar chart to represent the proportion of output, first create a canvas, draw the radar chart, and the radar chart shows the subjective proportion and the objective proportion.

[0083] Final integration layout and rendering: Arrange the three canvases on the same canvas, add a general title, legend, and necessary text descriptions to generate the final visualization image file, and output the visualization file so that users can see the evaluation results.

[0084] Image / video data quality assessment is a core means of measuring the performance of image processing algorithms and optimizing system parameters. Existing assessment methods are mainly divided into subjective and objective assessments, but both have significant drawbacks: 1. Limitations of subjective assessment methods: Subjective assessment relies on human observers to qualitatively score image quality (such as the Dual Stimulus Impairment Scale (DSIS), Dual Stimulus Continuous Quality Scale (DSCOS), and Single Stimulus Continuous Quality Assessment (SSCQE), which requires a fixed process of "dataset preparation - observer invitation - score processing." Its advantage is that it closely matches human visual perception, but it suffers from high cost, low efficiency, and large subjective bias—for example, it requires recruiting more than 50 raters, a single sample assessment takes more than 30 seconds, and multiple ratings are easily influenced by factors such as observer experience, emotions, and ambient lighting, making it difficult to meet the needs of large-scale, automated quality assessment. This disclosure establishes a direct mapping relationship between the media file to be evaluated, objective scores, and subjective scores through a target assessment model, eliminating the need for repeated large-scale manual subjective experiments and quickly outputting subjective scores that match human vision and usage perception. This technical solution effectively solves the technical problems of high cost, low efficiency and significant individual bias caused by the reliance on manual labor in traditional subjective evaluation. At the same time, it retains the core advantage of subjective ratings reflecting the user's real experience, significantly improves the efficiency and stability of subjective perception ratings, and can efficiently support the large-scale automated quality assessment needs in multiple fields such as image encoding and decoding and cloud desktop transmission.

[0085] 2. Limitations of Objective Evaluation Methods: Objective evaluation automatically calculates quality quantification values ​​through computer algorithms (such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Video Multi-Method Evaluation Fusion (VMAF), offering advantages such as high efficiency and unbiasedness. However, it suffers from low fidelity to human subjective perception. For example, PSNR only measures pixel-level errors and cannot reflect the human eye's sensitivity to "structural distortion"; SSIM has high computational complexity and relies on the original reference image; a single objective indicator cannot cover multiple dimensions of experience such as "image sharpness, audio synchronization, and keyboard and mouse operation smoothness," leading to a disconnect between evaluation results and actual user experience. This disclosure constructs a multi-dimensional objective indicator system encompassing traditional indicators, structural similarity indicators, perceptual indicators, and audio indicators. This system comprehensively covers core dimensions such as pixel-level error, image structure, visual perception, and audio quality, overcoming the shortcomings of single objective indicators in terms of coverage. After preprocessing such as standardization and deduplication, the multi-dimensional objective indicators are input into machine learning models such as XGBoost, which are trained using subjective ratings (MOS / DMOS) as labels. This establishes a precise mapping relationship between objective features and subjective perception, addressing the issues of PSNR only measuring pixel error and SSIM relying on reference images, which are disconnected from human perception. Finally, a dynamic weighting strategy is used to calculate a comprehensive score. This approach retains the efficiency of objective evaluation while ensuring that the evaluation results closely match the user's actual experience through subjective perception mapping, effectively solving the technical challenge of low alignment between objective evaluation and human subjective perception.

[0086] Specifically: 1. High perceptual accuracy: By integrating multi-dimensional subjective and objective data through machine learning models, the problem of "disconnect between objective indicators and human perception" is solved. The Pearson correlation coefficient between the comprehensive score (C-Score) and the subjective MOS is ≥0.85, and the evaluation results are more in line with the actual user experience.

[0087] 2. High automation efficiency: After the model is trained, only the medium to be tested needs to be input to automatically output the comprehensive score. No human intervention is required. The evaluation efficiency is more than 100 times higher than that of purely subjective evaluation, which can meet the needs of large-scale batch evaluation.

[0088] 3. Strong scene adaptability: The dataset covers multiple scenarios such as "office, medical, film and television, VR", supports multi-dimensional evaluation of "image, video, audio, keyboard and mouse operation", and can be adapted to various application scenarios such as VGTP protocol encoding and decoding, cloud desktop transmission.

[0089] 4. Good flexibility and scalability: The subjective and objective weights (α, β) can be adjusted according to business needs, and it supports the addition of new objective indicators (such as indicators for AI-generated images) or subjective dimensions (such as VR dizziness evaluation), which facilitates subsequent function iterations.

[0090] 5. High practical value: The output of visual reports and root cause analysis rules (such as "high latency + high packet loss → network problem") can directly provide direction for product optimization and reduce R&D decision-making costs.

[0091] Taking "VGTP Protocol Cloud Desktop Video Quality Assessment" as an example, the specific steps are as follows: 1. Hardware environment configuration: Data acquisition terminal: high-performance server (CPU: Intel Xeon Gold 6330, GPU: NVIDIA A100, memory: 128GB, storage: 2TB SSD), used to store datasets and run objective indicator calculation tools.

[0092] Subjective testing: 20 standard testing terminals (monitors: 27-inch 4K IPS screen, resolution 3840×2160, color gamut 100% sRGB; audio equipment: monitoring headphones (frequency response 20-20000Hz)), conforming to ITU-R BT.500-13 standard, specific parameters are as follows... Figure 3 As shown.

[0093] Deployment end: Edge computing node (CPU: Intel Core i7-12700, memory: 32GB), used to run automated evaluation tools, supporting the processing of 10-20 images or 2-3 video clips per second.

[0094] 2. Software tool selection: Dataset management: LabelStudio (for labeling image content and distortion types), MySQL (for storing subjective and objective data).

[0095] Objective metric calculation: Objective metrics for each sample are automatically calculated using open-source libraries (such as OpenCV, FFmpeg, Netflix VMAF library, and the official POLQA SDK). PSNR: Based on RGB or YUV444 format, the calculation formula is (PSNR=10\log_{10}\frac{(2^n-1)^2}{MSE}) (n is the number of pixel bits, usually taken as 8).

[0096] SSIM: Based on three elements: luminance (L), contrast (C), and structure (S), the formula is (SSIM=\frac{(2\mu_x\mu_y+C_1)(2\sigma_{xy}+C_2)}{(\mu_x^2+\mu_y^2+C_1)(\sigma_x^2+\sigma_y^2+C_2)}) ((C_1, C_2) are constants to avoid the denominator being 0).

[0097] VMAF: Calls Netflix VMAF version 4.0+, using the default model (vmaf_v0.6.1.pkl), and outputs a score of 0-100.

[0098] POLQA: Compliant with ETSI TS 103 281 standard, output score 1.0-5.0 (mapped to 0-100).

[0099] Model training: Python 3.9+, Scikit-learn (data preprocessing and evaluation), XGBoost (model training), TensorFlow (neural network implementation).

[0100] Visualization and Deployment: Streamlit (visual interface), Docker (containerized deployment), RESTful API (interfacing with external systems).

[0101] 3. Dataset Construction: Collect videos from 5 scenarios: "static desktop, Word operation, PPT playback, medical image rotation, and JD.com web browsing". Add 2 types of distortion to each video category: "JPEG compression (quality factor 20 / 40 / 60 / 80) and network packet loss (1% / 3% / 5% / 8%)", for a total of 5×4×4=80 video segments (20 seconds each).

[0102] 4. Subjective Experiment: Recruit 50 raters and use the DSCQS method. Each video is displayed for 30 seconds. The rating dimensions include "absolute evaluation (image quality), keyboard and mouse sensitivity, and audio synchronization". Outlier removal is performed on the original ratings (ratings that deviate from the mean by 3σ). The MOS score (S-Score) of each video is calculated.

[0103] 5. Objective index calculation: Automatically calculate PSNR (average 35.2dB), SSIM (average 0.92), VMAF (average 88), and POLQA (average 4.3) for each video segment. Perform Z-score standardization on the objective indexes ((z=\frac{x-\mu}{\sigma})), remove weakly correlated indexes with a correlation <0.3 with the label (S-Score), and obtain the O-Score after standardization.

[0104] 6. Model Training: Train the XGBoost model using objective metrics as input and S-Score as label. Set the parameters as follows: max_depth=6, learning_rate=0.1, n_estimators=100, subsample=0.8, and the test set Pearson correlation coefficient=0.88, RMSE=4.2.

[0105] 7. Comprehensive evaluation: With α=0.6 and β=0.4, the C-Score was calculated. The C-Score for "no distortion video" was 92 ("excellent"), and the C-Score for "8% packet loss video" was 58 ("average"). These results are consistent with actual user experience and verify the effectiveness of this disclosure.

[0106] This embodiment uses VGTP protocol cloud desktop video quality assessment as a scenario for instance verification. A dataset of 80 videos covering five types of cloud desktop scenarios, including static desktops and Word operations, and incorporating two types of distortion: JPEG encoding compression and network packet loss, was constructed. Fifty qualified raters conducted subjective experiments using the DSCQS method, evaluating aspects such as absolute rating and keyboard / mouse sensitivity. Simultaneously, four objective indicators, including PSNR and SSIM, were automatically calculated. An XGBoost model was used for training (test set Pearson correlation coefficient 0.88, RMSE 4.2, both meeting standards). The C-score was calculated with α=0.6 and β=0.4. Results show that the C-scores of samples with different levels of loss are highly consistent with actual user perception (C-score 92 for no-loss samples is "excellent," and C-score 41 for extremely high-loss samples is "very poor"). Verification demonstrates that the system accurately reflects user experience, with automated assessment efficiency 750 times higher than manual assessment. It is adaptable to multiple cloud desktop scenarios and can pinpoint quality bottlenecks through assessment results, providing quantitative evidence for product optimization.

[0107] In summary, this invention proposes a subjective and objective rating fusion evaluation system and method for image / video data quality, aiming to solve the technical defects of traditional subjective evaluation (high cost), objective evaluation (low consistency with human perception), and existing fusion schemes (crude). The system adopts a four-layer architecture: data acquisition layer, feature extraction layer, model fusion layer, and evaluation output layer. The data acquisition layer simultaneously collects subjective data (MOS / DMOS) covering four dimensions, including absolute evaluation (DSCQS) and relative evaluation (SSCQE), as well as objective data from multiple dimensions such as traditional indicators (PSNR / MSE) and structural similarity indicators (SSIM / MS-SSIM). The feature extraction layer generates subjective scores (S-Score) and objective scores (O-Score) through preprocessing such as outlier removal and standardization. The model fusion layer uses machine learning models such as XGBoost to construct a mapping relationship between objective indicators and subjective perception (Pearson correlation coefficient ≥ 0.85 and RMSE ≤ 5 on the test set) and calculates a comprehensive score (C-Score) with dynamic weights (α, β). The evaluation output layer outputs quality levels and visual reports, and can also be integrated with external systems such as ITSM and CMDB. It can also provide root cause analysis and decision support for product optimization. The solution is implemented in four stages: data preparation (constructing a sample set with multiple contents and distortion types, and designing subjective experiments that conform to the ITU-R BT.500-13 standard), index calculation, model training and verification, and system deployment and iteration. Combined with VGTP protocol cloud desktop scenario instance verification (covering 5 types of scenarios and 80 samples, with automated evaluation efficiency 750 times higher than manual evaluation), the system can accurately match the actual user experience and adapt to multiple fields such as image acquisition, encoding and decoding, and cloud desktops, providing quantitative basis for product optimization. It has high perception accuracy, high automation efficiency, and strong scenario adaptability.

[0108] Based on the same inventive concept, this application also provides an image / video data quality assessment apparatus for implementing the image / video data quality assessment method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image / video data quality assessment apparatus embodiments provided below can be found in the limitations of the image / video data quality assessment method described above, and will not be repeated here.

[0109] Figure 4 A functional module diagram of an image / video data quality assessment device provided in an embodiment of this application is shown below. Figure 4 As shown, the device includes: The receiving module is used to receive the media files to be evaluated; The first acquisition module is used to acquire the objective score corresponding to the media file to be evaluated; The second acquisition module is used to input the media file to be evaluated and the objective score into a trained target evaluation model to obtain the subjective score corresponding to the media file to be evaluated. The target evaluation model is obtained by training an evaluation model based on sample media files, the subjective scores of sample media files, and the objective scores of sample media files. The target evaluation model outputs the corresponding subjective score based on the input media file to be evaluated and the corresponding objective score. The subjective score refers to the score given by a human observer to the image quality. The objective score refers to the score given to the image quality through a computer algorithm. The first output module is used to output the subjective score corresponding to the media file to be evaluated.

[0110] In one embodiment, the apparatus further includes: The third acquisition module is used to obtain the comprehensive score of the media file to be evaluated based on the objective score and subjective score corresponding to the media file to be evaluated. The second output module is used to output the comprehensive score.

[0111] In one embodiment, the apparatus further includes: The fourth acquisition module is used to acquire a data sample set, which includes: differentiated sample media files and subjective and objective scores corresponding to the differentiated sample media files; The training module is used to train the evaluation model by taking the differentiated sample media files and the corresponding objective scores as inputs to the evaluation model, and the corresponding subjective scores as labels for training the evaluation model, so as to obtain the target evaluation model.

[0112] In one embodiment, The subjective ratings include: average opinion score and difference average opinion score; The subjective rating corresponds to at least one of the following rating dimensions: absolute evaluation, relative evaluation, keyboard and mouse sensitivity evaluation, and audio and microphone evaluation.

[0113] In one embodiment, The objective scoring corresponds to at least one of the following scoring dimensions: traditional indicators, structural similarity indicators, perceptual indicators, and audio indicators; The traditional metrics include: peak signal-to-noise ratio, mean square error, and mean absolute error. The structural similarity indicators include: structural similarity index and multi-scale structural similarity index; The perception metrics include: visual information fidelity, video multi-method evaluation fusion, and feature similarity index; The audio metrics include: perceptual objective hearing quality analysis and objective speech quality assessment.

[0114] In one embodiment, in terms of acquiring the data sample set, the fourth acquisition module is specifically used for: Obtain sample media files of different types; Different types and levels of distortion processing were applied to each sample media file to obtain multiple differentiated sample media files.

[0115] In one embodiment, the apparatus further includes: a first processing module; The first processing module is configured to perform the following steps on each of the differentiated sample media files: Collect the raw scores for each subjective rating dimension of the current sample media file; Remove outliers from the original scores; The original scores after removal are normalized; The subjective score of the current media file is obtained based on the normalized original score.

[0116] In one embodiment, the apparatus further includes: a second processing module; the second processing module is specifically used for: The quality level of the media file to be evaluated is obtained based on the comprehensive score. Obtain the proportion of subjective and objective scores in the overall score for the media file to be evaluated; Output the quality level and the percentage performance.

[0117] In one embodiment, regarding the aspect of obtaining a comprehensive score for the media file to be evaluated based on objective and subjective scores, the third acquisition module is specifically used for: Based on a preset weighting strategy, the comprehensive score of the media file to be evaluated is calculated by combining the objective score and the subjective score.

[0118] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for quality assessment of image / video data.

[0119] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0120] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0121] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0124] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for quality assessment of image / video data, characterized in that, The method includes: Receive media files to be evaluated; Obtain the objective score corresponding to the media file to be evaluated; The media file to be evaluated and the objective score are input into a trained target evaluation model to obtain a subjective score corresponding to the media file to be evaluated. The target evaluation model is obtained by training an evaluation model based on sample media files, their subjective scores, and their objective scores. The target evaluation model outputs a corresponding subjective score based on the input media file to be evaluated and its corresponding objective score. The subjective score refers to the score given by a human observer to the image quality. The objective score refers to the score given to the image quality through a computer algorithm. Output the subjective score corresponding to the media file to be evaluated.

2. The method according to claim 1, characterized in that, The method further includes: A comprehensive score for the media file to be evaluated is obtained based on the objective and subjective scores corresponding to the media file to be evaluated. Output the overall score.

3. The method according to claim 2, characterized in that, The method further includes: Obtain a data sample set, which includes: differentiated sample media files and subjective and objective scores corresponding to the differentiated sample media files; The evaluation model is trained by using the differentiated sample media files and their corresponding objective ratings as inputs, and the corresponding subjective ratings as labels for training the evaluation model, thereby obtaining the target evaluation model.

4. The method according to any one of claims 1-3, characterized in that, The subjective ratings include: average opinion score and difference average opinion score; The subjective rating corresponds to at least one of the following rating dimensions: absolute evaluation, relative evaluation, keyboard and mouse sensitivity evaluation, and audio and microphone evaluation.

5. The method according to claim 4, characterized in that, The objective scoring corresponds to at least one of the following scoring dimensions: traditional indicators, structural similarity indicators, perceptual indicators, and audio indicators; The traditional metrics include: peak signal-to-noise ratio, mean square error, and mean absolute error. The structural similarity indicators include: structural similarity index and multi-scale structural similarity index; The perception metrics include: visual information fidelity, video multi-method evaluation fusion, and feature similarity index; The audio metrics include: perceptual objective hearing quality analysis and objective speech quality assessment.

6. The method according to claim 3, characterized in that, The acquisition of the data sample set includes: Obtain the sample media files of different types; Different types and levels of distortion processing are applied to each of the sample media sample files to obtain the differentiated sample media files.

7. The method according to claim 6, characterized in that, The method further includes: For each of the differentiated sample media files, the following steps are performed: Collect the raw scores for each subjective rating dimension of the current sample media file; Remove outliers from the original scores; The original scores after removal are normalized; The subjective score of the current media file is obtained based on the normalized original score.

8. The method according to claim 7, characterized in that, The method further includes: The quality level of the media file to be evaluated is obtained based on the comprehensive score. Obtain the proportion of subjective and objective scores in the overall score for the media file to be evaluated; Output the quality level and the percentage performance.

9. The method according to claim 2, characterized in that, The process of obtaining a comprehensive score for the media file to be evaluated based on objective and subjective scores includes: Based on a preset weighting strategy, the comprehensive score of the media file to be evaluated is calculated by combining the objective score and the subjective score.

10. A quality assessment device for image / video data, characterized in that, The device includes: The receiving module is used to receive the media files to be evaluated. The first acquisition module is used to acquire the objective score corresponding to the media file to be evaluated; The second acquisition module is used to input the media file to be evaluated and the objective score into a trained target evaluation model to obtain the subjective score corresponding to the media file to be evaluated. The target evaluation model is obtained by training an evaluation model based on sample media files, the subjective scores of sample media files, and the objective scores of sample media files. The target evaluation model outputs the corresponding subjective score based on the input media file and the corresponding objective score. The subjective score refers to the score given by a human observer to the image quality. The objective score refers to the score given to the image quality through a computer algorithm. The first output module is used to output the subjective score corresponding to the media file to be evaluated.

Images