Method and system for no-reference screen image quality assessment based on multi-feature fusion

Abstract

The application provides a multi-feature fusion-based no-reference screen image quality evaluation method and system, which comprises the following steps: calculating the gradient amplitude, relative gradient amplitude and gradient direction mapping of the input screen content image, applying a local ternary pattern operator on the mapping for texture coding, combining the gradient domain values to statistically weight the coding modes, and generating a gradient-weighted local ternary pattern histogram; inputting the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtaining the deep feature map at the end of the network, and converting the deep feature map into a global perception feature vector through a global average pooling operation; fusing the gradient-weighted local ternary pattern histogram and the global perception feature vector, inputting them into a neural network quality prediction model, and outputting the no-reference quality evaluation score of the screen content image after nonlinear mapping of the model. The application realizes accurate mapping from multi-dimensional features to quality scores.

Description

Technical Field

[0001] This application relates to the field of screen content image quality evaluation, and more specifically, to a no-reference screen image quality evaluation algorithm and system based on multi-feature fusion. Background Technology

[0002] With the rapid development of multimedia technology and electronic communication systems, screen content images (SCIs) generated by mobile terminals, online conferencing, and cloud computing platforms have become deeply integrated into daily life. Unlike traditional natural scene images, SCIs, as hybrid entities, contain a large amount of computer-generated text, graphics, and vector animations. This fundamental difference in content attributes often makes traditional Natural Image Quality Assessment (IQA) algorithms exhibit significant limitations when processing SCIs. Therefore, designing a reliable Screen Content Image Quality Assessment (SCIQA) method can not only enrich the theoretical system of image quality assessment but also optimize multimedia transmission efficiency and improve the user experience of remote interactive experiences.

[0003] In full-reference screen content image quality assessment tasks, using a perfect-quality image as a reference and calculating and learning the fidelity and quality degradation between the distorted image and the reference is crucial for accurately evaluating the quality of the distorted image. However, in practical applications, obtaining a distortion-free reference image is often a difficult task due to bandwidth limitations or the lack of an original reference source. This is the problem that referenceless screen content image quality assessment aims to solve. Compared with full-reference methods, referenceless quality assessment methods generally have poorer effectiveness, stability, and consistency because they are highly dependent on feature extraction from the distorted image and are very sensitive to content. Currently, most quality assessment algorithms are designed primarily for Natural Scene Statistics (NSS) images, ignoring the characteristics of screen content images such as high-contrast edges, large amounts of text, and the mixing of artificial content with natural backgrounds. Traditional referenceless screen content image quality assessment methods still exhibit certain limitations when dealing with increasingly complex distorted scenes. Although methods based on handcrafted features have good physical interpretability, they often struggle to effectively capture the high-level semantic information of the image, resulting in current methods failing to achieve high accuracy in screen content image quality assessment tasks.

[0004] Therefore, the development of image quality assessment algorithms for content on referenceless screens is one of the important technical issues in this field.

[0005] A search revealed a Chinese patent application (application number 201910281019.0) that discloses a no-reference image quality assessment method based on a multi-task convolutional neural network. This method constructs a training set by extracting image patches and degradation labels, then trains the convolutional network for quality regression, achieving improved subjective-objective correlation ratios (SRCC / PLCC) compared to traditional methods. However, this method relies solely on local block-level features for prediction, lacking the ability to perceive the global context and high-level semantic structure of the image, making it difficult to accurately characterize overall visual quality degradation under complex distortions. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this application is to provide a referenceless screen image quality evaluation algorithm and system based on multi-feature fusion.

[0007] A first aspect of this application provides a method for evaluating the quality of a no-reference screen image based on multi-feature fusion, comprising: A mapping map of gradient magnitude, relative gradient magnitude, and gradient direction is calculated for the input screen content image. A local ternary mode operator is applied to all the mapping maps for texture encoding. The encoded patterns are weighted and statistically analyzed in combination with the gradient magnitude to generate a histogram. The screen content image is input into a pre-trained deep convolutional neural network for feature extraction to obtain a deep feature map, which is then converted into a globally perceptual feature vector. The histogram and the global perception feature vector are fused together and input into a pre-trained neural network quality prediction model to output a no-reference quality evaluation score for the screen content image.

[0008] Optionally, the step of calculating a mapping map of gradient magnitude, relative gradient magnitude, and gradient direction for the input screen content image, applying a local ternary mode operator for texture encoding on all the mapping maps, and weighting the encoded patterns by combining the gradient magnitude to generate a histogram includes: The screen content image is converted to grayscale to obtain a grayscale image; Calculate the horizontal and vertical gradients of the grayscale image to obtain a gradient magnitude map, a relative gradient magnitude map, and a gradient direction map; The local ternary mode operator is applied to the gradient magnitude map, relative gradient magnitude map, and gradient direction map respectively. By setting a threshold, the gray-level difference relationship between the center pixel and the neighboring pixels of each map is encoded into an upper-level mode and a lower-level mode. The upper-level mode and the lower-level mode are then jointly encoded to obtain three sets of local ternary mode feature indices. The three sets of local ternary pattern feature indices are weighted and statistically analyzed using the values ​​of the gradient magnitude mapping map to generate three sub-histograms. The three sub-histograms are then sequentially concatenated to form a gradient-weighted local ternary pattern histogram.

[0009] Optionally, the gradient magnitude Relative gradient magnitude Horizontal gradient Vertical gradient and gradient direction The calculation formulas are as follows: ; ; ; ; ; in, Let (i,j) represent a grayscale image, where (i,j) represents the row and column coordinate indices of the pixel. and Let be the local average gradient within a rectangular window of size H×W, where H and W are the height and width of the window, respectively. and The calculation formula is: ; ; Where F represents the set of all pixel offset coordinates (h, w) within the rectangular window; The upper-layer mode With lower-level mode The encoding formula is: ; ; Where th represents the threshold; GC is the center pixel value; Let t be the value of the t-th equidistant neighboring pixel surrounding the center pixel, where t = 1, 2, ..., n; (·)and (·) represents the step function, specifically:

[0010]

[0011] The histogram of the gradient-weighted local ternary mode The calculation formula is:

[0012] Where I and J are the width and height of the grayscale image, respectively; G(i,j) is the gradient magnitude value of the gradient magnitude map at pixel position (i,j); δ( ) is the Kronecker delta function, which takes the value 1 when the independent variable is equal to 0 and otherwise takes the value 0; m is the pattern index of the histogram; It is the local ternary pattern feature index at pixel position (i,j) obtained by joint encoding of the upper and lower layer patterns.

[0013] Optionally, the step of inputting the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtaining a deep feature map, and converting it into a globally perceptive feature vector includes: The screen content image is resized and cropped at the center, and channel normalization is performed to make it conform to the statistical characteristics of the dataset; The standardized screen content image is input into the deep convolutional neural network to perform forward propagation. The dense connection mechanism of the network is used to realize inter-layer feature reuse and extract deep semantic features of the image. The deep semantic features of the image are processed by nonlinear mapping through an activation function, spatial dimension reduction is performed using an adaptive average pooling layer, and an initial depth feature vector is obtained through a flattening operation. Principal component analysis is used to perform linear mapping and dimensionality reduction on the initial depth feature vector, outputting the final depth-sensing feature vector.

[0014] Optionally, the statistical characteristics of the dataset followed by the channel standardization process are as follows: the mean of each channel of the image is set to [0.485, 0.456, 0.406], and the standard deviation is set to [0.225, 0.224, 0.225]. The activation function uses the modified linear unit ReLU. The adaptive average pooling layer, combined with a flattening operation, compresses the feature map into an initial 1024-dimensional depth feature vector. This process is represented as follows: ; The principal component analysis process compresses the initial 1024-dimensional depth feature vector to 64 dimensions, which serves as the final depth-sensing feature vector.

[0015] Optionally, the neural network quality prediction model adopts a parallel ensemble architecture, consisting of multiple independently trained BP weak regressors and corresponding ensemble weights; The training process of the pre-trained neural network quality prediction model is as follows: A training sample set is constructed, wherein the training sample consists of a multi-dimensional fusion feature vector and its corresponding screen content image real quality score pairing, and the multi-dimensional fusion feature vector is obtained by concatenating the gradient-weighted local ternary mode histogram with the global perception feature vector; A multi-round serial iterative training based on the Adaboost strategy is performed: In each iteration, a backpropagation (BP) weak regressor is generated, and the multi-dimensional fused feature vector is input into the current BP weak regressor for prediction. The absolute prediction error of each training sample is calculated, and the weighted average prediction error of the BP weak regressor is calculated based on the current sample weights. The weight distribution of the training samples is dynamically updated according to the absolute prediction error of each training sample, so that samples with an absolute prediction error higher than a set value receive higher weights in subsequent iterations. Training is terminated when the number of iterations reaches a preset threshold, resulting in multiple BP weak regressors; the corresponding ensemble weights are calculated based on the weighted average prediction error of each BP weak regressor. The multiple backpropagation weak regressors are bound to the corresponding ensemble weights, and the neural network quality prediction model is constructed through weighted ensemble.

[0016] Optionally, generating multiple BP weak regressors specifically involves generating 32 BP weak regressors sequentially. The process of outputting the no-reference quality evaluation score of the screen content image is as follows: The multidimensional fusion feature vector corresponding to the screen content image to be evaluated is input into the trained neural network quality prediction model, and the 32 BP weak regressors make independent predictions and output intermediate scores respectively. The intermediate scores are weighted and summed according to the integrated weights, and the final no-reference quality evaluation score is output after integrated calculation.

[0017] A second aspect of this application provides a no-reference screen image quality assessment system based on multi-feature fusion, comprising: Gradient structure feature module: Calculates a mapping map of gradient magnitude, relative gradient magnitude, and gradient direction for the input screen content image; applies a local ternary mode operator to all the mapping maps for texture encoding; and performs weighted statistics on the encoded patterns in combination with the gradient magnitude to generate a histogram. Deep perception feature module: Input the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtain deep feature maps, and convert them into global perception feature vectors; Image quality scoring module: The histogram and the global perception feature vector are fused together and input into a pre-trained neural network quality prediction model to output a no-reference quality evaluation score for the screen content image.

[0018] A third aspect of this application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any of the multi-feature fusion-based no-reference screen image quality evaluation methods described above, or to implement the multi-feature fusion-based no-reference screen image quality evaluation system described above.

[0019] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the multi-feature fusion-based no-reference screen image quality assessment methods described in the present invention, or to implement the multi-feature fusion-based no-reference screen image quality assessment system described in the present invention.

[0020] The referenceless screen image quality assessment method based on multi-feature fusion provided in this application uses the gradient-weighted local ternary mode operator LTP to capture microstructure loss, and uses a pre-trained deep convolutional neural network to make up for the shortcomings of traditional methods in semantic perception. Finally, the neural network is used to achieve accurate mapping from multi-dimensional features to quality scores.

[0021] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a no-reference screen image quality assessment method based on multi-feature fusion according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the framework of a no-reference screen image quality assessment method based on multi-feature fusion according to an exemplary embodiment. Figure 3 This is a schematic diagram illustrating the structure of a no-reference screen image quality assessment system based on multi-feature fusion according to an exemplary embodiment. Detailed Implementation

[0023] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0024] Existing no-reference evaluation methods typically focus only on the statistical features of images (such as gradient and sharpness), lacking an understanding of the semantic content of the images and failing to capture high-level semantic information, resulting in inaccurate evaluations. To address these issues, this application provides a no-reference screen image quality evaluation method based on multi-feature fusion to resolve these problems.

[0025] Reference Figure 1 As shown in one embodiment of this application, a method for evaluating the quality of a no-reference screen image based on multi-feature fusion includes: S100: Calculate the gradient magnitude, relative gradient magnitude, and gradient direction mapping map of the input screen content image. Apply the local ternary mode operator to the mapping map for texture encoding. Combine the gradient magnitude values ​​to perform weighted statistics on the encoding mode and generate a gradient-weighted local ternary mode histogram. Specifically, the local ternary mode operator maps the grayscale difference between the neighboring pixels and the center pixel into upper, neutral, and lower ternary states by setting upper and lower thresholds in the neighborhood of the center pixel.

[0026] S200 inputs the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtains deep feature maps, and converts them into globally perceptive feature vectors.

[0027] Specifically, the deep convolutional neural network can be DenseNet-121. This network uses ImageNet pre-trained weights as a fixed feature extractor. The pre-trained model is loaded and all convolutional layer parameters are frozen. The input image is standardized and then passed through a feature extraction layer to generate a deep feature map. This map is then transformed into a 1024-dimensional feature vector through adaptive average pooling and flattening operations, without updating the network parameters during this process. Since the screen content image distortion database is relatively small (1800 images for SCID, 980 images for SIQAD), it is insufficient to support training DenseNet-121 from scratch. However, the ImageNet pre-trained model has already learned common low-level visual features such as edges and textures, which can be directly reused.

[0028] S300 fuses histograms and global-aware feature vectors, inputs them into a pre-trained neural network quality prediction model, and outputs a no-reference quality evaluation score for the screen content image.

[0029] In the above embodiments, S100, by calculating gradient magnitude and applying local ternary mode operators, can accurately capture the underlying physical distortions of the image (such as decreased sharpness of text edges, blockiness in graphic regions, jagged edges, etc.), which is highly sensitive to text and graphic regions that are crucial in screen content images. S200 utilizes a pre-trained DenseNet-121 network to extract deep features, enabling the acquisition of high-level semantic information and global perception features of the image, thus overcoming the shortcomings of traditional handcrafted features in understanding the semantics of image content. The fusion of these two approaches (S300) retains the sensitivity of handcrafted features to local structural distortions while introducing the understanding of overall content perception by deep features, solving the problem of single feature evaluation having a limited dimension and poor adaptability. S300 employs an Adaboost-BP neural network as the regression model, integrating multiple BP neural networks using the Adaboost algorithm. Compared to a single regression model (such as SVR or BP networks), Adaboost-BP effectively reduces the model's generalization error and enhances robustness to complex distortion types. It can more accurately fit the complex nonlinear relationship between "feature vector" and "subjective quality score", thereby significantly improving the accuracy of quality prediction.

[0030] Of course, other deep convolutional neural networks or neural network quality prediction models can also be used in other embodiments, and are not the only ones.

[0031] Screen content images differ significantly from natural images. Screen content images (such as documents and web pages) contain numerous sharp text edges and geometric structures, making them extremely sensitive to structural distortions (such as blurring and jagged edges). Traditional evaluation methods based on natural images struggle to effectively capture such structural degradation. Therefore, this application constructs a gradient-weighted feature extraction mechanism specifically for screen content images to accurately quantify their structural integrity and texture clarity. In some specific embodiments of this application, in S100, a mapping map of gradient magnitude, relative gradient magnitude, and gradient direction is calculated for the input screen content image. A local ternary mode operator is applied to the mapping map for texture encoding. The encoded mode is weighted and statistically analyzed based on the gradient magnitude values ​​to generate a gradient-weighted local ternary mode histogram. This can be achieved through the following steps: S1-1: Convert the input screen content image to grayscale, and calculate the gradient magnitude using the horizontal gradient dx and the vertical gradient dy. And introduce relative gradient magnitude The formula for measuring the microscopic fluctuations in local intensity is as follows:

[0032] Simultaneously calculate the local average gradient within an H×W rectangular window (with H and W set to 3). and To capture the dynamic features of image details, the formula is as follows:

[0033] S1-2, Introducing the gradient direction To reflect the degree of damage to the structural expressiveness caused by different types of distortion:

[0034] S1-3, at gradient magnitude Relative gradient magnitude and direction The LTP operator is applied to the mapping graph; by setting a threshold th=5, the center pixel GC is linked to its n equidistant neighboring pixels. Relationship encoding as upper-level schema Layer mode The formula is as follows:

[0035] Among them, formula and The definition is as follows:

[0036] S1-4, using the numerical values ​​in the gradient domain mapping graph to perform weighted calculations on the LTP distribution pattern, constructing a gradient-weighted LTP histogram. The formula is as follows:

[0037] Where I and J are the width and height of the grayscale image, respectively; G(i,j) is the gradient magnitude of the gradient magnitude map at pixel position (i,j); δ( ) is the Kronecker delta function, which takes the value 1 when the independent variable is equal to 0 and otherwise takes the value 0; m is the pattern index of the histogram; It is the local ternary pattern feature index at pixel position (i,j) obtained by joint encoding of the upper and lower layer patterns.

[0038] S1-1 to S1-4 of the above embodiments together constitute the core "gradient structure feature extraction" method of this application. S1-1 accurately separates the crucial structural skeleton information and micro-texture fluctuations in the screen content image by calculating the gradient magnitude and relative gradient magnitude, solving the problem that traditional methods struggle to simultaneously consider edge sharpness and texture details. S1-2 introduces a gradient direction map, further enhancing the ability to describe the directionality of image structure. Based on this, S1-3 employs the Local Triple Mode (LTP) operator, effectively suppressing high-frequency noise interference generated during compression or transmission of the screen content image through a threshold mechanism, thus improving the robustness of the features. Finally, S1-4 uses gradient magnitude to perform weighted statistics on the LTP mode, constructing a gradient-weighted histogram feature. This design allows the feature vector to adaptively focus on information-rich edges and text regions (i.e., high-gradient regions) in the image, making the final quality evaluation result more consistent with the visual perception characteristics of screen content images by the human eye, significantly improving the accuracy of the evaluation model.

[0039] Screen content images not only contain low-level pixel statistical characteristics but also complex high-level semantic information (such as text readability and interface layout), which cannot be ignored and requires feature extraction. Traditional feature extraction methods are often limited to shallow textures and struggle to capture such complex semantic structures and their impact on visual quality. Therefore, this application introduces a feature extraction mechanism based on deep convolutional neural networks. By simulating the human visual system's cognitive process of image content, from standardized preprocessing to deep semantic mining and feature dimensionality reduction compression, a feature extraction process capable of accurately representing the deep perceptual quality of screen content images is constructed. In some specific implementations, in S200, the screen content image is input into a pre-trained deep convolutional neural network for feature extraction to obtain a deep feature map, which is then converted into a globally perceptual feature vector. This can be achieved through the following steps: S2-1 performs resizing and center cropping operations on the input distorted screen content image, and performs channel normalization to match the pixel distribution of each channel of the image with the preset statistical parameters of the ImageNet dataset (mean set to [0.485, 0.456, 0.406], standard deviation set to [0.225, 0.224, 0.225]), and outputs a normalized image tensor; S2-2 inputs the normalized image tensor from S2-1 into the pre-trained DenseNet-121 network, utilizes the dense connection mechanism to achieve inter-layer feature reuse, simultaneously captures the fine outlines of text strokes and complex semantic structures of graphics in the image, and outputs the original feature map of the convolutional layer at the end of the network (also known as the deep semantic features of the image). Specifically, the dense connection mechanism for inter-layer feature reuse means that the input to each layer in the network does not come solely from the previous layer, but rather from the concatenation (concat) of feature maps from all previous layers along the channel dimension. Details extracted in shallow layers (such as character strokes and sharp edges) are not "filtered or smoothed out" by subsequent convolutions, but are directly reused by deeper networks.

[0040] S2-3: The original feature map is used as input, and nonlinear mapping is performed through the Modified Linear Unit (ReLU) activation function to enhance the spatial response intensity of key structural regions and obtain the activated deep feature map. S2-4 utilizes an adaptive average pooling layer (AdaptiveAvgPool2d) to reduce the dimensionality of the terminal feature map (the activated deep feature map), compressing its shape into a single-dimensional vector. A flattening operation then produces a 1024-dimensional depth-aware feature vector. This process can be described as follows: (13) S2-5: Principal component analysis (PCA) is introduced to perform linear mapping on the 1024-dimensional high-dimensional features, eliminate feature redundancy and noise, and compress the depth feature vector to 64 dimensions as the final depth perception feature, which is used to simulate the human eye's cognitive judgment of the macroscopic visual quality of screen content.

[0041] The above embodiments, taking into account the characteristic that screen content images contain complex semantic structures (such as text and icons), introduce a deep feature extraction process based on DenseNet-121. S2-1 ensures that the data distribution of the input image and the pre-trained model are aligned through standardized preprocessing, laying the foundation for subsequent feature extraction; S2-2 utilizes the dense connection mechanism of DenseNet-121 to effectively capture the fine strokes of text and the semantic structure of graphics in the image, solving the problem that traditional methods are difficult to extract high-level semantic information; S2-3 enhances the nonlinear expressive power of features through the ReLU activation function, enabling it to adapt to complex mixed distortion modes; S2-4 compresses the high-dimensional feature map into a one-dimensional vector through adaptive average pooling and Flatten operations, preserving global perception information; S2-5 introduces PCA dimensionality reduction to remove feature redundancy and noise, compressing the 1024-dimensional features to 64 dimensions, obtaining compact and representative deep perception features, simulating the human eye's cognitive judgment of the macroscopic visual quality of screen content, significantly improving the accuracy and computational efficiency of quality assessment.

[0042] Screen content images (SCI) possess complex feature distributions, including sharp structure, regular semantics, and mixed distortion types. The default parameters of general regression models are often difficult to directly adapt, easily leading to inaccurate quality mapping. Therefore, based on extensive experimental verification, this application optimizes the key hyperparameters of the quality prediction model specifically for the characteristics of the extracted multidimensional fusion feature distribution. In some specific embodiments of this application, the pre-trained neural network quality prediction model in S300 (the neural network quality prediction model adopts a parallel ensemble architecture, consisting of multiple independently trained BP weak regressors and corresponding ensemble weights) undergoes the following training steps: The training process of a pre-trained neural network quality prediction model is as follows: First, a training sample set is constructed. The training samples consist of a multi-dimensional fusion feature vector and its corresponding screen content image real quality score pair. The multi-dimensional fusion feature vector is obtained by concatenating a gradient-weighted local ternary pattern histogram with a globally perceived feature vector.

[0043] Next, multi-round serial iterative training is performed based on the Adaboost strategy (specifically, each iteration trains and generates only one BP weak regressor, and the weight distribution of the training samples in the current round is dynamically calculated from the prediction error of the weak regressor in the previous round; the next round of training is not started before the previous weak regressor is trained and the sample weights are updated; after the iteration terminates, the multiple BP weak regressors generated in sequence and their corresponding ensemble weights are bound together to construct a neural network quality prediction model): In each iteration, a backpropagation (BP) weak regressor is trained and generated. The multi-dimensional fused feature vector is input into the current BP weak regressor for prediction. The absolute prediction error of each training sample is calculated, and the weighted average prediction error of the BP weak regressor is calculated based on the current sample weights. The weight distribution of training samples is dynamically updated according to the absolute prediction error of each training sample, so that samples with an absolute prediction error higher than a set value receive higher weights in subsequent iterations. By increasing the weight proportion of high-error samples, the subsequent BP weak regressor is forced to focus on complex and distorted samples that are difficult to fit, such as blurred text, broken graphics, and color block breaks.

[0044] Then, training is terminated when the number of iterations reaches a preset threshold, resulting in multiple BP weak regressors; the corresponding ensemble weights are calculated based on the weighted average prediction error of each BP weak regressor. Finally, multiple backpropagation weak regressors are bound to their corresponding ensemble weights, and a neural network quality prediction model is constructed through weighted ensemble.

[0045] The above embodiments, by employing the Adaboost ensemble strategy to train a backpropagation (BP) neural network, achieve adaptive weighted learning for samples of varying difficulty. Specifically, this training process utilizes a multi-round iterative mechanism, enabling subsequent weak regressors to effectively correct errors in samples with large prediction biases from preceding models. This effectively overcomes the problem of uneven fitting ability of a single model when handling different types of distortion in screen content images. The resulting strong prediction model significantly improves the consistency between quality prediction results and human subjective perception, exhibiting superior generalization ability and accuracy.

[0046] In some specific embodiments of this application, in S300, the gradient-weighted local ternary pattern histogram and the globally perceived feature vector are fused, and the result is input into a pre-trained Adaboost-BP neural network quality prediction model. After nonlinear mapping by the model, the referenceless quality evaluation score of the screen content image is output. Specifically, the following steps can be used: Figure 2 As shown: S3-1, Input the multi-dimensional fusion feature vector corresponding to the screen content image to be evaluated into the trained Adaboost-BP neural network quality prediction model, and each BP weak regressor makes independent predictions and outputs intermediate scores. S3-2: The intermediate scores are weighted and summed according to the model weights, and the final no-reference quality evaluation score is output after integrated calculation.

[0047] The above embodiments, through the synergistic effect of multi-dimensional feature fusion input and weighted integrated inference, achieve high-precision nonlinear mapping of complex distorted screen content image quality, significantly improving the accuracy of scoring.

[0048] Based on the same technical concept, other embodiments of this application provide a referenceless screen image quality evaluation system 100 based on multi-feature fusion, such as... Figure 3 As shown, it includes: The gradient structure feature module 110 calculates a mapping map of gradient magnitude, relative gradient magnitude, and gradient direction for the input screen content image. It applies a local ternary mode operator to the mapping map for texture encoding and performs weighted statistics on the encoding mode by combining the gradient magnitude values ​​to generate a gradient-weighted local ternary mode histogram. The deep perception feature module 120 inputs the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtains a deep feature map, and converts it into a global perception feature vector. The image quality scoring module 130 fuses histograms and global perception feature vectors, inputs them into a pre-trained neural network quality prediction model, and outputs a no-reference quality evaluation score for the screen content image.

[0049] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the referenceless screen image quality evaluation method based on multi-feature fusion in the above embodiments, and will not be repeated here.

[0050] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0051] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0052] Application Example 1: The proposed multi-feature fusion-based no-reference screen image quality assessment method was quantitatively evaluated using the SCID and SIQAD datasets. The comparison results are shown in Table 1. The proposed method achieved the best results in all three core performance metrics: PLCC, SROCC, and RMSE. Compared to MF-SCIQA, the proposed method achieved gains of 0.53%, 0.69%, and 5% in these three metrics on the SCID dataset, respectively.

[0053] Table 1

[0054] The experimental results above show that the model in this application has high prediction accuracy, which mainly comes from the synergistic effect of Adaboost's "dynamic error correction mechanism" and "multi-feature fusion": at the feature level, the input integrates gradient structure, texture distribution and depth perception information to construct a multi-dimensional feature vector, providing a high-discrimination data foundation for accurate regression. At the algorithm level, to address the issue that a single network is prone to significant bias in specific distortions, Adaboost-BP automatically identifies difficult samples with large prediction residuals through multiple rounds of iterative training, dynamically increases their training weights, and guides subsequent weak regressors to focus on the evaluation blind spots of preceding models in order to achieve sequential error correction.

[0055] The aforementioned feature inputs and dynamic feedback mechanisms are not simply superimposed, but rather form a tightly coupled collaborative optimization loop, making the model's decision-making logic highly consistent with the subjective perception patterns of the human eye, thereby significantly improving the accuracy, robustness, and cross-scenario generalization ability of quality evaluation.

[0056] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0057] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned no-reference screen image quality evaluation method based on multi-feature fusion, or to run the above-mentioned no-reference screen image quality evaluation system based on multi-feature fusion.

[0058] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the program can be used to perform the above-described method for evaluating the quality of a no-reference screen image based on multi-feature fusion, or to run the above-described system for evaluating the quality of a no-reference screen image based on multi-feature fusion.

[0059] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0060] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0061] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0062] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for evaluating the quality of a no-reference screen image based on multi-feature fusion, characterized in that, include: A mapping map of gradient magnitude, relative gradient magnitude, and gradient direction is calculated for the input screen content image. A local ternary mode operator is applied to all the mapping maps for texture encoding. The encoded patterns are weighted and statistically analyzed in combination with the gradient magnitude to generate a histogram. The screen content image is input into a pre-trained deep convolutional neural network for feature extraction to obtain a deep feature map, which is then converted into a globally perceptual feature vector. The histogram and the global perception feature vector are fused together and input into a pre-trained neural network quality prediction model to output a no-reference quality evaluation score for the screen content image.

2. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 1, characterized in that, The process involves calculating a mapping of gradient magnitude, relative gradient magnitude, and gradient direction for the input screen content image; applying a local ternary mode operator to all mappings for texture encoding; and weighting the encoded patterns based on the gradient magnitude to generate a histogram. This includes: The screen content image is converted to grayscale to obtain a grayscale image; Calculate the horizontal and vertical gradients of the grayscale image to obtain a gradient magnitude map, a relative gradient magnitude map, and a gradient direction map; The local ternary mode operator is applied to the gradient magnitude map, the relative gradient magnitude map, and the gradient direction map respectively. By setting a threshold, the gray-level difference relationship between the center pixel and the neighboring pixels of each map is encoded into an upper-level mode and a lower-level mode. The upper-level mode and the lower-level mode are then jointly encoded to obtain three local ternary mode feature indices. The three local ternary pattern feature indices are weighted and statistically analyzed using the values ​​of the gradient magnitude mapping map to generate three sub-histograms, and the three sub-histograms are sequentially concatenated to form a gradient-weighted local ternary pattern histogram.

3. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 2, characterized in that, The gradient magnitude Relative gradient magnitude Horizontal gradient Vertical gradient and gradient direction The calculation formulas are as follows: ； ; ; ; ; in, Let (i,j) represent a grayscale image, where (i,j) represents the row and column coordinate indices of the pixel. and Let be the local average gradient within a rectangular window of size H×W, where H and W are the height and width of the window, respectively. and The calculation formula is: ; ; Where F represents the set of all pixel offset coordinates (h, w) within the rectangular window; The upper-layer mode With lower-level mode The encoding formula is: ; ; Where th represents the threshold; GC is the center pixel value; Let t be the value of the t-th equidistant neighboring pixel surrounding the center pixel, where t = 1, 2, ..., n; (·)and (·) represents the step function, specifically: ; ; The histogram of the gradient-weighted local ternary mode The calculation formula is: ; Where I and J are the width and height of the grayscale image, respectively; G(i,j) is the gradient magnitude value of the gradient magnitude map at pixel position (i,j); δ( ) is the Kronecker delta function, which takes the value 1 when the independent variable is equal to 0 and otherwise takes the value 0; m is the pattern index of the histogram; It is the local ternary pattern feature index at pixel position (i,j) obtained by joint encoding of the upper and lower layer patterns.

4. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 1, characterized in that, The step of inputting the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtaining a deep feature map, and converting it into a globally perceptive feature vector includes: The screen content image is resized and cropped at the center, and channel normalization is performed to make it conform to the set statistical characteristics; The standardized screen content image is input into the deep convolutional neural network to perform forward propagation. The dense connection mechanism of the network is used to realize inter-layer feature reuse and extract deep semantic features of the image. The deep semantic features of the image are processed by nonlinear mapping through an activation function, spatial dimension reduction is performed using an adaptive average pooling layer, and an initial depth feature vector is obtained through a flattening operation. Principal component analysis is used to perform linear mapping and dimensionality reduction on the initial depth feature vector, outputting the final depth-sensing feature vector.

5. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 4, characterized in that, The channel standardization process ensures that the channel conforms to the set statistical characteristics, specifically: the mean of each channel of the image is set to [0.485, 0.456, 0.406], and the standard deviation is set to [0.225, 0.224, 0.225]. The activation function uses the modified linear unit ReLU. The adaptive average pooling layer, combined with a flattening operation, compresses the feature map into an initial 1024-dimensional depth feature vector. This process is represented as follows: ; The principal component analysis process compresses the initial 1024-dimensional depth feature vector to 64 dimensions, which serves as the final depth-sensing feature vector.

6. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 1, characterized in that, The neural network quality prediction model adopts a parallel ensemble architecture, consisting of multiple independently trained BP weak regressors and corresponding ensemble weights. The training process of the neural network quality prediction model is as follows: A training sample set is constructed, wherein the training sample consists of a multi-dimensional fusion feature vector and its corresponding screen content image real quality score pairing, and the multi-dimensional fusion feature vector is obtained by concatenating the gradient-weighted local ternary mode histogram with the global perception feature vector; The Adaboost strategy is used for multi-round serial iterative training. Specifically, in each iteration, a BP weak regressor is generated, the multi-dimensional fused feature vector is input into the current BP weak regressor for prediction, the absolute prediction error of each training sample is calculated, and the weighted average prediction error of the BP weak regressor is calculated based on the weight of the current training sample. The weight distribution of the training samples is dynamically updated based on the absolute prediction error of each training sample. Training terminates when the number of iterations reaches a preset threshold, resulting in multiple BP weak regressors. The corresponding ensemble weights are calculated based on the weighted average prediction errors of each BP weak regressor. The multiple BP weak regressors are bound to the corresponding ensemble weights, and the neural network quality prediction model is constructed through weighted ensemble.

7. The method for evaluating the quality of a no-reference screen image based on multi-feature fusion according to claim 6, characterized in that, The generation of multiple BP weak regressors specifically involves generating 32 BP weak regressors sequentially. The process of outputting the no-reference quality evaluation score of the screen content image is as follows: The multidimensional fusion feature vector corresponding to the screen content image to be evaluated is input into the trained neural network quality prediction model, and the 32 BP weak regressors make independent predictions and output intermediate scores respectively. The intermediate scores are weighted and summed according to the integrated weights, and the final no-reference quality evaluation score is output after integrated calculation.

8. A referenceless screen image quality assessment system based on multi-feature fusion, characterized in that, include: Gradient structure feature module: Calculates a mapping map of gradient magnitude, relative gradient magnitude, and gradient direction for the input screen content image; applies a local ternary mode operator to all the mapping maps for texture encoding; and performs weighted statistics on the encoded patterns in combination with the gradient magnitude to generate a histogram. Deep perception feature module: Input the screen content image into a pre-trained deep convolutional neural network for feature extraction, obtain deep feature maps, and convert them into global perception feature vectors; Image quality scoring module: The histogram and the global perception feature vector are fused together and input into a pre-trained neural network quality prediction model to output a no-reference quality evaluation score for the screen content image.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-7, or to implement the system of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-7, or to implement the system of claim 8.

Images