A reference-free screen content image quality assessment method based on multi-scale edge feature fusion
Through the combination of multi-scale edge feature fusion and Transformer model, the problem of shallow image feature degradation in image quality evaluation of referenceless screen content is solved, more accurate quality evaluation is achieved, and model performance and information utilization are improved.
Patent Information
- Application Number
- CN202210684718.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-17
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-06-17
AI Technical Summary
The existing image quality evaluation method of non-reference screen content has the problem of image shallow feature degradation in deep learning models. The traditional method relies on manual extraction of features, which is highly limited and difficult to effectively utilize image information, resulting in inconsistent evaluation effect with subjective perception. .
Using a method based on multi-scale edge feature fusion, the image edge features are extracted through the Gaussian Laplace operator, and combined with the Transformer model, the multi-scale distorted image features and edge structure features are fused to form a global information representation, and the image quality evaluation model is improved. performance.
It effectively improves the performance of the image quality evaluation model without reference screen content, can accurately predict the quality evaluation score of the image, and enhances the non-local correlation and information utilization of the model.
Smart Images
Figure CN114897884B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing and computer vision technology, and in particular relates to a reference-free screen content image quality assessment method based on multi-scale edge feature fusion. Background Art
[0002] With the rapid development of technologies such as cloud computing, mobile Internet, and the Internet of Things, multimedia applications such as multi-client interaction have gradually increased, and the large number of screen content images generated have attracted widespread attention. Unlike traditional natural images taken from real scenes, screen content images are usually composed of computer-generated content, including media forms such as images, texts, tables, and patterns, and sometimes also traditional natural image content. In addition, screen content images usually have some characteristics different from natural images. The former often have higher or lower frequency content, such as large smooth areas and sharp text or edges, and high saturation colors, repeated patterns, etc. are also typical characteristics of screen content images. Usually, screen content images are affected by various distortion factors during the process of acquisition, transmission, encoding, and display, resulting in varying degrees of image quality degradation, which affects user experience and system interaction performance. Therefore, it is extremely important to design effective and accurate quality evaluation methods for various screen image processing applications.
[0003] According to the different amounts of reference image information required in the quality evaluation process, current image quality evaluation methods are mainly divided into three categories: full reference, semi-reference, and no-reference. Since we often cannot obtain reference information of the original screen image in real life, research on no-reference image quality evaluation algorithms is more realistic, and at the same time, it is more difficult than full reference and semi-reference methods. For screen content images with prominent statistical features, traditional quality evaluation methods are based on some carefully selected features. Scholars manually design and extract relevant features from screen content images, and then combine effective machine learning algorithms to represent image features as a quality assessment score. However, traditional quality evaluation methods rely more on the effectiveness of manually extracted features, and the extracted features have certain limitations, which limits the utilization of image information and the improvement of model performance.
[0004] In recent years, convolutional neural networks have demonstrated strong learning capabilities and excellent performance in tasks such as image processing, attracting many scholars to study deep learning methods for screen content image quality assessment. Although the use of deep learning methods has made greater progress and breakthroughs than traditional quality assessment methods, the actual evaluation results of existing no-reference methods are less consistent with subjective perception, and often ignore the deep implicit features of the image itself. Summary of the invention
[0005] In order to fill the gaps and shortcomings of the prior art, the present invention proposes a reference-free screen content image quality assessment method based on multi-scale edge feature fusion. Taking into account the fact that screen content images contain a large amount of sharp edges and text information, and the human visual system is highly sensitive to changes in high-frequency information such as the edge structure of the image, a learning strategy for fusing multi-scale distorted image features and image edge structure features is proposed, and the edge structure information of the image is used to provide additional information gain for model training. At the same time, a Transformer model is introduced to further model the non-local correlation of multi-scale image features extracted from the convolutional neural network, thereby effectively improving the performance of the reference-free screen content image quality assessment model.
[0006] The present invention specifically adopts the following technical solutions:
[0007] A method for assessing the quality of screen content without reference based on multi-scale edge feature fusion, characterized by comprising the following steps:
[0008] Firstly, the data in the distorted screen content image dataset is preprocessed, and the image edge features are extracted on the grayscale image of each distorted image using the Gaussian Laplacian operator to generate an edge structure graph dataset corresponding to the distorted screen content image dataset. Then, the two datasets are divided into a training set and a test set in a unified way.
[0009] Train an image quality score prediction network based on multi-scale edge feature fusion to obtain a reference-free screen content image quality score prediction model;
[0010] The image quality score prediction network based on multi-scale edge feature fusion uses a twin network supporting dual-source input to perform multi-scale feature extraction and feature fusion on the input distorted image and the corresponding edge structure graph;
[0011] Then use the Transformer encoder module to form a global information representation of features at different scales;
[0012] Finally, the distorted image and the corresponding edge structure map are input into the trained image quality score prediction network model based on multi-scale edge feature fusion, and the quality assessment score of the distorted image is output.
[0013] Furthermore, the data preprocessing of the data in the distorted screen content image data set specifically includes the following steps:
[0014] Step S11, constructing a Laplacian Gaussian convolution kernel with a window size of H×W and a standard deviation of σ according to the Laplacian Gaussian operator;
[0015] Step S12: convert each distorted screen content image I into a grayscale image g, and perform a convolution operation on the converted grayscale image g using the Gaussian Laplacian convolution kernel obtained in step S11 to obtain an intermediate result image I. cov ;
[0016] Step S13: The intermediate result image I obtained in step S12 cov Threshold processing is performed to determine the pixel points with edge strength greater than the specified threshold as edge points, and set their value to 255. Pixel points with edge strength less than or equal to the specified threshold are eliminated and their value is set to 0, and the final edge structure diagram I is obtained. edge ;
[0017] Step S14, generating an edge structure graph corresponding to each distorted image in the distorted screen content image dataset through steps S11, S12 and S13, obtaining an edge structure graph dataset corresponding to the distorted screen content image dataset, and dividing the two datasets into a training set and a test set in a unified manner, that is, a distorted screen content image and its corresponding edge structure graph belong to the same training set or the same test set.
[0018] Furthermore, the twin network supporting dual-source input is specifically:
[0019] Using two shared-parameter ResNet50s as the backbone network, the distorted image and the corresponding edge structure graph are input into two branch subnetworks for multi-scale feature extraction. In order to achieve parameter sharing between the two branch subnetworks, the edge structure graph needs to be replicated to obtain the same number of channels as the distorted image. Then, stage three and stage four in the backbone network ResNet50 are used as feature extraction modules;
[0020] The Siamese network that supports dual-source input is used to perform multi-scale feature extraction and feature fusion on the input distorted image and the corresponding edge structure map. Specifically:
[0021] Input a batch of distorted images and the corresponding edge structure graphs in the training set into the twin network that supports dual-source input, and obtain feature graphs of distorted images of two different scales and feature graphs of the corresponding edge structure graphs; the feature graph of the distorted image output in stage 3 of the backbone network ResNet50 is recorded as F a3 , the feature map of the edge structure map is F e3 , whose dimensions are C3×h3×w3, C3 represents the number of channels of the feature map, h3 and w3 represent the height and width of the feature map respectively; the feature map of the distorted image output in stage 4 of the backbone network ResNet50 is denoted as F a4 , the feature map of the edge structure map is F e4, whose dimensions are C4×h4×w4, C4 represents the number of channels of the feature map, h4 and w4 represent the height and width of the feature map respectively;
[0022] The feature maps of the two distorted images of different scales and the feature maps of the corresponding edge structure maps are fused, and the sharp edge information in the feature map of the original distorted image is enhanced through the feature information of the edge structure map. The specific calculation formula is as follows:
[0023]
[0024]
[0025] in, represents the matrix addition operation, and F3 represents the fused distorted image feature map F a3 and edge structure feature map F e3 The feature map is obtained, whose dimension is C3×h3×w3, and F4 represents the fused distorted image feature map F a4 and edge structure feature map F e4 The feature map is obtained, whose dimension is C4×h4×w4.
[0026] Furthermore, the specific process of using the Transformer encoder module to form a global information representation of features at different scales is as follows:
[0027] The two different-scale image features F3 and F4 formed by fusing the distorted image feature map and the edge structure map feature map output by the twin network supporting dual-source input are processed to obtain the processed feature maps F′3 and F′4, and the dimensions of F′3 and F′4 are the same:
[0028] Specifically, the feature map F3 is input into a maximum pooling layer and a convolution layer with a convolution kernel of 1×1 in sequence, and its dimension is adjusted to D×h4×w4; the feature map F4 is input into a convolution layer with a convolution kernel of 1×1, and its dimension is adjusted to D×h4×w4. The calculation formula is as follows:
[0029] F′3=ReLU(Conv1(ReLU(Maxpool(F3))))
[0030] F′4=ReLU(Conv2(F4))
[0031] Wherein, ReLU(·) represents the ReLU activation function, Maxpool(*) represents the spatial maximum pooling layer with a step size of 2, Conv1(*) and Conv2(*) represent two convolutional layers with a convolution kernel of 1×1, respectively, and F′3 and F′4 represent the feature maps obtained by processing the feature maps F3 and F4, respectively;
[0032] Then, the feature maps F′3 and F′4 are transformed into N×D by dimensional transformation, where N=h4×w4, and the two-dimensional input matrices X3 and X4 are obtained. Then, an additional learnable feature embedding X is added to the first row of the two-dimensional input matrices X3 and X4 respectively. class_3 and X class_4 , whose dimensions are all 1×D, adding learnable feature embedding X class_3 and X class_4 The dimensions of X3 and X4 after that become (N+1)×D; then add a learnable position embedding X to the two-dimensional input matrix X3 and X4 respectively. pos_3 and X pos_4 , whose dimensions are (N+1)×D, thus obtaining the final two-dimensional input matrices Z3 and Z4; the specific calculation formula is as follows:
[0033] X3=Reshape(F′3)
[0034] X4=Reshape(F′4)
[0035]
[0036]
[0037] Among them, Reshape(·) represents the dimension transformation operation, represents matrix addition operation, Concat(·) represents feature concatenation operation;
[0038] The Transformer encoder module is composed of two Transformer encoders with the same structure;
[0039] Each Transformer encoder is composed of a layer normalization module, a multi-head self-attention module MHSA, a random dropout layer, a layer normalization module, a multi-layer perceptron module MLP consisting of two fully connected layers with GELU activation function, and a random dropout layer;
[0040] The two-dimensional input matrices Z3 and Z4 are respectively input into the two Transformer encoders to obtain non-local representations of image features at different scales; the additional learnable feature embedding X added to each two-dimensional input matrix Z3 and Z4 is taken class_3 and X class_4 The output of the Transformer encoder is expressed as the final result. The specific calculation formula is as follows:
[0041] Z′3=Dropout(MHSA(LN(Z3)))+Z3
[0042] Y3=Dropout(MLP(LN(Z′3)))+Z′3
[0043] T3=Y3[:0]
[0044] Z′4=Dropout(MHSA(LN(Z4)))+Z4
[0045] Y4=Dropout(MLP(LN(Z′4)))+Z′4
[0046] T4=Y4[:0]
[0047] Among them, LN(·) represents the layer normalization operation, Dropout(·) represents the random dropout operation, MHSA(·) represents the operation in the multi-head self-attention module, Z′3 and Z′4 represent the intermediate results of the original two-dimensional input matrices Z3 and Z4 after the multi-head self-attention module, MLP(·) represents the operation in the MLP module, Y3 and Y4 represent the output results of the intermediate results Z′3 and Z′4 after the MLP module, “[:0]” represents the matrix slicing operation, that is, taking all the elements in the first row of the two-dimensional matrix, T3 and T4 represent all the elements in the first row of the two-dimensional matrices Y3 and Y4, that is, the additionally added learnable feature embedding X class_3 and X class_4 ;
[0048] Finally, the feature vectors T3 and T4 output by the Transformer encoder module are fused to form the final encoding representation of the input distorted image and the corresponding edge structure map; the specific calculation formula is as follows:
[0049] Y = Concat(T3, T4)
[0050] Among them, Concat(·) represents the concatenation operation of features, and Y represents the final encoded representation of the input distorted image and the corresponding edge structure map.
[0051] Furthermore, the training is based on the image quality score prediction network of multi-scale edge feature fusion to obtain the reference-free screen content image quality score prediction model as follows:
[0052] The final encoded representation Y of the distorted image and the corresponding edge structure map is input into two different sets of fully connected layers and activation layers, wherein the first set of fully connected layers and activation layers is used to predict the distorted image I a The mass fraction F Score , the second set of fully connected layers and activation layers are used to predict the distorted image I a Distortion type F Style ;
[0053] Then, according to the loss function of the image quality score prediction network based on multi-scale edge feature fusion, the gradient of each parameter in the deep network for distorted image quality score evaluation is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; the calculation formula of the loss function Loss is as follows:
[0054]
[0055]
[0056] Loss = β1 × L score +β2×L style
[0057] Among them, m is the number of samples in the training set, L CE (·) represents the cross entropy loss function, L score , L style denote the quality score prediction loss and distortion type prediction loss respectively, y i represents the true quality score of the i-th image sample, represents the predicted quality score of the i-th image sample after passing through the network, t i represents the true distortion type of the i-th image sample, represents the predicted distortion type of the i-th image sample after passing through the network, β1 and β2 represent the weights of the two types of loss functions respectively;
[0058] The training is repeated in batches until the calculated loss value converges and stabilizes, the network parameters are saved, and the training process of the reference-free screen content image quality assessment network based on multi-scale edge feature fusion is completed.
[0059] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for assessing the image quality of screen content without reference based on multi-scale edge feature fusion as described above is implemented.
[0060] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for assessing the image quality of screen content without reference based on multi-scale edge feature fusion as described above is implemented.
[0061] Compared with the prior art, the present invention and its preferred embodiment have the following beneficial effects:
[0062] The present invention aims to solve the problem of image shallow feature degradation caused by the increase of network depth in the existing deep learning-based no-reference screen content image quality assessment model. The edge structure information of the original distorted image feature map is enhanced by the edge structure map corresponding to the distorted screen content image, so as to improve the performance of the no-reference screen content image quality assessment model. The proposed method can integrate multi-scale distorted image features and image edge structure features for learning, and introduces the Transformer encoder to further model the non-local correlation of multi-scale image features extracted from the convolutional neural network, which can accurately and effectively predict the quality assessment score of the distorted screen content image. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.
[0064] Figure 2 This is a diagram of the network model structure of an embodiment of the present invention.
[0065] Figure 3 This is a structural diagram of a Transformer encoder according to an embodiment of the present invention.
[0066] Figure 4 2 is a block diagram of a multilayer perceptron in a Transformer encoder according to an embodiment of the present invention. DETAILED DESCRIPTION
[0067] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:
[0068] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0069] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0070] The specific design process of the non-reference screen content image quality assessment method based on multi-scale edge feature fusion proposed in the present invention is as follows: Figure 1-Figure 4 As shown,
[0071] This embodiment provides specific steps:
[0072] Step S1, preprocessing the data in the distorted screen content image dataset, firstly using the Gaussian Laplacian operator to extract image edge features on the grayscale image of each distorted image, generating an edge structure map dataset corresponding to the distorted screen content image dataset, and then dividing the two datasets into a training set and a test set in a unified manner;
[0073] Step S2, designing a twin network that supports dual-source input, and using the designed network to perform multi-scale feature extraction and feature fusion on the input distorted image and the corresponding edge structure graph;
[0074] Step S3, designing a Transformer encoder module, which consists of two Transformer encoders with the same structure and is used to form a global information representation of features at different scales;
[0075] Step S4, designing an image quality score prediction network based on multi-scale edge feature fusion, and training the designed network to obtain a reference-free screen content image quality score prediction model;
[0076] Step S5: input the distorted image and the corresponding edge structure map into the trained image quality score prediction network model based on multi-scale edge feature fusion, and output the quality assessment score of the distorted image.
[0077] Wherein, step S1 comprises the following steps:
[0078] Step S11, constructing a Laplacian Gaussian convolution kernel with a window size of H×W and a standard deviation of σ according to the Laplacian Gaussian operator.
[0079] Step S12: convert each distorted screen content image I into a grayscale image g, and perform a convolution operation on the converted grayscale image g using the Gaussian Laplacian convolution kernel obtained in step S11 to obtain an intermediate result image I. cov .
[0080] Step S13: The intermediate result image I obtained in step S12 cov Threshold processing is performed to determine the pixel points with edge strength greater than the specified threshold as edge points, and set their value to 255. Pixel points with edge strength less than or equal to the specified threshold are eliminated and their value is set to 0, and the final edge structure diagram I is obtained. edge .
[0081] Step S14, generating an edge structure graph corresponding to each distorted image in the distorted screen content image dataset through steps S11, S12 and S13, obtaining an edge structure graph dataset corresponding to the distorted screen content image dataset, and dividing the two datasets into a training set and a test set in a unified manner, that is, a distorted screen content image and its corresponding edge structure graph belong to the same training set or the same test set.
[0082] Step S2 includes the following steps:
[0083] Step S21: Use two ResNet50 networks with shared parameters as the backbone network to transform the distorted image (such as I a ) and the corresponding edge structure diagram (such as I e ) are respectively input into the two branch sub-networks for multi-scale feature extraction. In order to achieve parameter sharing between the two branch sub-networks, the edge structure diagram I e Perform channel duplication to obtain the distorted image I a The same number of channels. Since the image features extracted by the shallow network contain more detailed information, while the image features extracted by the deep network contain more semantic information, this embodiment uses the stage three and stage four in the backbone network ResNet50 as the feature extraction module.
[0084] Step S22: Input a batch of distorted images and the corresponding edge structure graphs in the training set obtained in step S1 into the model of step S21 to obtain feature graphs of distorted images of two different scales and feature graphs of the corresponding edge structure graphs. The feature graph of the distorted image output in stage 3 of the backbone network ResNet50 is denoted as F a3 , the feature map of the edge structure map is F e3 , whose dimensions are C3×h3×w3, C3 represents the number of channels of the feature map, h3 and w3 represent the height and width of the feature map respectively; the feature map of the distorted image output in stage 4 of the backbone network ResNet50 is denoted as F a4 , the feature map of the edge structure map is F e4 , whose dimensions are C4×h4×w4, C4 represents the number of channels of the feature map, h4 and w4 represent the height and width of the feature map respectively;
[0085] Step S23: feature fusion is performed on the two distorted image feature maps of different scales obtained in step S22 and the feature map of the corresponding edge structure map, and the sharp edge information in the original distorted image feature map is enhanced by the feature information of the edge structure map. The specific calculation formula is as follows:
[0086]
[0087]
[0088] in, represents the matrix addition operation, and F3 represents the fused distorted image feature map F a3 and edge structure feature map F e3 The feature map is obtained, whose dimension is C3×h3×w3, and F4 represents the fused distorted image feature map F a4 and edge structure feature map F e4 The feature map is obtained, whose dimension is C4×h4×w4.
[0089] Step S3 includes the following steps:
[0090] Step S31, process the two different scale image features F3 and F4 formed by fusing the distorted image feature map and the edge structure map feature map obtained in step 23, and obtain the processed feature maps F′3 and F′4, and the dimensions of F′3 and F′4 are the same. Specifically, the feature map F3 is input into a maximum pooling layer and a convolution layer with a convolution kernel of 1×1 in sequence, and its dimension is adjusted to D×h4×w4; the feature map F4 is input into a convolution layer with a convolution kernel of 1×1, and its dimension is adjusted to D×h4×w4. The calculation formula is as follows:
[0091] F′3=ReLU(Conv1(ReLU(Maxpool(F3))))
[0092] F′4=ReLU(Conv2(F4))
[0093] Among them, ReLU(·) represents the ReLU activation function, Maxpool(*) represents the spatial maximum pooling layer with a step size of 2, Conv1(*) and Conv2(*) represent two convolutional layers with a convolution kernel of 1×1, respectively, and F′3 and F′4 represent the feature maps obtained by processing the feature maps F3 and F4, respectively.
[0094] Step S32: Change the feature dimensions of the feature graphs F′3 and F′4 obtained in step S31 from D×h4×w4 to N×D (where N=h4×w4) through a dimensional transformation operation, and obtain two-dimensional input matrices X3 and X4. Then, add an additional learnable feature embedding X to the first row of the two-dimensional input matrices X3 and X4, respectively. class_3 and X class_4 , whose dimensions are all 1×D, adding learnable feature embedding X class_3 and X class_4 The dimensions of X3 and X4 after ts are changed to (N+1)×D. Then add a learnable position embedding X to the two-dimensional input matrix X3 and X4 respectively. pos_3 and X pos_4, whose dimensions are (N+1)×D, thus obtaining the final two-dimensional input matrices Z3 and Z4. The specific calculation formula is as follows:
[0095] X3=Reshape(F′3)
[0096] X4=Reshape(F′4)
[0097]
[0098]
[0099] Among them, Reshape(·) represents the dimension transformation operation, represents the matrix addition operation, and Concat(·) represents the feature concatenation operation.
[0100] Step S33, design a Transformer encoder module, which consists of two Transformer encoders with the same structure. Each Transformer encoder is composed of a layer normalization module, a multi-head self-attention module (denoted as MHSA), a random dropout layer, a layer normalization module, a multi-layer perceptron module (denoted as MLP) consisting of two fully connected layers with GELU activation function, and a random dropout layer. The two-dimensional input matrices Z3 and Z4 obtained in step S32 are respectively input into the two Transformer encoders to obtain non-local representations of image features of different scales. Take the additional learnable feature embedding X added in each two-dimensional input matrix Z3 and Z4 class_3 and X class_4 The output of the Transformer encoder is expressed as the final result. The specific calculation formula is as follows:
[0101] Z′3=Dropout(MHSA(LN(Z3)))+Z3
[0102] Y3=Dropout(MLP(LN(Z′3)))+Z′3
[0103] T3=Y3[:0]
[0104] Z′4=Dropout(MHSA(LN(Z4)))+Z4
[0105] Y4=Dropout(MLP(LN(Z′4)))+Z′4
[0106] T4=Y4[:0]
[0107] Among them, LN(·) represents the layer normalization operation, Dropout(·) represents the random dropout operation, MHSA(·) represents the operation in the multi-head self-attention module, Z′3 and Z′4 represent the intermediate results of the original two-dimensional input matrices Z3 and Z4 after the multi-head self-attention module, MLP(·) represents the operation in the MLP module, Y3 and Y4 represent the output results of the intermediate results Z′3 and Z′4 after the MLP module, “[:0]” represents the matrix slicing operation (i.e., taking all the elements in the first row of the two-dimensional matrix), T3 and T4 represent all the elements in the first row of the two-dimensional matrices Y3 and Y4 (i.e., the additionally added learnable feature embedding X class_3 and X class_4 ).
[0108] Step S34: feature fusion is performed on the feature vectors T3 and T4 output by the Transformer encoder module obtained in step S33 to form the final encoding representation of the input distorted image and the corresponding edge structure graph. The specific calculation formula is as follows:
[0109] Y = Concat(T3, T4)
[0110] Among them, Concat(·) represents the concatenation operation of features, and Y represents the final encoded representation of the input distorted image and the corresponding edge structure map.
[0111] Step S4 includes the following steps:
[0112] Step S41: input the final encoded representation Y of the input distorted image and the corresponding edge structure graph obtained in step S34 into two different sets of fully connected layers and activation layers. The first set of fully connected layers and activation layers is used to predict the distorted image I a The mass fraction F Score , the second set of fully connected layers and activation layers are used to predict the distorted image I a Distortion type F Style .
[0113] Step S42: Design a loss function for the reference-free screen content image quality assessment network based on multi-scale edge feature fusion, use the back propagation method to calculate the gradient of each parameter in the deep network for distorted image quality score assessment, and use the stochastic gradient descent method to update the parameters. The calculation formula of the loss function Loss is as follows:
[0114]
[0115]
[0116] Loss = β1 × L score +β2×L style
[0117] Among them, m is the number of samples in the training set, L CE (·) represents the cross entropy loss function, L score , L style denote the quality score prediction loss and distortion type prediction loss respectively, y i represents the true quality score of the i-th image sample, represents the predicted quality score of the i-th image sample after passing through the network, t i represents the true distortion type of the i-th image sample, It represents the predicted distortion type of the i-th image sample after passing through the network, and β1 and β2 represent the weights of the two types of loss functions respectively.
[0118] Step S43, repeating the above steps S41 to S42 in batches until the loss value calculated in step S42 converges and stabilizes, saving the network parameters, and completing the training process of the reference-free screen content image quality assessment network based on multi-scale edge feature fusion.
[0119] Step S5 includes the following steps:
[0120] Step S51: input the distorted screen content image and its corresponding edge structure graph in the test set into the trained image quality score prediction network model based on multi-scale edge feature fusion, and output the quality assessment score of the distorted image.
[0121] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0122] The present invention is described with reference to flowcharts of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process in the flowchart, as well as the combination of processes in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.
[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple functions specified in a flowchart.
[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.
[0125] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
[0126] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of reference-free screen content image quality assessment methods based on multi-scale edge feature fusion under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.
Claims
1. A method for assessing screen content image quality without reference based on multi-scale edge feature fusion, characterized in that: The following steps are involved: Firstly, the data in the distorted screen content image dataset is preprocessed, and the image edge features are extracted on the grayscale image of each distorted image using the Gaussian Laplacian operator to generate an edge structure graph dataset corresponding to the distorted screen content image dataset. Then, the two datasets are divided into a training set and a test set in a unified way. Train an image quality score prediction network based on multi-scale edge feature fusion to obtain a reference-free screen content image quality score prediction model; The image quality score prediction network based on multi-scale edge feature fusion uses a twin network supporting dual-source input to perform multi-scale feature extraction and feature fusion on the input distorted image and the corresponding edge structure graph; Then use the Transformer encoder module to form a global information representation of features at different scales; Finally, the distorted image and the corresponding edge structure map are input into the trained image quality score prediction network model based on multi-scale edge feature fusion, and the quality assessment score of the distorted image is output; The data preprocessing of the data in the distorted screen content image data set specifically includes the following steps: Step S11, constructing a Laplacian Gaussian convolution kernel with a window size of H×W and a standard deviation of σ according to the Laplacian Gaussian operator; Step S12: convert each distorted screen content image I into a grayscale image g, and perform a convolution operation on the converted grayscale image g using the Gaussian Laplacian convolution kernel obtained in step S11 to obtain an intermediate result image I. cov ; Step S13: The intermediate result image I obtained in step S12 cov Threshold processing is performed to determine the pixel points with edge strength greater than the specified threshold as edge points, and set their value to 255. Pixel points with edge strength less than or equal to the specified threshold are eliminated and their value is set to 0, and the final edge structure diagram I is obtained. edge ; Step S14, generating an edge structure graph corresponding to each distorted image in the distorted screen content image dataset through steps S11, S12 and S13, obtaining an edge structure graph dataset corresponding to the distorted screen content image dataset, and dividing the two datasets into a training set and a test set in a unified manner, that is, a distorted screen content image and its corresponding edge structure graph belong to the same training set or the same test set; The twin network supporting dual-source input is specifically: Two ResNet50s with shared parameters are used as the backbone network. The distorted image and the corresponding edge structure map are respectively input into two branch sub-networks for multi-scale feature extraction. In order to achieve parameter sharing between the two branch sub-networks, the edge structure map needs to be channel-copied to obtain the same number of channels as the distorted image. Then, the stages 3 and 4 in the backbone network ResNet50 are used as feature extraction modules. The Siamese network that supports dual-source input is used to perform multi-scale feature extraction and feature fusion on the input distorted image and the corresponding edge structure map. Specifically: Input a batch of distorted images and the corresponding edge structure graphs in the training set into the twin network that supports dual-source input, and obtain feature graphs of distorted images of two different scales and feature graphs of the corresponding edge structure graphs; the feature graph of the distorted image output in stage 3 of the backbone network ResNet50 is recorded as F a3 , the feature map of the edge structure map is F e3 , whose dimensions are C3×h3×w3, C3 represents the number of channels of the feature map, h3 and w3 represent the height and width of the feature map respectively; the feature map of the distorted image output in stage 4 of the backbone network ResNet50 is denoted as F a4 , the feature map of the edge structure map is F e4 , whose dimensions are C4×h4×w4, C4 represents the number of channels of the feature map, h4 and w4 represent the height and width of the feature map respectively; The feature maps of the two distorted images of different scales and the feature maps of the corresponding edge structure maps are fused, and the sharp edge information in the feature map of the original distorted image is enhanced through the feature information of the edge structure map. The specific calculation formula is as follows: in, represents the matrix addition operation, and F3 represents the fused distorted image feature map F a3 and edge structure feature map F e3 The feature map is obtained, whose dimension is C3×h3×w3, and F4 represents the fused distorted image feature map F a4 and edge structure feature map F e4 Then we get a feature map with the dimension of C4×h4×w4; The specific process of using the Transformer encoder module to form a global information representation of features at different scales is: The two different-scale image features F3 and F4 formed by fusing the distorted image feature map and the edge structure map feature map output by the twin network supporting dual-source input are processed to obtain the processed feature maps F'3 and F'4, and the dimensions of F'3 and F'4 are the same: Specifically, the feature map F3 is input into a maximum pooling layer and a convolution layer with a convolution kernel of 1×1 in sequence, and its dimension is adjusted to D×h4×w4; the feature map F4 is input into a convolution layer with a convolution kernel of 1×1, and its dimension is adjusted to D×h4×w 4； The calculation formula is as follows: F'3=ReLU(Conv1(ReLU(Maxpool(F3)))) F'4 = ReLU(Conv2(F4)) Wherein, ReLU(·) represents the ReLU activation function, Maxpool(*) represents the spatial maximum pooling layer with a stride of 2, Conv1(*) and Conv2(*) represent two convolutional layers with a convolution kernel of 1×1, respectively, and F'3 and F'4 represent the feature maps obtained by processing the feature maps F3 and F4, respectively; Then, the feature maps F'3 and F'4 are transformed into N×D by dimensionality transformation, where N=h4×w4, and the two-dimensional input matrices X3 and X4 are obtained. Then, an additional learnable feature embedding X is added to the first row of the two-dimensional input matrices X3 and X4 respectively. class_3 and X class_4 , whose dimensions are all 1×D, adding learnable feature embedding X class_3 and X class_4 The dimensions of X3 and X4 after that become (N+1)×D; then add a learnable position embedding X to the two-dimensional input matrix X3 and X4 respectively. pos_3 and X pos_4 , whose dimensions are (N+1)×D, thus obtaining the final two-dimensional input matrices Z3 and Z4; the specific calculation formula is as follows: X3=Reshape(F'3) X4=Reshape(F'4) Among them, Reshape(·) represents the dimension transformation operation, represents matrix addition operation, Concat(·) represents feature concatenation operation; The Transformer encoder module is composed of two Transformer encoders with the same structure; Each Transformer encoder is composed of a layer normalization module, a multi-head self-attention module MHSA, a random dropout layer, a layer normalization module, a multi-layer perceptron module MLP consisting of two fully connected layers with GELU activation function, and a random dropout layer; The two-dimensional input matrices Z3 and Z4 are respectively input into the two Transformer encoders to obtain non-local representations of image features at different scales; the additional learnable feature embedding X added to each two-dimensional input matrix Z3 and Z4 is taken class_3 and X class_4 The output of the Transformer encoder is expressed as the final result. The specific calculation formula is as follows: Z'3=Dropout(MHSA(LN(Z3)))+Z3 Y3=Dropout(MLP(LN(Z'3)))+Z3' T3=Y3[:0] Z'4=Dropout(MHSA(LN(Z4)))+Z4 Y4=Dropout(MLP(LN(Z'4)))+Z4' T4=Y4[:0] Among them, LN(·) represents the layer normalization operation, Dropout(·) represents the random dropout operation, MHSA(·) represents the operation in the multi-head self-attention module, Z'3 and Z'4 represent the intermediate results of the original two-dimensional input matrices Z3 and Z4 after the multi-head self-attention module, MLP(·) represents the operation in the MLP module, Y3 and Y4 represent the output results of the intermediate results Z'3 and Z'4 after the MLP module, "[:0]" represents the matrix slicing operation, that is, taking all the elements in the first row of the two-dimensional matrix, T3 and T4 represent all the elements in the first row of the two-dimensional matrix Y3 and Y4, that is, the additionally added learnable feature embedding X class_3 and X class_4 ; Finally, the feature vectors T3 and T4 output by the Transformer encoder module are fused to form the final encoding representation of the input distorted image and the corresponding edge structure map; the specific calculation formula is as follows: Y=Concat(T3,T4) Among them, Concat(·) represents the concatenation operation of features, and Y represents the final encoded representation of the input distorted image and the corresponding edge structure map.
2. The method for non-reference screen content image quality assessment based on multi-scale edge feature fusion according to claim 1, characterized in that: The training of the image quality score prediction network based on multi-scale edge feature fusion to obtain the reference-free screen content image quality score prediction model is specifically: The final encoded representation Y of the distorted image and the corresponding edge structure map is input into two different sets of fully connected layers and activation layers, wherein the first set of fully connected layers and activation layers is used to predict the distorted image I a The mass fraction F Score , the second set of fully connected layers and activation layers are used to predict the distorted image I a Distortion type F Style ; Then, according to the loss function of the image quality score prediction network based on multi-scale edge feature fusion, the gradient of each parameter in the deep network for distorted image quality score evaluation is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; the calculation formula of the loss function Loss is as follows: Loss=β1×L score +β2×L style Among them, m is the number of samples in the training set, L CE (·) represents the cross entropy loss function, L score , L style denote the quality score prediction loss and distortion type prediction loss respectively, y i represents the true quality score of the i-th image sample, represents the predicted quality score of the i-th image sample after passing through the network, t i represents the true distortion type of the i-th image sample, represents the predicted distortion type of the i-th image sample after passing through the network, β1 and β2 represent the weights of the two types of loss functions respectively; The training is repeated in batches until the calculated loss value converges and stabilizes, the network parameters are saved, and the training process of the reference-free screen content image quality assessment network based on multi-scale edge feature fusion is completed.
3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the reference-free screen content image quality assessment method based on multi-scale edge feature fusion as described in claim 1 or 2 is implemented.
4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for assessing the quality of screen content without reference based on multi-scale edge feature fusion as described in claim 1 or 2 is implemented.