A SR image quality evaluation method and system based on KLT theory and parallax compensation

Abstract

The invention discloses an SR image quality evaluation method and system based on KLT theory and parallax compensation, which comprises the following steps: obtaining a to-be-predicted super-resolution reconstructed image for frequency domain filtering and extracting mid-high frequency domain images as model input; calculating a covariance vector matrix of each frequency domain image channel and calculating a KLT coefficient matrix according to a KLT kernel, spectrally decomposing and combining to obtain image detail feature maps of different depths; constructing image structure features from shallow feature maps and obtaining image structure information quality scores through a first regression network; using convolutional neural networks of different depths to respectively refine and integrate image features of multi-level deep feature maps, and dividing the images into two parts according to the principle of parallax compensation, and obtaining left eye and right eye detail feature quality scores through second and third regression networks; adaptively weighting the three quality scores to obtain a final objective evaluation quality prediction score; while perceiving the overall structure of the image, the invention utilizes the left-eye parallax compensation principle to make the objective evaluation result close to human eye perception.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a SR image quality evaluation method and system based on KLT theory and parallax compensation. Background Art

[0002] Since the fundamental task of image super-resolution (SR) is to reconstruct a high-resolution image from a low-resolution image, multiple different low-resolution images may reconstruct the same high-resolution image. This instability makes it still a challenging problem to objectively and accurately evaluate the reconstruction effect of these SR images.

[0003] There are many methods for image quality assessment (IQA) of super-resolution reconstruction, including full-reference method, semi-reference method and no-reference method. Since super-resolution reconstruction images are prone to spatial misalignment, full-reference and semi-reference quality assessment methods are often not accurate enough on SR images and deviate from the subjective perception of the human eye.

[0004] SR images fill in pixel information rather than reconstruct the entire image. The color restoration accuracy of the reconstructed image is very high. Currently, feature extraction in IQA methods mainly includes two types: theory-based methods and deep learning-based methods. Both have shown certain advantages and also have certain problems.

[0005] Although traditional machine learning methods that use theory plus logistic regression ideas have certain theoretical and generalization capabilities, there is a "ceiling" in the amount of data when extracting information. When the amount of data exceeds its information fitting ability, the actual application effect of these methods is not good. In contrast, methods based on convolutional neural networks have strong information fitting capabilities, but on the one hand, these methods do not combine the characteristics of SR tasks for feature extraction, and on the other hand, these methods are difficult to avoid the problems of convolutional neural networks being too dependent on data sets, the interpretability of the model, and insufficient generalization ability.

[0006] Therefore, how to provide a no-reference quality evaluation method that is close to both the SR image characteristics and the human eye perception is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] In view of this, the present invention provides a SR image quality evaluation method and system based on KLT theory and parallax compensation, which aims to perceive the overall structure of the image while making full use of the image detail features by using the principle of left-eye and right-eye parallax compensation, so that the objective evaluation results are close to human eye perception, and combined with KLT theory to ensure generalization performance.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A SR image quality evaluation method based on KLT theory and parallax compensation includes the following steps:

[0010] S1. Obtaining a super-resolution reconstructed image of image quality to be predicted;

[0011] S2. Perform frequency domain filtering on the super-resolution reconstructed image and extract the mid- and high-frequency domain images that are more relevant to the human eye’s perception sensitivity, and input them into the trained deep SR image quality evaluation network model;

[0012] S3. Calculate the covariance vector matrix of each frequency domain image channel, and calculate the KLT coefficient matrix corresponding to each frequency domain image according to the offline constructed KLT kernel;

[0013] S4. Perform spectral decomposition on the KLT coefficient matrix corresponding to each frequency domain image to obtain different spectral components and combine them to obtain coarse-grained image detail feature maps with different degrees of refinement, and input them into the trained CNN-based multi-scale feature refinement network for refinement;

[0014] S5. Refining the coarse-grained image detail feature map representing the shallow features to form an image structural feature, and obtaining a separate image structural information quality score through the trained first regression network;

[0015] S6. Using convolutional neural networks of different depths, the coarse-grained image detail feature maps representing multi-level deep features are respectively refined and integrated, and randomly divided into two parts according to the principle of parallax compensation. By simulating the processing and evaluation process of the left eye and right eye to obtain the image, the quality scores of the left eye and right eye detail features are obtained using the trained second and third regression networks;

[0016] S7. Adaptively weight the left eye and right eye detail feature quality scores and the image structure feature quality scores to obtain the final objective evaluation quality prediction score of the super-resolution reconstructed image.

[0017] Preferably, the specific contents of calculating the covariance vector matrix of each frequency domain image channel in step S3 include:

[0018] Calculate the MSCN coefficient map corresponding to each frequency domain image channel according to the medium and high frequency domain images, and calculate the corresponding covariance vector matrix;

[0019] According to the covariance vector matrix and the offline constructed KLT kernel, the KLT coefficient matrix corresponding to each MSCN coefficient map is calculated.

[0020] Preferably, the specific contents of offline construction of the KLT kernel include:

[0021] Input multiple high-resolution images for frequency domain filtering and extract mid- and high-frequency domain image channels that are more relevant to human eye perception sensitivity;

[0022] Calculate the MSCN coefficient map corresponding to each frequency domain image channel and calculate the corresponding covariance vector matrix;

[0023] Perform eigenvalue decomposition on the covariance matrix corresponding to the image in each frequency domain space to obtain eigenvalues ​​and eigenvectors, arrange the eigenvectors from large to small according to the eigenvalues, and obtain the KLT kernel corresponding to each frequency domain space.

[0024] Preferably, the specific content of step S4 is:

[0025] The KLT coefficient matrix of the image is spectrally decomposed to obtain n different spectral components. The first two spectral components represent the coarse-grained image structure feature map of shallow features, and the first n / 8, n / 4, and n / 2 spectral components represent the coarse-grained image detail feature map of multi-level deep features.

[0026] Preferably, in step S6, the specific contents of image feature refinement and integration are:

[0027] For coarse-grained image detail feature maps with different degrees of refinement, a CNN-based multi-scale refinement structure is used to extract feature information to obtain refined multi-scale deep features.

[0028] The multi-scale deep features are upsampled to different degrees, and the extracted multi-level information is retained through channel splicing. The shallow convolutional neural network is used to further fuse the features after feature splicing to obtain the detailed features of the image.

[0029] Preferably, in step S6, the specific contents of the process of simulating the processing and evaluation of the left eye and the right eye to obtain the image include:

[0030] Through the feature separation operation at the channel level, the left eye detail features and the right eye detail features are separated;

[0031] Weaken the correlation between the two parts of the detailed features to simulate the left and right eye parallax;

[0032] The left eye detail features and the right eye detail features are passed through their own trained regression networks respectively to obtain the quality scores of the left eye and right eye detail features that are close to the overall perception of the human eye.

[0033] Preferably, the specific content of step S7 is: setting three learnable weight parameters, autonomously learning and combining the three parts of feature information obtained by the first, second and third regression networks, and obtaining the final quality score of the objective image evaluation.

[0034] Preferably, the loss of deep SR image quality evaluation network model training includes:

[0035] Minimize the distance between the final objective evaluation quality prediction score and the subjective reference quality score, minimize the distance between the left eye detail feature quality score and the subjective reference quality score, minimize the distance between the right eye detail feature quality score and the subjective reference quality score, and minimize the correlation coefficient between the detail features of the left and right eyes;

[0036] The final loss function is formed by manually setting weights to combine the four losses.

[0037] A SR image quality evaluation system based on KLT theory and parallax compensation, based on the SR image quality evaluation method based on KLT theory and parallax compensation, including a super-resolution reconstruction image acquisition module, a preprocessing module and a trained deep SR image quality evaluation network model;

[0038] The trained deep SR image quality assessment network model includes a multi-layer coarse-grained feature extraction network structure based on KLT theory, a multi-scale feature refinement network structure based on CNN, and a three-branch feature regression network structure based on disparity compensation.

[0039] The multi-layer coarse-grained feature extraction network structure based on KLT theory includes a covariance vector matrix calculation module, an offline constructed KLT kernel and a spectral decomposition module;

[0040] The CNN-based multi-scale feature refinement network structure includes a shallow convolutional neural network feature refinement module and deep convolutional neural network feature refinement modules of different depths;

[0041] The three-branch feature regression network structure based on disparity compensation includes a first regression network, a channel separation module, a second regression network, a third regression network, an adaptive weight learning module and a prediction output module.

[0042] Preferably, the SR image quality evaluation system based on KLT theory and parallax compensation further includes a model training module.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a SR image quality evaluation method and system based on KLT theory and parallax compensation, which has the following advantages:

[0044] By filtering the super-resolution reconstructed image to be predicted in the frequency domain and extracting the mid- and high-frequency domain images that are more relevant to the human eye's perception sensitivity as input, it helps to focus on the mid- and high-frequency image features that are more conducive to the evaluation of SR image quality, while avoiding the influence of color information on the evaluation of SR image quality;

[0045] By extracting image feature vectors based on KLT theory, the generalization ability advantage of the theoretical guidance model is retained to a certain extent;

[0046] The shallow features constitute the image structure features, and a single quality score is obtained through the trained regression network. The multi-level deep features correspond to the convolutional neural networks of different depths to refine the image features. The image detail features obtained by integrating the refined multi-scale deep features are randomly divided into two parts according to the principle of parallax compensation, respectively simulating the processing and evaluation process of the left eye and the right eye to obtain the image, which helps to use the advantage of the strong fitting ability of the deep convolutional neural network to extract and fit the feature information in the case of high data volume;

[0047] By weakening the correlation between the two parts of the detail features and using a trained regression network to obtain a quality score close to the overall perception of the human eye, and then adaptively weighting the three quality scores to obtain the final objective quality prediction score of the super-resolution reconstructed image, we can fully utilize the image detail features by using the principle of left-eye and right-eye parallax compensation while ensuring the overall structure of the perceived image, so that the objective evaluation results are close to human eye perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 The accompanying drawing is a schematic diagram of a SR image quality evaluation method based on KLT theory and parallax compensation provided by the present invention;

[0050] Figure 2 The accompanying drawing is a schematic diagram of a deep SR image quality assessment network model provided by the present invention;

[0051] Figure 3 The accompanying drawing is a schematic diagram of the three-branch feature regression network structure based on disparity compensation provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The embodiment of the present invention discloses a SR image quality evaluation method based on KLT theory and parallax compensation, such as Figure 1 , including the following steps:

[0054] S1. Obtaining a super-resolution reconstructed image of image quality to be predicted;

[0055] S2. Perform frequency domain filtering on the super-resolution reconstructed image and extract the mid- and high-frequency domain images that are more relevant to the human eye’s perception sensitivity, and input them into the trained deep SR image quality evaluation network model;

[0056] S3. Calculate the covariance vector matrix of each frequency domain image channel, and calculate the KLT coefficient matrix corresponding to each frequency domain image according to the offline constructed KLT kernel;

[0057] S4. Perform spectral decomposition on the KLT coefficient matrix corresponding to each frequency domain image to obtain different spectral components and combine them to obtain coarse-grained image detail feature maps with different degrees of refinement, and input them into the trained CNN-based multi-scale feature refinement network;

[0058] S5. Refining the coarse-grained image detail feature map representing the shallow features to form an image structural feature, and obtaining a separate image structural information quality score through the trained first regression network;

[0059] S6. Using convolutional neural networks of different depths, the coarse-grained image detail feature maps representing multi-level deep features are respectively refined and integrated, and randomly divided into two parts according to the principle of parallax compensation. By simulating the processing and evaluation process of the left eye and right eye to obtain the image, the quality scores of the left eye and right eye detail features are obtained using the trained second and third regression networks;

[0060] S7. Adaptively weight the left eye and right eye detail feature quality scores and the image structure feature quality scores to obtain the final objective evaluation quality prediction score of the super-resolution reconstructed image.

[0061] In this embodiment, the frequency domain filtering uses a low-pass filter to decompose the original image to be predicted into multiple spatial frequency bands. Specifically, the multi-scale spatial filtering is performed on the predicted image using the following formula:

[0062]

[0063] Where I represents the original input super-resolution reconstructed image to be predicted, Ll{.} represents the low-pass filter operation, and H l and L l Respectively represent the lth level of low-pass filtering;

[0064] The low-pass filtering adopts a variety of filtering methods such as guided filtering, Gaussian filtering or bilateral filtering. With the change of the filtering index, the decomposition of the image is transferred from the high-frequency band to the low-frequency band. The high-frequency band images of the first three layers are taken as the input images of the trained deep SR image quality evaluation network model based on KLT theory and disparity compensation. The image shape is H×W×3, denoted as O=[O1,O2,O3], O l ∈R H×W , l=1,2,3.

[0065] In order to further implement the above technical solution, the specific content of calculating the covariance vector matrix of each frequency domain image channel in step S3 includes:

[0066] Calculate the MSCN coefficient map corresponding to each frequency domain image channel according to the medium and high frequency domain images, and calculate the corresponding covariance vector matrix;

[0067] According to the covariance vector matrix and the offline constructed KLT kernel, the KLT coefficient matrix corresponding to each MSCN coefficient map is calculated.

[0068] In this embodiment, the calculation method of the KLT coefficient matrix is:

[0069]

[0070] Among them, P l represents the KLT kernel of the lth frequency domain space, represents the covariance vector matrix of the lth frequency domain space image, Y l represents the KLT coefficient matrix of the lth frequency domain spatial image, l = 1, 2, 3, from which the KLT coefficient matrix of the input mid- and high-frequency domain images can be obtained Y = [Y1, Y2, Y3], Y l ∈R H×W .

[0071] In order to further implement the above technical solution, the specific contents of offline construction of KLT kernel include:

[0072] Input multiple high-resolution images for frequency domain filtering and extract mid- and high-frequency domain image channels that are more relevant to human eye perception sensitivity;

[0073] Calculate the MSCN coefficient map corresponding to each frequency domain image channel and calculate the corresponding covariance vector matrix;

[0074] In this embodiment, the process of calculating the MSCN coefficient map can be expressed by the following formula:

[0075]

[0076]

[0077]

[0078] Where i and j are the spatial coordinates of the pixel; c = 1 is a constant that prevents instability when the denominator approaches zero; ω p,q is a 2D circularly symmetric Gaussian weighting function sampled to 3 standard deviations and rescaled to unit volume;

[0079] For a given image of size H×W Extract a set of size The non-overlapping patches of the image are vectorized and combined together to form a new matrix represents the sth vector block, S is The total number of image blocks in ;

[0080] The process of the covariance vector matrix can be expressed as follows:

[0081]

[0082] in, Indicates that through and C∈R K×K The average vector obtained by averaging each row of ;

[0083] Perform eigenvalue decomposition on the covariance matrix corresponding to each image in the frequency domain space to obtain eigenvalues ​​and eigenvectors, and arrange the eigenvectors from large to small according to the eigenvalues ​​to form the KLT kernel P = [p1, p2, ..., p K ]∈R K×K , where p k ∈R K , k=1,2,...,K represents the kth eigenvector, and the input image can be obtained The three KLT kernels P1, P2, and P3 corresponding to the frequency domains are respectively used to calculate the KLT coefficient matrices in the frequency domain space corresponding to the image to be predicted.

[0084] In order to further implement the above technical solution, the specific content of step S4 is:

[0085] Perform spectral decomposition on the KLT coefficient matrix of the image to obtain n different spectral components. Take the first two spectral components to represent the coarse-grained image structure feature map F of the shallow features. coarse1 , the first n / 8, n / 4, and n / 2 spectral components represent the coarse-grained image detail feature map F of multi-level deep features coarse2 、F coarse3 、F coarse4 .

[0086] The coarse-grained structural features of the image F coarse1 Enter the shallow convolutional neural network for feature refinement, and the process can be expressed by the following formula:

[0087] F refined1 =f conv1 (F coarse1 ) where f conv1 represents the feature mapping operation of the shallow convolutional neural network, F refined1 Represents the refined image structure features, which is Figure 2 V1 in

[0088] V1 passes through a regression network trained on this branch, namely the first regression network, and can obtain a separate structural information quality score S1.

[0089] In order to further implement the above technical solution, in step S6, the specific contents of image feature refinement and integration are as follows:

[0090] For coarse-grained image detail feature maps F with different levels of refinement coarse2 、F coarse3 、F coarse4 , use the CNN-based multi-scale refinement structure to extract feature information and obtain the refined multi-scale deep features;

[0091] In this embodiment, since the super-resolution reconstructed image is a pixel supplement to the low-resolution image, there is a strong uncertainty in the pixel information at a fixed position, and to a certain extent, the uncertainty gradually decreases as the image size becomes smaller. In order to extract more accurate detail features, a multi-scale refinement structure based on deep convolution is used to extract feature information according to the characteristics of the super-resolution reconstructed image. The process is expressed by the following formula:

[0092] F refined2 =f conv2 (F coarse2 )

[0093] F refined3 =f conv3 (f concat ((f down (F coarse3 ), f down (F refined2 )))

[0094] F refined4 =f conv4 (f concat ((f down (F coarse4 ), f down (F refined3 )))

[0095] Among them, fconv2 、f conv3 、f conv4 Represents the feature mapping operations of convolutional neural networks at three different depths, F refined2 、F refined3 、F refined4 Respectively represent the image detail features after different depth network refinement, F down represents the downsampling operation, F concat Represents the feature concatenation operation at the channel level;

[0096] The multi-scale deep features are upsampled to different degrees, and the extracted multi-level information is retained through channel splicing. The shallow convolutional neural network is used to further fuse the features after feature splicing to obtain the detailed features of the image;

[0097] In this example, the fusion process of the depth detail features is shown in the following formula:

[0098]

[0099]

[0100] In the formula, F refined2 、F refined3 、F refined4 The refined features after upsampling, f conv5 It means that the features after feature concatenation are further fused using shallow convolutional neural network. refined Represents the multi-scale detail features of the image obtained by feature fusion.

[0101] In order to further implement the above technical solution, in step S6, the specific contents of simulating the processing and evaluation process of the left eye and the right eye to obtain the image include:

[0102] Through the feature separation operation at the channel level, the left eye detail features and the right eye detail features are separated;

[0103] In this embodiment, the expression of feature segmentation is as follows:

[0104] F left , F right =f split (F refined )

[0105] In the formula, f split represents the feature separation operation at the channel level, F left , F right They are the separated left eye detail features and right eye detail features, such as Figure 2 As shown in V1 and V2, these two features will be used for feature analysis of left and right eye disparity;

[0106] Weaken the correlation between the two parts of the detailed features to simulate the left and right eye parallax;

[0107] In this embodiment, this method is used to improve the performance of the network in extracting effective features, and the specific process of reducing feature correlation is expressed by the following formula:

[0108]

[0109] Among them, E(.) represents the vector composed of the mean of each row or column vector in the feature matrix, E{(F left -E{F left})(F right -E{F rieht}) T}Calculates F left and F right The cross-covariance matrix, σ 2 F left and σ 2 F right Respectively represent F left and F right The variance of L4 represents the feature matrix F left and F right The correlation coefficient gives the degree of correlation between the two parts of the features. If it is close to 0, it means that the similarity between the two parts of the features is small, and if it is close to 1, it means that the similarity between the two parts of the features is large;

[0110] The left eye detail features and the right eye detail features are respectively passed through their own trained regression networks to obtain the quality scores of the left eye and right eye detail features that are close to the overall perception of the human eye;

[0111] In this embodiment, based on the obtained fine-grained structural features F refined1 And the refined left and right eye detail features F left 、F right , we obtain the corresponding three quality scores through three different regression networks that have been trained. The specific process is shown in the following formula:

[0112] S1=f reg1 (F refined1 )

[0113] S2=f reg2 (F refined2 )

[0114] S3=f reg3 (F refined3 )

[0115] In the formula, f reg1 、f reg2 、freg3 Corresponding to three regression networks with different parameters, S1, S2, and S3 are the structure quality score, left eye detail quality score, and right eye detail quality score respectively obtained by regression of the three networks.

[0116] In order to further implement the above technical solution, the specific content of step S7 is: set three learnable weight parameters ω1, ω2, ω3, and autonomously learn and combine the three parts of feature information obtained by the first, second and third regression networks to obtain the final image objective evaluation quality score S:

[0117] S=ω1·S1+ω2·S2+·ω3·S3, ω1+ω2+ω3=1.

[0118] In order to further implement the above technical solution, Figure 3 , the losses of deep SR image quality evaluation network model training include:

[0119] Minimize the distance between the final objective evaluation quality prediction score and the subjective reference quality score, minimize the distance between the left eye detail feature quality score and the subjective reference quality score, minimize the distance between the right eye detail feature quality score and the subjective reference quality score, and minimize the correlation coefficient between the detail features of the left and right eyes;

[0120] The final loss function is formed by manually setting weights to combine the four losses.

[0121] In this embodiment, the distance between the final objective quality score and the subjective reference quality score is minimized, which is calculated using the L1 loss between the two quality scores:

[0122]

[0123] Among them, S i represents the corresponding objective prediction quality score of the i-th image block in the training batch in the above-mentioned deep super-resolution reconstruction image quality evaluation network based on KLT theory and disparity compensation, S gt,i represents the human subjective evaluation quality score for reference of the i-th image block in this training batch, N is the total number of image blocks in this training batch, and L1 represents the L1 distance value between the calculated final objective quality score and the subjective reference quality score;

[0124] The two losses of minimizing the distance between the left eye detail quality score and the subjective reference quality score and minimizing the distance between the right eye detail quality score and the subjective reference quality score are based on the principle of parallax compensation. The purpose is to make the two parts of the features corresponding to the left eye and right eye perception close to the subjective perception of the human eye. Specifically:

[0125]

[0126]

[0127] Among them, S 2,i and S 3,i They respectively represent the corresponding objective prediction quality scores of the i-th image block in the training batch in the two detail branches of the above-mentioned deep super-resolution reconstruction image quality evaluation network based on KLT theory and disparity compensation, and L2 and L3 respectively represent the L1 distance values ​​between the calculated left and right eye detail quality scores and the subjective reference quality scores;

[0128] In addition, in order to be close to the disparity compensation principle and improve the feature extraction efficiency of the deep model, a new disparity compensation loss L4 is designed to record the correlation between the left and right eye visual features:

[0129]

[0130] Among them, E(.) represents the vector composed of the mean of each row or column vector in the feature matrix, E{(V i，1 -E{V i，1})(V i，2 -E{V i，2}) T} calculates the F corresponding to the i-th image block in the training batch left and F right The cross-covariance matrix between 2 V i，1 and σ 2 V i，2 Respectively represent the image block F left and F right The variance of L4 represents the feature matrix F corresponding to all image blocks in the training batch. left and F right The average correlation coefficient between the two parts gives the average correlation between the two parts of the features. If it is close to 0, it means that the similarity between the two parts of the features is small, and if it is close to 1, it means that the similarity between the two parts of the features is large.

[0131] The final loss function is formed by manually setting weights to combine the four parts of loss, so as to simultaneously achieve the three goals of making the final quality score close to human eye perception, making better use of the extracted visual features by using the principle of parallax compensation, and reducing the redundancy between features. The final loss function L is specifically:

[0132] L=W1·L1+W2·L2+W3·L3+W4·L4

[0133] Where W1, W2, W3, and W4 are four manually set loss function weight parameters, and L is the overall loss function of the model training. In this example, W1=1.0, W2=0.2, W3=0.2, and W4=0.2 can be set.

[0134] An SR image quality evaluation system based on KLT theory and parallax compensation, based on an SR image quality evaluation method based on KLT theory and parallax compensation, such as Figure 2 and 3 , including a super-resolution reconstruction image acquisition module, a preprocessing module and a trained deep SR image quality evaluation network model;

[0135] The trained deep SR image quality assessment network model includes a multi-layer coarse-grained feature extraction network structure based on KLT theory, a multi-scale feature refinement network structure based on CNN, and a three-branch feature regression network structure based on disparity compensation.

[0136] The multi-layer coarse-grained feature extraction network structure based on KLT theory includes a covariance vector matrix calculation module, an offline constructed KLT kernel and a spectral decomposition module;

[0137] The CNN-based multi-scale feature refinement network structure includes a shallow convolutional neural network feature refinement module and deep convolutional neural network feature refinement modules of different depths;

[0138] The three-branch feature regression network structure based on disparity compensation includes a first regression network, a channel separation module, a second regression network, a third regression network, an adaptive weight learning module and a prediction output module.

[0139] In order to further implement the above technical solution, a SR image quality evaluation system based on KLT theory and parallax compensation also includes a model training module.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A SR image quality evaluation method based on KLT theory and parallax compensation, characterized in that: The following steps are involved: S1. Obtaining a super-resolution reconstructed image of image quality to be predicted; S2. Perform frequency domain filtering on the super-resolution reconstructed image and extract the mid- and high-frequency domain images that are more relevant to the human eye’s perception sensitivity, and input them into the trained deep SR image quality evaluation network model; S3. Calculate the covariance vector matrix of each frequency domain image channel, and calculate the KLT coefficient matrix corresponding to each frequency domain image according to the offline constructed KLT kernel; S4. Perform spectral decomposition on the KLT coefficient matrix corresponding to each frequency domain image to obtain different spectral components and combine them to obtain coarse-grained image detail feature maps with different degrees of refinement, and input them into the trained CNN-based multi-scale feature refinement network for refinement; S5. Refining the coarse-grained image detail feature map representing the shallow features to form an image structural feature, and obtaining a separate image structural information quality score through the trained first regression network; S6. Using convolutional neural networks of different depths, the coarse-grained image detail feature maps representing multi-level deep features are respectively refined and integrated, and randomly divided into two parts according to the principle of parallax compensation. By simulating the processing and evaluation process of the left eye and right eye to obtain the image, the quality scores of the left eye and right eye detail features are obtained using the trained second and third regression networks; S7. Adaptively weight the left eye and right eye detail feature quality scores and the image structure feature quality scores to obtain the final objective evaluation quality prediction score of the super-resolution reconstructed image.

2. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: The specific contents of calculating the covariance vector matrix of each frequency domain image channel in step S3 include: Calculate the MSCN coefficient map corresponding to each frequency domain image channel according to the medium and high frequency domain images, and calculate the corresponding covariance vector matrix; According to the covariance vector matrix and the offline constructed KLT kernel, the KLT coefficient matrix corresponding to each MSCN coefficient map is calculated.

3. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: The specific contents of offline construction of KLT kernel include: Collect and input multiple high-resolution images for frequency domain filtering and extract mid- and high-frequency domain image channels that are more relevant to human eye perception sensitivity; Calculate the MSCN coefficient map corresponding to each frequency domain image channel and calculate the corresponding covariance vector matrix; Perform eigenvalue decomposition on the covariance matrix corresponding to the image in each frequency domain space to obtain eigenvalues ​​and eigenvectors, arrange the eigenvectors from large to small according to the eigenvalues, and obtain the KLT kernel corresponding to each frequency domain space.

4. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: The specific content of step S4 is: The KLT coefficient matrix of the image is spectrally decomposed to obtain n different spectral components. The first two spectral components represent the coarse-grained image structure feature map of shallow features, and the first n / 8, n / 4, and n / 2 spectral components represent the coarse-grained image detail feature map of multi-level deep features.

5. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: In step S6, the specific contents of image feature refinement and integration are as follows: For coarse-grained image detail feature maps with different degrees of refinement, a CNN-based multi-scale refinement structure is used to extract feature information using convolutional neural networks of different depths to obtain refined multi-scale depth features. The multi-scale deep features are upsampled to different degrees, and the extracted multi-level information is retained through channel splicing. The shallow convolutional neural network is used to further fuse the features after feature splicing to obtain the detailed features of the image.

6. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: In step S6, the specific contents of the process of simulating the processing and evaluation of the left eye and right eye images include: Through the feature separation operation at the channel level, the left eye detail features and the right eye detail features are separated; Weaken the correlation between the two parts of the detailed features to simulate the left and right eye parallax; The left eye detail features and the right eye detail features are passed through their own trained regression networks respectively to obtain the quality scores of the left eye and right eye detail features that are close to the overall perception of the human eye.

7. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: The specific content of step S7 is: setting three learnable weight parameters, autonomously learning and combining the three parts of feature information obtained by the first, second and third regression networks, and obtaining the final quality score of the objective evaluation of the image.

8. The SR image quality evaluation method based on KLT theory and parallax compensation according to claim 1, characterized in that: The losses of deep SR image quality evaluation network model training include: Minimize the distance between the final objective evaluation quality prediction score and the subjective reference quality score, minimize the distance between the left eye detail feature quality score and the subjective reference quality score, minimize the distance between the right eye detail feature quality score and the subjective reference quality score, and minimize the correlation coefficient between the detail features of the left and right eyes; The final loss function is formed by manually setting weights to combine the four losses.

9. A SR image quality evaluation system based on KLT theory and parallax compensation, based on the SR image quality evaluation method based on KLT theory and parallax compensation according to any one of claims 1 to 8, characterized in that: It includes a super-resolution reconstruction image acquisition module, a preprocessing module and a trained deep SR image quality evaluation network model; The trained deep SR image quality assessment network model includes a multi-layer coarse-grained feature extraction network structure based on KLT theory, a multi-scale feature refinement network structure based on CNN, and a three-branch feature regression network structure based on disparity compensation. The multi-layer coarse-grained feature extraction network structure based on KLT theory includes a covariance vector matrix calculation module, an offline constructed KLT kernel and a spectral decomposition module; The CNN-based multi-scale feature refinement network structure includes a shallow convolutional neural network feature refinement module and deep convolutional neural network feature refinement modules of different depths; The three-branch feature regression network structure based on disparity compensation includes a first regression network, a channel separation module, a second regression network, a third regression network, an adaptive weight learning module and a prediction output module.

10. The SR image quality evaluation system based on KLT theory and parallax compensation according to claim 9, characterized in that: Also includes model training module.

Images