A night image quality evaluation method

Abstract

The application provides a night image quality evaluation method. First, different scales of night images are obtained through downsampling, and then the original image and the image sequence after downsampling are simultaneously sent into a feature extraction unit; in the feature extraction stage, two types of quality-related features, namely, bottom visual features and high-level semantic features, are extracted. The bottom visual features include texture features and contrast features of the night image; for the high-level semantic features, first, a deep convolution network is used to extract primary high-level semantic features of the night image, and then mean and variance two feature functions are used to aggregate the primary high-level semantic features to obtain secondary high-level semantic features as the final high-level semantic features of the night image. Then, the extracted bottom features and high-level features are aggregated. Finally, a partial least squares regression method is used to regress the aggregated features, so as to obtain an objective score of the night image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of night image processing, and particularly relates to a night image quality evaluation method. BACKGROUND

[0002] Night image processing technology is widely applied in the fields of video monitoring and intelligent transportation. Since night images are images captured in low-illumination environment, there are generally problems such as fuzzy details, low brightness, low contrast and color distortion, which greatly reduce the application value. Therefore, it is of great theoretical research value to study the quality evaluation model of night images. In addition, an efficient night image quality evaluation model can be applied to the performance evaluation and optimization of night image enhancement, defogging and other algorithms. Therefore, the night image quality evaluation model also has very important practical application value.

[0003] Due to the particularity of night scenes and the lack of public data sets, there are still few studies on the quality evaluation of night images. Night images have problems such as fuzzy details, low brightness, low contrast and color distortion, and their quality needs to be evaluated. Both low-level visual features and high-level semantic features have an impact on the quality of night images. However, current night image algorithms only consider low-level visual features or high-level semantic features. It is necessary to design a night image quality evaluation method that is more consistent with human perception and has more accurate prediction performance. SUMMARY

[0004] In view of the above technical problems, the present application provides a night image quality evaluation method. The present application first extracts low-level visual features and high-level semantic features of a night image, then aggregates the low-level and high-level features, and uses a partial least squares regression (PLSR) method to regress the aggregated features, thereby obtaining an objective score of the quality of the night image.

[0005] This invention proposes a method for evaluating the quality of nighttime images. Considering that the human visual system perceives different qualities of images at different scales, the method first acquires nighttime images at different scales through downsampling. Then, the original images and the downsampled image sequences are simultaneously fed into a feature extraction unit. In the feature extraction stage, two types of quality-related features are extracted: low-level visual features and high-level semantic features. Low-level visual features include texture and contrast features of the nighttime images. For high-level semantic features, a deep convolutional network is first used to extract first-level high-level semantic features of the nighttime images. Then, mean and variance feature functions are used to aggregate the first-level high-level semantic features to obtain second-level high-level semantic features as the final high-level semantic features of the nighttime images. Next, the extracted low-level and high-level features are aggregated. Finally, partial least squares regression is used to regress the aggregated features, thereby obtaining an objective score for the quality of the nighttime images. Attached Figure Description

[0006] Figure 1 This is a flowchart illustrating the image quality evaluation algorithm of the present invention. Detailed Implementation

[0007] The following is in conjunction with the appendix Figure 1 The image quality evaluation algorithm of the present invention will be further explained.

[0008] like Figure 1 As shown, the image quality assessment algorithm of the present invention includes four steps: step 1 multi-scale representation; step 2 feature extraction; step 3 multi-layer feature aggregation; and step 4 quality score prediction.

[0009] Step 1, multi-scale representation.

[0010] Let the original nighttime image be denoted as I, and the downsampled image be denoted as I. S Then include I S The image sequences of I and I are sent to the feature extraction unit for feature extraction. The multi-scale features of nighttime images can be represented as follows:

[0011]

[0012] Where O represents the features extracted from the original nighttime image I, and S represents the features extracted from I. S Extracted features It is a concatenation operator.

[0013] Step 2, Feature Extraction

[0014] Feature extraction includes 2.1 low-level visual feature extraction and 2.2 high-level semantic feature extraction.

[0015] Wherein, the 2.1 bottom view angle feature extraction further comprises 2.1.1 texture feature extraction and 2.2.2 contrast feature extraction.

[0016] Wherein, the 2.1.1 texture feature extraction refers to using the gray level co-occurrence matrix (GLCM) to extract the texture feature to describe the texture naturalness. Firstly, the co-occurrence matrix is obtained by calculating the gray image. Assuming that the size of the night image I is M x N, the GLCM is defined as follows:

[0017]

[0018] Some statistics of the GLCM include energy, contrast, and homogeneity, which can represent the texture feature. The formula of the corresponding index is as follows:

[0019]

[0020]

[0021]

[0022] Wherein, f energy represents the energy, f contrast represents the contrast, and f homogeneity represents the homogeneity.

[0023] 2.1.2 Contrast feature extraction refers to using the local histogram of the image to represent the change of the local contrast information of the image. Using I R , I G and I B represent the red, green and blue three channel components of the image I respectively, and I Gray represents the gray image of I. The gray histogram represents the frequency distribution of different gray levels of the image. We denote the gray histogram of the gray image I Gray at the gray level i as H Gray (i), which represents the total number of pixels with the gray level i in I Gray . H Gray (i) can be represented as follows:

[0024]

[0025] Wherein X represents the image pixel set, δ represents the Kronecker function, i ranges from 0 to 255, and represents the gray level of the image. H Gray (i) represents the distribution of the image brightness from black to white. Similarly, the color histogram of the image I can be represented as follows:

[0026]

[0027] where ch e {R, G, B} represents the three color channels of the image. Color histogram represents the frequency distribution of the gray levels of each color component. The RGB histogram of image I can be represented as:

[0028]

[0029] Color-Gray-Difference (CGD) histogram can be obtained by subtracting the gray histogram from the color histogram as shown in equation (9),

[0030] H CGD (i) = H RGB (i) - H Gray (i) (9)

[0031] Based on this, we randomly select a set of image blocks in the whole image, and then calculate the histogram of each image block to describe the local changes of the image. The CGD histogram of the image block is represented as follows:

[0032]

[0033] where the matrix H is the set of color-gray-difference histograms. H is a P x 256 matrix, and P represents the number of image blocks.

[0034] After obtaining the matrix H, Principal Components Analysis (PCA) is used to reduce the dimension of H, and the specific steps are as follows:

[0035] 1) Standardize H by subtracting the average value μ j of each column in H to obtain a new matrix

[0036] The column mean μ j of the matrix H can be represented as equation:

[0037]

[0038] Generate a zero-mean data matrix which can be represented as:

[0039]

[0040] 2) Calculate the covariance matrix C;

[0041]

[0042] 3) Calculate the eigenvalues and corresponding eigenvectors of C, and arrange the eigenvalues from large to small as λ1≥ λ2≥…, λ P, the eigenvectors are arranged in the order of eigenvalues as w1, w2, … w P .

[0043] 4) Select the first m eigenvalues, and the eigenvectors corresponding to the first m eigenvalues are taken as the histogram matrix of the local image block

[0044] The bottom-layer feature of the night image can be expressed as:

[0045]

[0046] 2.2 The high-level semantic feature can be calculated by taking the whole night image as the input of the ResNet-50 without the full connection layer. The first-level high-level feature d extracted from each image can be expressed as:

[0047] d = DCNN (θ, ω) (15)

[0048] In the formula: DCNN (θ, ω) represents the pre-trained ResNet-50 network model, and θ, ω represents the weight parameters of the pre-trained ResNet-50.

[0049] The mean aggregation feature f1 of d can be expressed as:

[0050] f1 = f mean {d} (16)

[0051] The standard deviation aggregation feature f2 of d can be expressed as:

[0052] f2 = f std {d} (17)

[0053] Wherein, f mean {d} represents the mean of d, and f std {d} represents the standard deviation of d.

[0054] The final high-level semantic feature can be expressed as: Wherein, is a splicing operator.

[0055] Step 3, multi-layer feature extraction

[0056] The bottom-layer feature and the high-level feature are aggregated together as the standard of the night image quality evaluation. The aggregated feature can be expressed as

[0057]

[0058] Step 4: The feature f after the aggregation is input into the Partial Least Squares Regression (PLSR) to get the objective score of the night image quality.

[0059] The performance of the algorithm is verified on the NNID database. Pearson linear correlation coefficient (PLCC), Root Mean Squared Error (RMSE), Spearman Rank order Correlation Coefficient (SROCC), KROCC (Kendall Rank-order Correlation Coefficient) and four kinds of evaluation criteria are used to measure the prediction accuracy and monotonicity of the proposed model. The value range of PLCC, SROCC and KROCC is [-1, 1], and the closer the absolute value is to 1, the stronger the correlation and the higher the consistency. The range of RMSE is [0, ∞], and the closer to 0, the better the prediction performance.

[0060]

[0061] Table 1 Test results on the NNID database

[0062] As can be seen from Table 1, for the monotonicity evaluation indicators SROCC and KROCC, the method is the highest on the NNID database; for the prediction accuracy indicators PLCC and RESM, the value of PLCC of the method is the highest, and the value of RESM is the lowest. In summary, the proposed method achieves good night image quality evaluation results and can maintain good consistency with the subjective evaluation of human visual perception.

[0063] For those skilled in the art, the above embodiments are illustrative rather than limiting, and the scope of the present application is defined by the claims rather than the above embodiments, and therefore all changes falling within the meaning and scope of the essential elements of the claims are intended to be covered by the present application. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural.

Claims

1. A method for evaluating the quality of nighttime images, comprising: Step 1, Multi-scale representation, Represent the original nighttime image as The original nighttime image was downsampled to obtain , including and The image sequence is sent to the feature extraction unit for feature extraction. The multi-scale features of the night image are represented as follows: （1） ， in, From the original nighttime image Extracted features From Extracted features It is a concatenation operator; Step 2, Feature Extraction: This includes the extraction of low-level visual features and high-level semantic features. The low-level visual features include texture features extracted using the gray-level co-occurrence matrix and contrast features extracted using the color-gray-level histogram. The high-level semantic features are extracted from the ResNet-50 network model. The contrast feature extraction refers to using local histograms of an image to represent changes in local contrast information. , and Representing images respectively The red, green, and blue channel components, express A grayscale image; a grayscale histogram represents the frequency distribution of different gray levels in an image; a grayscale image. In grayscale The grayscale histogram is denoted as ,express medium gray level is Total number of pixels; （6）， in Represents the set of image pixels. Represents the Kronecker function. The range is 0~255, representing the gray level of the image; It represents the distribution of image brightness from black to white; image The color histogram is shown below: （7）， in The image has three color channels, and the color histogram represents the gray-level frequency distribution of each color component; the image... The RGB histogram is represented as: （8）， Subtracting the grayscale histogram from the color histogram yields the color-grayscale-difference histogram, as shown in formula (9). （9）， A set of image patches is randomly selected from the entire image, and then the histogram of each image patch is calculated to describe the local changes in the image. The CGD histogram of the image patch is represented as follows: （10）， Among them, matrix A collection of color-grayscale difference histograms. It is The matrix, Indicates the number of image patches. Obtain the matrix Then, principal component analysis was used to... The specific steps for dimensionality reduction are as follows: 1) To Standardize the process and... Subtract the average of the column from each element in the column. A new matrix is ​​obtained. ; matrix column mean Represented as: （11） ， Generate a data matrix with zero mean Represented as: （12）， 2) Calculate the covariance matrix ; （13）， 3) Calculation The eigenvalues ​​and corresponding eigenvectors are arranged in descending order of eigenvalues. The eigenvectors are arranged in order of their eigenvalues. ; 4) Before selection Each eigenvalue, the first The eigenvectors corresponding to each eigenvalue are used as the histogram matrix of the local image patch. ; The low-level features of nighttime images are represented as follows: （14）， Step 3, Multi-layer feature aggregation: The low-level visual features and high-level semantic features extracted in Step 2 are concatenated. Step 4: Partial least squares regression is used to regress the aggregated features from Step 3 to obtain an objective score for the quality of nighttime images.

2. The nighttime image quality evaluation method according to claim 1, characterized in that, The texture feature extraction refers to using the Gray-Level Co-occurrence Matrix (GLCM) to extract texture features to describe the naturalness of the texture. First, the GLCM is calculated from the gray-level image to obtain the co-occurrence matrix, assuming a nighttime image. The size is The definition of GLCM is as shown in formula (2): （2） ， （3）， （4）， （5）， in, Indicates energy. Indicates contrast. Indicates homogeneity.

3. The nighttime image quality evaluation method according to any one of claims 1-2, characterized in that, The high-level semantic features are calculated by using the entire nighttime image as input to a ResNet-50 with the fully connected layers removed. Each image extracts a first-level high-level feature. Represented as: （15）， In the formula: This represents a pre-trained ResNet-50 network model. , This represents the weight parameters of the pre-trained ResNet-50; right Using mean aggregation features Represented as: （16）， right Aggregation features based on standard deviation Represented as: （17）， in, Indicates to Find the mean. Indicates to Calculate the standard deviation; The final high-level semantic features are represented as follows: ,in, It is a concatenation operator.

4. The nighttime image quality evaluation method according to claim 1, wherein, Step 3, multi-layer feature extraction, refers to combining low-level and high-level features as a standard for evaluating the quality of nighttime images. The aggregated features are represented as follows: （18）。

Images