Quality evaluation method of high dynamic range stereoscopic omnidirectional image based on visual perception

Abstract

The present invention provides a high dynamic range stereo omnidirectional image quality evaluation method based on visual perception, which relates to the field of image quality evaluation technology, and includes the following steps: S1: establishing a monocular perception model, including extracting global color features based on a preset feature extraction process; sampling the viewport according to a preset sampling process; performing symmetrical or asymmetrical distortion quantization processing on the sampled viewport image to obtain symmetrical or asymmetrical distortion measurement features; performing scene characteristic perception feature extraction processing on the sampled viewport image to obtain detail contrast features of left and right distorted images and structural features of left and right distorted images; S2: performing feature aggregation on the symmetrical or asymmetrical distortion measurement features, detail contrast features of left and right distorted images and structural features of left and right distorted images. The method can provide a HSOI image quality evaluation method with higher objective prediction accuracy for omnidirectional image acquisition technology limited to a low dynamic range.

Description

Technical Field

[0001] The present invention relates to the technical field of image quality assessment, and in particular to a high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception. Background Art

[0002] Stereoscopic omnidirectional vision content can capture real scenes within the entire 360°×180° field of view, and because the human eye visual system is binocular, the stereoscopic sense can bring an experience closer to the real world. On the other hand, due to the large field of view, the illumination range in real scenes is very large, and traditional omnidirectional image acquisition technology is usually limited to low dynamic range. Therefore, stereoscopic omnidirectional vision content may have overexposure in bright areas and underexposure in dark areas, especially in areas with inconsistent light sources. Combining it with HDR (high dynamic range) imaging technology to generate HSOI (high dynamic range stereo omnidirectional image) can restore the underlying real scene. HSOI consists of HSOI generation, JPEG XT encoding / decoding, TM (tone mapping) and HMD (head-mounted display device) visualization, etc. These processes introduce distortion and cause HSOI quality to deteriorate.

[0003] Currently, there is a lack of research on HSOI visual quality evaluation. In order to optimize the user terminal visual experience, it is urgent to develop an effective HSOI quality assessment method. Summary of the invention

[0004] The problem solved by the present invention is how to provide a HSOI image quality assessment method with higher objective prediction accuracy for the omnidirectional image acquisition technology limited by the low dynamic range.

[0005] To solve the above problems, the present invention provides a high dynamic range stereo omnidirectional image quality evaluation method based on visual perception, comprising the steps of:

[0006] S1: Establishing a monocular perception model includes extracting global color features based on a preset feature extraction process; sampling the viewport according to a preset sampling process; performing symmetrical or asymmetrical distortion quantization processing on the sampled viewport image to obtain symmetrical or asymmetrical distortion measurement features; performing scene characteristic perception feature extraction processing on the sampled viewport image to obtain detail contrast features of left and right distorted images and structural features of left and right distorted images;

[0007] S2: feature aggregation of symmetrical or asymmetrical distortion measurement features, detail contrast features of left and right distorted images, and structural features of left and right distorted images;

[0008] S3: Establishing a binocular perception model includes: performing binocular fusion processing on the sampled viewport image to obtain features extracted by the binocular fusion model; performing binocular difference processing on the sampled viewport image to obtain binocular difference features; and performing feature aggregation on the features extracted by the binocular fusion model and the binocular difference features;

[0009] S4: Screening the features aggregated in step S2 and step S3 according to a preset feature selection process to obtain a feature set after feature selection;

[0010] S5: Take the feature set after feature selection as input, select the regression model, and get the prediction score.

[0011] In the above methods, for monocular perception, the global color distortion is first measured based on the method combining the antagonistic channel spatial domain and the DCT domain; secondly, to measure the symmetric / asymmetric distortion characteristics, the absolute correlation coefficient and correlation distance of the left and right images are calculated as features based on the MSR decomposition; then, considering the indoor, outdoor and night scene designs, the detail contrast and structural features are extracted based on the LP decomposition. For the binocular perception model, based on the joint image filtering, the binocular fusion map and binocular difference map calculation method are proposed. Based on the binocular fusion map, brightness segmentation is performed, and texture, narrow contour and chromaticity statistical features are extracted according to the high brightness, low darkness and medium brightness areas. This process conforms to the information transmission mechanism of the binocular visual system V1-V2-V4 area. For the binocular difference map, the 7-coefficient derivative map is calculated and the natural statistical features are extracted. This method has higher objective prediction accuracy.

[0012] Furthermore, the preset feature extraction process in step S1 includes:

[0013] S101: Adopt I L and I R Represent the distorted left image and the distorted right image respectively, and I L and I R Convert from RGB space to antagonistic color channels κ1-κ2;

[0014] Among them, the preliminary joint calculation of κ1-κ2 is expressed as: and denote the pixel means of κ1 and κ2 channels respectively; and represent the pixel standard deviation of κ1 and κ2 channels respectively;

[0015] S102: Calculate I respectively L and I R of As the spatial color feature F sc ;

[0016] Among them, the spatial color feature is expressed as and Advanced joint statistical measures Right now

[0017] S103: Based on the two channels κ1 and κ2, I L and I R Perform DCT transformation of 5×5 image block size;

[0018] S104: for the transformed DCT coefficient matrix, remove the DC component, divide the AC component into three frequency bands of low frequency, medium frequency and high frequency, calculate the variance of the three frequency bands of each image block as the frequency band energy, and then calculate the mean of all image blocks as the final energy feature of the corresponding frequency band;

[0019] S105: The final energy feature is used as the DCT domain color feature F dc , the global color feature is represented as F ec ={F sc ,F dc}.

[0020] Furthermore, the preset sampling process in step S1 includes:

[0021] S106: I L and I R Perform viewport sampling to obtain the viewport image V L,m and V R,m , where m = 1, 2, ..., M + 2; the viewport for viewport sampling is I = {I E ,I B}, where I E represents the equatorial region, I B Represents the polar region, in I E In the example, M viewports are uniformly sampled at equal angle intervals, and the angle is set to 2π / M;

[0022] S107: Select I B The corresponding binocular product saliency map S LR The point with the largest pixel value is taken as the center point of the viewport, and the total number of viewports is M+2;

[0023] S108: Binocular product saliency map S LR The steps to obtain are: calculate I L and I R The left and right plane saliency maps S LR , binocular product saliency map S LR is the correlation measure after disparity matching: S LR (i,j)=S L (i,j)·S R((i,j)+d i,j ), where d i,j Represents S at pixel (i, j) L and S R The horizontal disparity of the image is obtained by using the optical flow method.

[0024] Furthermore, the symmetrical or asymmetrical distortion quantization processing in step S1 includes:

[0025] S109: Viewport image V L,m and V R,m The MSR decomposition is performed under three Gaussian function scale parameters l, where m = 1, 2, ..., M + 2; l is set to {25, 100, 240}, and the decomposed illumination components are obtained, which are recorded as and and against and The feature map is calculated using LDP texture descriptor;

[0026] S110: Binary encoding of the image pixels after the second-order derivative is performed in four directions {0°, 45°, 90°, 135°} by LDP. The LDP feature map in each direction can be quantized by a ten-dimensional histogram to obtain and The LDP quantized histogram is expressed as and

[0027] S111: Based on and Calculate the absolute correlation coefficient and correlation distance of each ten-dimensional histogram as the similarity features of the left and right images The symmetric or asymmetric distortion metric feature is obtained as f corr ={T,O}.

[0028] Furthermore, the perceptual feature extraction process of the scene characteristics in step S1 includes:

[0029] S112: Viewport image V L,m and V R,m Perform LP decomposition at three scales, where m = 1, 2, ..., M + 2; and take the three detail layers and the first two basic layers as the layers after LP decomposition;

[0030] Among them, V L,m The output detail layer set is represented as The output base layer set is represented as V R,m The output detail layer set is represented as The output base layer set is represented as

[0031] S113: Based on detail layer D L,m and D R,m Calculate the LBP map and V respectively L,m and V R,m Contrast map of V L,m The contrast diagram is in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; ε is a constant to prevent the denominator from being zero, set to 0.00001; V Rm The contrast diagram is in, Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All are obtained by a two-dimensional circularly symmetric Gaussian weighted filter;

[0032] S114: Using C L Right D L,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features; using C R Right D R,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features respectively; the detail contrast features of the left and right distorted images are expressed as f clbp ;

[0033] S115: According to the basic layer set B′ L,m and B′ R,m Calculate the global structure tensor, perform two-dimensional structure tensor transformation respectively, perform matrix decomposition on the output structure tensor matrix, take the two eigenvalues ​​χ1 and χ2 obtained from each basic layer as the structural features, and get the structural features of the final left and right distorted images represented as f st .

[0034] Furthermore, the step S2 comprises:

[0035] S201: binocular product saliency map S LR Perform viewport sampling to obtain a significant viewport map Calculate the normalized saliency value W of the salient viewport map S ={W1,W2,...,W M+2} as the significance weight to represent the subject's preference for images from different viewports:

[0036]

[0037] Where p represents a pixel position of the nth (n=1, 2, ..., M+2) viewport image, represents the significant value at position p;

[0038] S202: Using saliency weights to measure the symmetric / asymmetric distortion feature f obtained based on the viewport image corr , detail contrast features of left and right distorted images f clbp And the structural features of the left and right distorted images f st Perform viewport feature aggregation to obtain the distortion metric feature F corr , detail contrast feature F clbp And structural features F st .

[0039] Furthermore, the binocular fusion processing in step S3 includes:

[0040] S301: Viewport image V L,m and V R,m Convert from RGB space to YUV space; perform joint image filtering on the three channels of Y, U, and V respectively, and the filtered left viewport image is recorded as The filtered right viewport image is recorded as where f g represents a joint filter;

[0041] S302: Calculation and The Log-Gabor filter amplitude response is used as the energy factor to fuse the information, The amplitude response is recorded as The amplitude response is recorded as calculate Where ε=0.00001;

[0042] S303: Use E L,m and E R,m right and Weighted fusion image

[0043] S304: Based on Y, U, V single channel, Combine the fusion images of the three channels Y, U, and V to get the final binocular fusion image

[0044] S305: Fusion graph Perform brightness segmentation based on the maximum entropy threshold to obtain the highlighted area Low dark area and the middle brightness area

[0045] Calculate the highlighted area Based on the GGCM matrix, five features including grayscale mean square error, gradient mean square error, grayscale entropy, gradient entropy and mixed entropy are calculated as the texture features of the highlight area f GGCM ;

[0046] Calculate low dark areas using EOG The average EOG of all pixel values, and Perform three downsamplings and take the average EOG at four scales: original image size, downsampling one, downsampling two, and downsampling three as the narrow contour feature f of the low dark area. EOG ;

[0047] The middle brightness area Convert from RGB space to YUV space, calculate the MSCN coefficient based on U and V chromaticity channels, and use AGGD to fit the MSCN coefficient. Take the four parameters after AGGD fitting as the mean δ m , shape parameter θ m , left scale parameter Right scale parameter As chromaticity statistical features; the chromaticity statistical features of the U and V channels are used as the natural statistical features of the intermediate brightness area f AGGD ;

[0048] S306: The texture feature f is divided according to the viewport saliency GGCM , narrow contour feature f EOG and natural statistical characteristics f AGGD Perform feature aggregation to obtain the final feature extracted by the binocular fusion model, which is represented as F fus ={F GGCM ,F EOG ,F AGGD}.

[0049] Furthermore, the binocular difference processing in step S3 includes:

[0050] S307: Viewport image V L,m and V R,m Grayscale is performed respectively, and joint image filtering is performed. The left viewport image after joint image filtering is recorded as The right viewport image after joint image filtering is The original distorted viewport image V L,m and V R,m And the viewport map obtained after joint filtering and Subtract, and the absolute difference map is used as the monocular difference map MD L,m and MD R,m ,Right now

[0051] S308: Calculate V L,m Contrast weighted map of in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; calculate V R,m Contrast weighted map of in Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All obtained by a two-dimensional circularly symmetric Gaussian weighted filter, ε = 0.00001;

[0052] S309: Binocular disparity map BD m Calculated as:

[0053]

[0054] Calculate binocular disparity map BD m The 7-coefficient derivative graph of , select the first-order x-direction g x , first-order y direction g y , second-order x-direction g xx , second-order y direction g yy and second-order g xy These five derivative graphs, where x represents the horizontal direction and y represents the vertical direction;

[0055] S310: Calculate the MSCN coefficients of the five derivative graphs, and use the GGD model to fit the MSCN coefficients, taking the shape of the fitting curves of the five derivative graphs α g and variance As the binocular difference feature, denoted as f dif ;

[0056] S311: The binocular difference features are aggregated according to the viewport saliency and represented as F dif .

[0057] Furthermore, the preset feature selection process in step S4 includes:

[0058] S401: All the extracted features are recorded as:

[0059] F′HSOI ={F ec ,F corr ,F clbp ,F st ,F fus ,F dif};

[0060] The contribution of a single feature vector to each decision tree is calculated using the Gini coefficient of the random forest. The average contribution of all decision trees is used as the contribution value of the feature, which is also the weight of the feature in regression prediction.

[0061] S402: Arrange the contribution values ​​of all feature vectors in descending order, and selectively retain features with high contribution values ​​for feature selection; the feature set after feature selection is recorded as Fs′ HSOI .

[0062] Furthermore, the step S5 comprises:

[0063] S501: feature set Fs′ HSOI As input, random forest is selected as the regression model and the prediction score Q3 is obtained.

[0064] The present invention adopts the above technical solution to achieve the following beneficial effects:

[0065] The present invention provides a reference-free HSOI quality evaluation method that combines a monocular perception model with a binocular perception model from the perspective of binocular vision system perception. The reference-free model does not require the use of reference information of the original image, and has stronger versatility. Based on joint image filtering, a new binocular fusion map and binocular difference map modeling method is designed to achieve a balance between efficiency and accuracy, with higher objective prediction accuracy, and objectively improves the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of a high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception provided by an embodiment of the present invention;

[0067] Figure 2 A structural block diagram of a high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0069] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0070] Example

[0071] This embodiment provides a high dynamic range stereo omnidirectional image quality evaluation method based on visual perception, such as Figure 1 and Figure 2 As shown, the method comprises the steps of:

[0072] S1: Establishing a monocular perception model includes extracting global color features based on a preset feature extraction process; sampling the viewport according to a preset sampling process; performing symmetrical or asymmetrical distortion quantization processing on the sampled viewport image to obtain symmetrical or asymmetrical distortion measurement features; performing scene characteristic perception feature extraction processing on the sampled viewport image to obtain detail contrast features of left and right distorted images and structural features of left and right distorted images;

[0073] S2: feature aggregation of symmetrical or asymmetrical distortion measurement features, detail contrast features of left and right distorted images, and structural features of left and right distorted images;

[0074] S3: Establishing a binocular perception model includes: performing binocular fusion processing on the sampled viewport image to obtain features extracted by the binocular fusion model; performing binocular difference processing on the sampled viewport image to obtain binocular difference features; and performing feature aggregation on the features extracted by the binocular fusion model and the binocular difference features;

[0075] S4: Screening the features aggregated in step S2 and step S3 according to a preset feature selection process to obtain a feature set after feature selection;

[0076] S5: Take the feature set after feature selection as input, select the regression model, and get the prediction score.

[0077] Specifically, for monocular perception, the global color distortion is first measured based on a method combining the antagonistic channel spatial domain and the DCT domain; secondly, in order to measure the symmetric / asymmetric distortion characteristics, the absolute correlation coefficient and correlation distance of the left and right images are calculated as features based on MSR decomposition; then, considering the indoor, outdoor and night scene designs, detail contrast and structural features are extracted based on LP decomposition.

[0078] Specifically, the method includes two parts: a monocular perception model and a binocular perception model. The effectiveness of the method of the present invention is verified based on a HSOI subjective evaluation dataset, namely the NBU-SHOID dataset, which obtains an objective evaluation result that is more consistent with the subjective perception quality.

[0079] Specifically, for the binocular perception model, based on joint image filtering, a method for calculating binocular fusion images and binocular disparity images is proposed. Brightness segmentation is performed based on the binocular fusion image, and texture, narrow contour and chromaticity statistical features are extracted according to the high-brightness, low-darkness and medium-brightness areas. This process conforms to the information transmission mechanism of the binocular visual system V1-V2-V4 area. For the binocular disparity map, its 7-coefficient derivative map is calculated and natural statistical features are extracted. This method has higher objective prediction accuracy.

[0080] Experiments on this method: First, this method is tested on the high dynamic range stereo omnidirectional image subjective dataset NBU-SHOID, and the results obtained by this method are compared with the objective prediction results of the currently available advanced image quality assessment algorithms.

[0081] Specifically, the general strategy of the image quality assessment method is to use the regression prediction model of machine learning to map the feature vector obtained in the early stage into an objective prediction score. This process is obtained by supervised training of subjective MOS. Then the objective prediction score obtained is nonlinearly mapped with the subjective MOS value to ensure that the prediction score is consistent with the MOS value range. Three standard indicators provided by the Video Quality Evaluation Experts Group (VQEG), namely Pearson Linear Correlation Coefficient (PLCC), Spearman Rank Order Correlation Coefficient (SROCC) and Root Mean Square Error (RMSE) are used to measure the correlation between the prediction score and subjective MOS. Among them, PLCC represents the correlation between subjective and objective scores, and SROCC calculates the monotonic correlation index between two ordinal variables. Both values ​​are between [-1,1] and the closer the absolute value is to 1, the higher the accuracy of the regression task is, and the closer the RMSE is to 0, the better.

[0082] Specifically, this method will be compared with seventeen no-reference methods. Including eleven 2D image objective evaluation methods, namely NIQE, IL-NIQE, OG, SISBLIM, dipIQ, BRISQUE, BPRI, BMPRI, ContrastQA, NIQMC, GWH-GLBP. A stereo image quality evaluation method, namely SINQ. Three tone mapping quality evaluation methods, namely BTMQI, BTMIQA and HIGRADE-2. A stereo omnidirectional image quality evaluation method, namely VP-BSOIQA, and a quality evaluation method for high dynamic range omnidirectional images, namely QM-HOI.

[0083] Among them, the results of this method and 17 image quality evaluation methods on the three performance indicators PLCC, SROCC and RMSE of NBU-SHOID are shown in Table 1:

[0084]

[0085]

[0086] Table 1

[0087] Table 1 shows the values ​​of three prediction performance indicators PLCC, SROCC and RMSE of this method and 17 comparison methods on the NBU-SHOID dataset. In Table 1, the objective quality evaluation method of image quality with the best performance is marked. It can be seen that this method has the best performance.

[0088] In the high dynamic range stereo omnidirectional image quality evaluation method based on visual perception, the preset feature extraction process in step S1 includes:

[0089] S101: Adopt I L and I R Represent the distorted left image and the distorted right image respectively, and I L and I R Convert from RGB space to antagonistic color channels κ1-κ2;

[0090] Among them, the preliminary joint calculation of κ1-κ2 is expressed as: and denote the pixel means of κ1 and κ2 channels respectively; and represent the pixel standard deviation of κ1 and κ2 channels respectively;

[0091] S102: Calculate I respectively L and I R of As the spatial color feature F sc ;

[0092] Among them, the spatial color feature is expressed as and Advanced joint statistical measures Right now

[0093] S103: Based on the two channels κ1 and κ2, I L and I R Perform DCT transformation of 5×5 image block size;

[0094] S104: for the transformed DCT coefficient matrix, remove the DC component, divide the AC component into three frequency bands of low frequency, medium frequency and high frequency, calculate the variance of the three frequency bands of each image block as the frequency band energy, and then calculate the mean of all image blocks as the final energy feature of the corresponding frequency band;

[0095] S105: The final energy feature is used as the DCT domain color feature F dc , the global color feature is represented as F ec ={F sc ,F dc}.

[0096] Specifically, DCT is discrete cosine transform.

[0097] The preset sampling process in step S1 includes:

[0098] S106: I L and I R Perform viewport sampling to obtain the viewport image V L,m and V R,m , where m = 1, 2, ..., M + 2; the viewport for viewport sampling is I = {I E ,I B}, where I E represents the equatorial region, I B Represents the polar region, in I E In the example, M viewports are uniformly sampled at equal angle intervals, and the angle is set to 2π / M;

[0099] S107: Select I B The corresponding binocular product saliency map S LR The point with the largest pixel value is taken as the center point of the viewport, and the total number of viewports is M+2;

[0100] S108: Binocular product saliency map S LR The steps to obtain are: calculate I L and I R The left and right plane saliency maps S LR , binocular product saliency map S LR is the correlation measure after disparity matching: S LR (i,j)=S L (i,j)·S R ((i,j)+d i,j ), where d i,j Represents S at pixel (i, j) L and S R The horizontal disparity of the image is obtained by using the optical flow method.

[0101] The symmetrical or asymmetrical distortion quantization processing in step S1 includes:

[0102] S109: Viewport image V L,m and V R,m The MSR decomposition is performed under three Gaussian function scale parameters l, where m = 1, 2, ..., M + 2; l is set to {25, 100, 240}, and the decomposed illumination components are obtained, which are recorded as and and against and The feature map is calculated using LDP texture descriptor;

[0103] S110: Binary encoding of the image pixels after the second-order derivative is performed in four directions {0°, 45°, 90°, 135°} by LDP. The LDP feature map in each direction can be quantized by a ten-dimensional histogram to obtain and The LDP quantized histogram is expressed as and

[0104] S111: Based on and Calculate the absolute correlation coefficient and correlation distance of each ten-dimensional histogram as the similarity features of the left and right images The symmetric or asymmetric distortion metric feature is obtained as f corr ={T,O}.

[0105] Specifically, MSR is multi-scale Retinex; LDP is local derivative mode.

[0106] The perceptual feature extraction process of the scene characteristics in step S1 includes:

[0107] S112: Viewport image V L,m and V R,m Perform LP decomposition at three scales, where m = 1, 2, ..., M + 2; and take the three detail layers and the first two basic layers as the layers after LP decomposition;

[0108] Among them, V L,m The output detail layer set is represented as The output base layer set is represented as V R,m The output detail layer set is represented as The output base layer set is represented as

[0109] S113: Based on detail layer D L,m and D R,m Calculate the LBP map and V respectively L,mand V R,m Contrast map of V L,m The contrast diagram is in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; ε is a constant to prevent the denominator from being zero, set to 0.00001; V R,m The contrast diagram is in, Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All are obtained by a two-dimensional circularly symmetric Gaussian weighted filter;

[0110] S114: Using C L Right D L,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features; using C R Right D R,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features respectively; the detail contrast features of the left and right distorted images are expressed as f clbp ;

[0111] S115: According to the basic layer set B′ L,m and B′ R,m Calculate the global structure tensor, perform two-dimensional structure tensor transformation respectively, perform matrix decomposition on the output structure tensor matrix, take the two eigenvalues ​​χ1 and χ2 obtained from each basic layer as the structural features, and get the structural features of the final left and right distorted images represented as f st .

[0112] Specifically, LP is a Laplacian pyramid, and LBP is a local binary pattern map.

[0113] Wherein, step S2 comprises:

[0114] S201: binocular product saliency map S LR Perform viewport sampling to obtain a significant viewport map Calculate the normalized saliency value W of the salient viewport map S ={W1,W2,...,W M+2} as the significance weight to represent the subject's preference for images from different viewports:

[0115]

[0116] Where p represents a pixel position of the nth (n=1, 2, ..., M+2) viewport image, represents the significant value at position p;

[0117] S202: Using saliency weights to measure the symmetric / asymmetric distortion feature f obtained based on the viewport image corr , detail contrast features of left and right distorted images f clbp And the structural features of the left and right distorted images f st Perform viewport feature aggregation to obtain the distortion metric feature F corr , detail contrast feature F clbp And structural features F st .

[0118] The binocular fusion processing in step S3 includes:

[0119] S301: Viewport image V L,m and V R,m Convert from RGB space to YUV space; perform joint image filtering on the three channels of Y, U, and V respectively, and the filtered left viewport image is recorded as The filtered right viewport image is recorded as where f g represents a joint filter;

[0120] S302: Calculation and The Log-Gabor filter amplitude response is used as the energy factor to fuse the information, The amplitude response is recorded as The amplitude response is recorded as calculate Where ε=0.00001;

[0121] S303: Use E L,m and E R,m right and Weighted fusion image

[0122] S304: Based on Y, U, V single channel, Combine the fusion images of the three channels Y, U, and V to get the final binocular fusion image

[0123] S305: Fusion graph Perform brightness segmentation based on the maximum entropy threshold to obtain the highlighted area Low dark area and the middle brightness area

[0124] Calculate the highlighted area Based on the GGCM matrix, five features including grayscale mean square error, gradient mean square error, grayscale entropy, gradient entropy and mixed entropy are calculated as the texture features of the highlight area f GGCM ;

[0125] Calculate low dark areas using EOG The average EOG of all pixel values, and Perform three downsamplings and take the average EOG at four scales: original image size, downsampling one, downsampling two, and downsampling three as the narrow contour feature f of the low dark area. EOG ;

[0126] The middle brightness area Convert from RGB space to YUV space, calculate the MSCN coefficient based on U and V chromaticity channels, and use AGGD to fit the MSCN coefficient. Take the four parameters after AGGD fitting as the mean δ m , shape parameter θ m , left scale parameter Right scale parameter As chromaticity statistical features; the chromaticity statistical features of the U and V channels are used as the natural statistical features of the intermediate brightness area f AGGD ;

[0127] S306: The texture feature f is divided according to the viewport saliency GGCM , narrow contour feature f EOG and natural statistical characteristics f AGGD Perform feature aggregation to obtain the final feature extracted by the binocular fusion model, which is represented as F fus ={F GGCM ,F EOG ,F AGGD}.

[0128] Specifically, GGCM is the grayscale gradient co-occurrence matrix; EOG is the energy gradient function; MSCN is the de-mean contrast normalization coefficient; AGGD is the asymmetric generalized Gaussian distribution.

[0129] The binocular difference processing in step S3 includes:

[0130] S307: Viewport image V L,m and V R,m Grayscale is performed respectively, and joint image filtering is performed. The left viewport image after joint image filtering is recorded as The right viewport image after joint image filtering is The original distorted viewport image V L,m and V R,mAnd the viewport map obtained after joint filtering and Subtract, and the absolute difference map is used as the monocular difference map MD L,m and MD R,m ,Right now

[0131] S308: Calculate V L,m Contrast weighted map of in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; calculate V R,m Contrast weighted map of in Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All obtained by a two-dimensional circularly symmetric Gaussian weighted filter, ε = 0.00001;

[0132] S309: Binocular disparity map BD m Calculated as:

[0133]

[0134] Calculate binocular disparity map BD m The 7-coefficient derivative graph of , select the first-order x-direction g x , first-order y direction g y , second-order x-direction g xx , second-order y direction g yy and second-order g xy These five derivative graphs, where x represents the horizontal direction and y represents the vertical direction;

[0135] S310: Calculate the MSCN coefficients of the five derivative graphs, and use the GGD model to fit the MSCN coefficients, taking the shape of the fitting curves of the five derivative graphs α g and variance As the binocular difference feature, denoted as f dif ;

[0136] S311: The binocular difference features are aggregated according to the viewport saliency and represented as F dif .

[0137] Specifically, GGD is a generalized Gaussian distribution.

[0138] The preset feature selection process in step S4 includes:

[0139] S401: All the extracted features are recorded as:

[0140] F′ HSOI ={F ec ,F corr ,F clbp ,F st ,F fus ,F dif};

[0141] The contribution of a single feature vector to each decision tree is calculated using the Gini coefficient of the random forest. The average contribution of all decision trees is used as the contribution value of the feature, which is also the weight of the feature in regression prediction.

[0142] S402: Arrange the contribution values ​​of all feature vectors in descending order, and selectively retain features with high contribution values ​​for feature selection; the feature set after feature selection is recorded as Fs′ HSOI .

[0143] Wherein, step S5 comprises:

[0144] S501: feature set Fs′ HSOI As input, random forest is selected as the regression model and the prediction score Q3 is obtained.

[0145] This method The present invention provides a reference-free HSOI quality evaluation method that combines a monocular perception model with a binocular perception model from the perspective of binocular vision system perception. For monocular perception, firstly, the global color distortion is measured based on a method combining the antagonistic channel spatial domain and the DCT domain; secondly, in order to measure the symmetric / asymmetric distortion characteristics, the absolute correlation coefficient and correlation distance of the left and right images are calculated as features based on the MSR decomposition; then, considering the indoor, outdoor and night scene designs, the detail contrast and structural features are extracted based on the LP decomposition. For the binocular perception model, a binocular fusion image and a binocular difference image calculation method are proposed based on joint image filtering. Brightness segmentation is performed based on the binocular fusion image, and texture, narrow contour and chromaticity statistical features are extracted according to the high-brightness, low-darkness and medium-brightness areas, respectively. This process conforms to the information transmission mechanism of the V1-V2-V4 area of ​​the binocular vision system.

[0146] In order to verify the effectiveness of the proposed method, 17 methods with different evaluation objects were selected to conduct comparative experiments on the NBU-SHOID dataset. The results show that the proposed method can be regarded as an effective quality evaluator for HSOI.

[0147] Although the disclosure is disclosed as above, the protection scope of the disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the disclosure, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A high dynamic range stereoscopic omnidirectional image quality evaluation method based on visual perception, characterized in that: Includes steps: S1: Establishing a monocular perception model includes extracting global color features based on a preset feature extraction process; sampling the viewport according to a preset sampling process; Performing symmetrical or asymmetrical distortion quantization processing on the sampled viewport image to obtain symmetrical or asymmetrical distortion measurement features; performing perceptual feature extraction processing on the sampled viewport image of scene characteristics to obtain detail contrast features of left and right distorted images and structural features of left and right distorted images; S2: feature aggregation of symmetrical or asymmetrical distortion measurement features, detail contrast features of left and right distorted images, and structural features of left and right distorted images; S3: Establishing a binocular perception model includes performing binocular fusion processing on the sampled viewport images to obtain features extracted by the binocular fusion model; The sampled viewport image is subjected to binocular difference processing to obtain binocular difference features; and the features extracted by the binocular fusion model and the binocular difference features are subjected to feature aggregation; S4: Screening the features aggregated in step S2 and step S3 according to a preset feature selection process to obtain a feature set after feature selection; S5: Take the feature set after feature selection as input, select the regression model, and get the prediction score.

2. The high dynamic range stereoscopic omnidirectional image quality evaluation method based on visual perception according to claim 1, characterized in that: The preset feature extraction process in step S1 includes: S101: Adopt I L and I R Represent the distorted left image and the distorted right image respectively, and I L and I R Convert from RGB space to antagonistic color channels κ1-κ2; Among them, the preliminary joint calculation of κ1-κ2 is expressed as: and denote the pixel means of κ1 and κ2 channels respectively; and represent the pixel standard deviation of κ1 and κ2 channels respectively; S102: Calculate I respectively L and I R of As the spatial color feature F sc ; Among them, the spatial color feature is expressed as and Advanced joint statistical measures Right now S103: Based on the two channels κ1 and κ2, I L and I R Perform DCT transformation of 5×5 image block size; S104: for the transformed DCT coefficient matrix, remove the DC component, divide the AC component into three frequency bands of low frequency, medium frequency and high frequency, calculate the variance of the three frequency bands of each image block as the frequency band energy, and then calculate the mean of all image blocks as the final energy feature of the corresponding frequency band; S105: The final energy feature is used as the DCT domain color feature F dc , the global color feature is represented as F ec ={F sc ,F dc }.

3. The high dynamic range stereoscopic omnidirectional image quality evaluation method based on visual perception according to claim 2, characterized in that: The preset sampling process in step S1 includes: S106: I L and I R Perform viewport sampling to obtain the viewport image V L,m and V R,m , where m = 1, 2, ..., M + 2; the viewport for viewport sampling is I = {I E ,I B }, where I E represents the equatorial region, I B Represents the polar region, in I E In the example, M viewports are uniformly sampled at equal angle intervals, and the angle is set to 2π / M; S107: Select I B The corresponding binocular product saliency map S LR The point with the largest pixel value is taken as the center point of the viewport, and the total number of viewports is M+2; S108: Binocular product saliency map S LR The steps to obtain are: calculate I L and I R The left and right plane saliency maps S L , S R , binocular product saliency map S LR is the correlation measure after disparity matching: S LR (i,j)=S L (i,j)·S R ((i,j)+d i,j ), where d i,j represents S at pixel (i, j) L and S R The horizontal disparity of the image is obtained by using the optical flow method.

4. The high dynamic range stereoscopic omnidirectional image quality evaluation method based on visual perception according to claim 3, characterized in that: The symmetrical or asymmetrical distortion quantization process in step S1 includes: S109: Viewport image V L,m and V R,m The MSR decomposition is performed under three Gaussian function scale parameters l, where m = 1, 2, ..., M + 2; l is set to {25, 100, 240}, and the decomposed illumination components are obtained, which are recorded as and against and The feature map is calculated using LDP texture descriptor; S110: Binary encoding of the image pixels after the second-order derivative is performed in four directions {0°, 45°, 90°, 135°} by LDP. The LDP feature map in each direction can be quantized by a ten-dimensional histogram to obtain and The LDP quantized histogram is expressed as and S111: Based on and Calculate the absolute correlation coefficient and correlation distance of each ten-dimensional histogram as the similarity features of the left and right images The symmetric or asymmetric distortion metric feature is obtained as f corr ={T,O}.

5. The high dynamic range stereoscopic omnidirectional image quality evaluation method based on visual perception according to claim 4, characterized in that: The perceptual feature extraction process of the scene characteristics in step S1 includes: S112: Viewport image V L,m and V R,m Perform LP decomposition at three scales, where m = 1, 2, ..., M + 2; and take the three detail layers and the first two basic layers as the layers after LP decomposition; Among them, V L,m The output detail layer set is represented as The output base layer set is represented as V R,m The output detail layer set is represented as The output base layer set is represented as S113: Based on detail layer D L,m and D R,m Calculate the LBP map and V respectively L,m and V R,m Contrast map of V L,m The contrast diagram is in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; ε is a constant to prevent the denominator from being zero, set to 0.00001; V R,m The contrast diagram is in, Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; S114: Using C L Right D L,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features; using C R Right D R,m The LBP mapping images are weighted and quantized into ten-dimensional histogram features respectively; the detail contrast features of the left and right distorted images are expressed as f clbp ; S115: According to the basic layer set B L ' ,m and B′ R,m Calculate the global structure tensor, perform two-dimensional structure tensor transformation respectively, perform matrix decomposition on the output structure tensor matrix, take the two eigenvalues ​​χ1 and χ2 obtained from each basic layer as the structural features, and get the structural features of the final left and right distorted images represented as f st .

6. The high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception according to claim 5, characterized in that: The step S2 comprises: S201: binocular product saliency map S LR Perform viewport sampling to obtain a significant viewport map Calculate the normalized saliency value W of the salient viewport map S ={W1,W2,...,W M+2 } as the significance weight to represent the subject's preference for images from different viewports: Where p represents a pixel position of the nth viewport image, where n = 1, 2, ..., M + 2, represents the significant value at position p; S202: Using saliency weights to measure the symmetric / asymmetric distortion feature f obtained based on the viewport image corr , detail contrast features of left and right distorted images f clbp And the structural features of the left and right distorted images f st Perform viewport feature aggregation to obtain the distortion metric feature F corr , detail contrast feature F clbp And structural features F st .

7. The high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception according to claim 6, characterized in that: The binocular fusion processing in step S3 includes: S301: Viewport image V L,m and V R,m Convert from RGB space to YUV space; perform joint image filtering on the three channels of Y, U, and V respectively, and the filtered left viewport image is recorded as The filtered right viewport image is recorded as where f g represents a joint filter; S302: Calculation and The Log-Gabor filter amplitude response is used as the energy factor to fuse the information, The amplitude response is recorded as The amplitude response is recorded as calculate Where ε=0.00001; S303: Use E L,m and E R,m right and Weighted fusion image S304: Based on Y, U, V single channel, Combine the fusion images of the three channels Y, U, and V to get the final binocular fusion image S305: Fusion graph Perform brightness segmentation based on the maximum entropy threshold to obtain the highlighted area Low dark area and the middle brightness area Calculate the highlighted area Based on the GGCM matrix, five features including grayscale mean square error, gradient mean square error, grayscale entropy, gradient entropy and mixed entropy are calculated as the texture features of the highlight area f GGCM ; Calculate low dark areas using EOG The average EOG of all pixel values ​​and Perform three downsamplings and take the average EOG at four scales: original image size, downsampling one, downsampling two, and downsampling three as the narrow contour feature f of the low dark area. EOG ; The middle brightness area Convert from RGB space to YUV space, calculate the MSCN coefficient based on U and V chromaticity channels, and use AGGD to fit the MSCN coefficient. Take the four parameters after AGGD fitting as the mean δ m , shape parameter θ m , left scale parameter Right scale parameter As chromaticity statistical features; the chromaticity statistical features of the U and V channels are used as the natural statistical features of the intermediate brightness area f AGGD ; S306: The texture feature f is divided according to the viewport saliency GGCM , narrow contour feature f EOG and natural statistical characteristics f AGGD Perform feature aggregation to obtain the final feature extracted by the binocular fusion model, which is represented as F fus ={F GGCM ,F EOG ,F AGGD }.

8. The high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception according to claim 7, characterized in that: The binocular difference processing in step S3 includes: S307: Viewport image V L,m and V R,m Grayscale is performed respectively, and joint image filtering is performed. The left viewport image after joint image filtering is recorded as The right viewport image after joint image filtering is The original distorted viewport image V L,m and V R,m And the viewport map obtained after joint filtering and Subtract, and the absolute difference map is used as the monocular difference map MD L,m and MD R,m ,Right now S308: Calculate V L,m Contrast weighted map of in Indicates V L,m The standard deviation plot of is the corresponding mean plot, and are obtained by a two-dimensional circularly symmetric Gaussian weighted filter; calculate V R,m Contrast weighted map of in Indicates V R,m The standard deviation plot of is the corresponding mean plot, and All obtained by a two-dimensional circularly symmetric Gaussian weighted filter, ε = 0.00001; S309: Binocular disparity map BD m Calculated as: Calculate binocular disparity map BD m The 7-coefficient derivative graph of , select the first-order x-direction g x , first-order y direction g y , second-order x-direction g xx , second-order y direction g yy and second-order g xy These five derivative graphs, where x represents the horizontal direction and y represents the vertical direction; S310: Calculate the MSCN coefficients of the five derivative graphs, and use the GGD model to fit the MSCN coefficients, taking the shape of the fitting curves of the five derivative graphs α g and variance As the binocular difference feature, denoted as f dif ; S311: The binocular difference features are aggregated according to the viewport saliency and represented as F dif .

9. The high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception according to claim 8, characterized in that: The preset feature selection process in step S4 includes: S401: All the extracted features are recorded as: F H ′ SOI ={F ec ,F corr ,F clbp ,F st ,F fus ,F dif }; The contribution of a single feature vector to each decision tree is calculated using the Gini coefficient of the random forest. The average contribution of all decision trees is used as the contribution value of the feature, which is also the weight of the feature in regression prediction. S402: Arrange the contribution values ​​of all feature vectors in descending order, and selectively retain features with high contribution values ​​for feature selection; the feature set after feature selection is recorded as Fs′ HSOI .

10. The high dynamic range stereoscopic omnidirectional image quality assessment method based on visual perception according to claim 9, characterized in that: The step S5 comprises: S501: Feature set Fs′ HSOI As input, random forest is selected as the regression model and the prediction score Q3 is obtained.

Images