Stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching

Abstract

The invention discloses a stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching, and belongs to the technical field of stereo image quality evaluation. The method comprises the following steps: segmenting a left view and a right view of an original three-dimensional image into non-overlapped image blocks as input samples of a network; the method comprises the following steps: converting left and right view image blocks into a frequency domain by using FFT and decomposing according to high and low frequencies, reducing by using IFFT to obtain left and right view high-frequency and low-frequency image blocks, and extracting primary features of the left and right view high-frequency and low-frequency image blocks by using a group of convolution and pooling operations; the high-frequency and low-frequency primary features are interactively enhanced through a double-frequency interaction attention module and then are sent to a double-frequency recombination module to obtain a high-frequency and low-frequency weighted feature map; meanwhile, left and right view image blocks are sent to a group of convolution and pooling to extract full-frequency primary features, and feature enhancement is performed by using high and low frequency information; binocular progressive registration and competitive selection are carried out on the enhanced full-frequency features through a binocular matching fusion module to obtain binocular fusion features; the binocular feature fusion features are further extracted through two groups of convolution, and after the features are spliced with left and right view high and low frequency recombination features, an objective prediction score is obtained through a full connection layer. According to the objective evaluation method, a binocular fusion and competition mechanism of a human vision system can be well simulated, and the accuracy of the objective evaluation method is improved by utilizing the influence of different spatial frequency information on quality perception.

Description

Technical Field

[0001] The present invention relates to the technical field of stereoscopic image quality evaluation, and in particular to a stereoscopic image quality evaluation method based on dual-frequency interactive enhancement and binocular matching. Background Art

[0002] As people's lifestyles and entertainment methods become increasingly diverse, they are no longer satisfied with the flat image experience, but are pursuing more immersive and realistic stereoscopic images. Compared with flat (2D) images, stereoscopic images (3D) can make viewers feel the sense of three-dimensionality while displaying the image content, creating an immersive atmosphere. Therefore, they are widely used in 3D movies, 3D games, virtual reality and other fields. However, during the acquisition, compression and transmission of stereoscopic images, the images will be distorted due to some unavoidable factors, thus affecting the viewer's visual experience. Therefore, it is particularly important to build a stereoscopic image quality perception model that is highly consistent with human evaluation criteria. In addition, stereoscopic image quality assessment (SIQA) can also be used as a part of the image processing system to provide optimization guidance for stereoscopic image super-resolution, segmentation and other fields.

[0003] Generally speaking, IQA methods can be divided into two categories, namely subjective IQA methods and objective IQA methods. Subjective IQA methods are to evaluate and score the quality of images based on the subjective feelings of human observers and various indicators of the images under certain experimental conditions. Since the final recipients of the images are humans, the subjective evaluation method based on the opinion scores given by human observers can more realistically and reliably evaluate the quality of the images, and this method has a low technical threshold and is considered to be the most effective method. Although the subjective evaluation method is the most accurate in evaluating image quality, it has many shortcomings. In practice, the results of subjective evaluation methods are easily affected by factors such as the psychological state, cognitive level, and environment of the tester. Therefore, the application scenarios of subjective evaluation in real life are very small. The objective image quality evaluation method evaluates the quality of images by simulating the human visual system (HVS) and establishing an evaluation model, which can well solve the limitations of subjective evaluation and is more in line with actual needs. The objective evaluation method is divided into full reference (FR), reduced reference (RR), and no reference (NR) according to the degree of participation of the reference image. Since ideal reference images cannot be obtained in many application scenarios, objective image quality evaluation methods without reference that do not require ideal reference images are more widely used.

[0004] At present, in the field of SIQA, researchers' work is mostly focused on the study of NR-type stereo image quality evaluation methods, and many NR-SIQA methods have been proposed. The early FR-SIQA method was improved based on the classic FR-2DIQA method. For example, Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM). However, this type of method cannot reflect the characteristics of human visual perception, nor does it consider the influence of binocular parallax, and cannot be directly applied to stereo image quality evaluation. In addition, there are some NR-SIQA methods based on traditional artificial feature extraction of image features based on human visual system (HVS) or natural scene statistics (NSS). This type of method has a high design threshold, requiring designers to have a high level and rich experience, and has poor generalization ability, making it difficult to adapt to the actual situation with diverse distortion changes.

[0005] In recent years, driven by deep learning technology, many stereoscopic image binocular vision models based on convolutional neural networks (CNNs) have emerged, further promoting the development of NR-SIQA. However, existing CNN-based SIQA methods do not pay attention to the impact of image spatial frequency on human perception of image quality. Most methods do not take into account the mutual influence of high- and low-frequency information and the different contributions of high and low frequencies to image quality perception. According to biological knowledge and human visual psychology, the human visual system performs differently in processing high and low spatial frequency information of the same image. In fact, extracting features from the entire frequency range may encounter the problem of uneven information distribution in the spatial domain. In fact, visual information processing first extracts the overall contour features of the visual scene based on low spatial frequency information, and then uses this information to promote the extraction of detailed texture features corresponding to high-frequency information. On the other hand, strong edges tend to attract more attention and can enhance the perception of surrounding smooth areas. These mechanisms indicate a complex interaction between high- and low-frequency information.

[0006] In addition, since the evaluation object of SIQA is a stereo image, binocular visual features are usually considered. However, most methods simply concatenate the left and right features and then perform quality regression, but there is no interaction and fusion process between the left and right views. Although some methods use the difference and summation of the left and right view features or the twin network to achieve the interaction between the left and right views, they obviously cannot simulate the relatively complex process of binocular interaction and fusion of the human visual system. According to biological knowledge and human visual psychology knowledge, the human visual system usually first fuses the matching features in the left and right views, and then uses a competition mechanism to select and suppress the mismatching features. The matching features between the left and right views are the key factors for stabilizing binocular perception. The alignment of binocular features can promote the integration of binocular information and form a unified and coherent visual perception. By simulating this process, stable and accurate binocular visual features can be effectively extracted.

[0007] Based on the above content, the present invention proposes a stereo image quality evaluation method based on dual-frequency interactive enhancement and binocular matching. Summary of the invention

[0008] The purpose of the present invention is to propose a stereoscopic image quality evaluation method based on dual-frequency interactive enhancement and binocular matching to solve the problem that the existing technology does not take into account the different high and low frequency sensitivities of the human eye to pictures with different distortion types and different semantic information when evaluating the quality of stereoscopic images.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The stereo image quality evaluation method based on dual-frequency interactive enhancement and binocular matching includes the following steps:

[0011] Step 1: Cut the left and right view original images into 32×32 non-overlapping image blocks, use FFT to convert the left and right view image blocks to the frequency domain and decompose them into high and low frequencies, and use IFFT to restore the high-frequency and low-frequency image blocks of the left and right views;

[0012] Step 2: Training a convolutional neural network based on dual-frequency interaction enhancement and binocular matching: The convolutional neural network includes three main sub-networks: a left view dual-frequency interaction feature enhancement sub-network, a right view dual-frequency interaction feature enhancement sub-network, and a binocular matching fusion sub-network, specifically including the following contents:

[0013] Since the left and right view sub-networks are symmetrical, only the implementation of the left view dual-frequency interaction feature enhancement sub-network is introduced here. In the left view dual-frequency interaction feature enhancement sub-network, a set of convolution and pooling operations are used to extract the primary feature map of the high-frequency and low-frequency image blocks of the left view, and the dual-frequency interaction attention module is used to calculate the attention weights of the high-frequency and low-frequency signals, and the high- and low-frequency features are interactively weighted, and then three sets of convolutions are used to further extract features, and finally a dual-frequency reorganization module is used to obtain the high- and low-frequency weighted feature map;

[0014] In the binocular matching fusion subnetwork, a set of convolution and pooling operations are used to extract the full-frequency primary feature map of the image blocks of the left and right views. The high- and low-frequency information of the dual-frequency interactive attention module is used to enhance the full-frequency primary feature map to obtain the full-frequency enhanced feature map. Then, the enhanced full-frequency features of the left and right views are sequentially subjected to low-frequency binocular registration and high-frequency binocular registration. Finally, the binocular competition selection module is used to perform feature screening in the spatial dimension and channel dimension to obtain the binocular fusion feature map, and the features are further extracted through two sets of convolutions.

[0015] The binocular fusion feature map is concatenated with the high- and low-frequency weighted feature maps of the left and right views and then passed through a fully connected layer to obtain the objective prediction score. During the training process, the loss function of the network model uses L1 loss to measure the difference between the image quality score predicted by the network and the actual image quality score, and guides the optimization of the model.

[0016] Step 3: Use the trained network to evaluate image quality. Slice the stereo image to be evaluated and input it into the network. Average the quality scores of all image blocks of each stereo image output by the network to obtain the quality score of the entire image.

[0017] Preferably, the processing flow of the dual-frequency interactive attention module in step 2 specifically includes the following contents:

[0018] The low-frequency primary features are passed through three convolution layers with a convolution kernel of 1×1 in parallel to obtain three feature maps Q, K and V. The features Q and K are transformed in dimension and then matrix multiplication is performed, and then the low-frequency attention map is obtained through the SoftMax layer. Similarly, the same operation is performed on the high-frequency features to obtain the high-frequency attention map; then, the low-frequency feature map V is matrix multiplied with the high-frequency attention map to obtain the low-frequency feature map after high-frequency information enhancement. The same operation is performed on the high-frequency primary features to obtain the high-frequency feature map after low-frequency information enhancement.

[0019] Preferably, the processing flow of the dual-frequency reorganization module in step 2 specifically includes the following contents:

[0020] The low-frequency and high-frequency high-level features are differentiated and summed, and the difference and sum are cascaded and sent to the residual module ResBlock composed of two convolutional layers and one residual connection to further extract features. The features are then passed through a global average pooling layer GAP and a fully connected layer to obtain adaptive weights. Finally, the weights are normalized and the low-frequency and high-frequency high-level features are weighted to obtain high- and low-frequency weighted features.

[0021] Preferably, the binocular matching fusion module in step 2 is composed of a binocular progressive registration module and a binocular competition selection module, and specifically includes the following contents:

[0022] ① The binocular progressive registration module consists of a low-frequency binocular registration unit and a high-frequency binocular registration unit;

[0023] In the low-frequency binocular registration unit, the low-frequency primary features of the left and right views are cascaded with their differential features and sent to a convolution layer with a convolution kernel size of 1×1 to obtain the low-frequency offset. The full-frequency primary features of the left and right views and the low-frequency offset are sent to the deformable convolution to obtain the features of the left and right views registered at the low-frequency scale.

[0024] In the high-frequency binocular registration unit, the high-frequency primary features of the left and right views are cascaded with their differential features and sent to a convolution layer with a convolution kernel size of 1×1 to obtain the high-frequency offset. The features of the left and right views after registration at the low-frequency scale and the high-frequency offset are sent to the deformable convolution to obtain the features of the left and right views registered at both the high and low frequency scales.

[0025] ② The binocular competitive selection module consists of a spatial dimension selection block and a channel dimension selection block. The registration features obtained by the binocular progressive registration module are added to the full-frequency primary features as the input of the module;

[0026] In the spatial dimension selection block, the input left and right view features are first passed through a convolution layer with a convolution kernel size of 1×1 to obtain two sets of feature maps. Then the two sets of feature maps are dimensionally transformed and matrix multiplied. Then they are passed through the SoftMax layer to obtain the disparity attention map. The disparity attention map is matrix multiplied with the left and right view features respectively and weighted in the spatial dimension.

[0027] In the channel dimension selection block, the weighted features in the spatial dimension are sent to the global average pooling layer to obtain the left and right view channel attention map, and then the channel attention map is normalized and point-multiplied with the weighted features of the left and right views in the spatial dimension respectively. Finally, the weighted left and right view features in the channel dimension are added to obtain the binocular fusion feature.

[0028] Compared with the prior art, the present invention provides a stereo image quality evaluation method based on dual-frequency interactive enhancement and binocular matching, which has the following beneficial effects:

[0029] The present invention makes full use of the information of different spatial frequencies of the image to enhance the features, simulating the mechanism of different frequency sensitivity of the quality perception of the human visual system; by simulating the binocular fusion and binocular competition mechanism of the human visual system, and using the information of different spatial frequencies of the image to assist binocular fusion, it can effectively extract stereoscopic visual features that meet the quality perception of human vision. The evaluation results of this method are highly consistent with the subjective evaluation results of the human eye, and have important value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching proposed in the present invention, wherein LFBRU represents a low-frequency binocular progressive registration unit, HFBRU represents a high-frequency binocular progressive registration unit, BRS represents a binocular competitive selection module, CCP represents two convolutional layers superimposed on a pooling layer, and nCCP represents n cascaded CCP modules;

[0031]

[0032] Figure 2 It is an overall block diagram of the binocular progressive registration module in the binocular feature matching and fusion module mentioned in Example 1 of the present invention;

[0033] Figure 3 It is an overall block diagram of the binocular competition selection module in the binocular feature matching and fusion module mentioned in Example 1 of the present invention; DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0035] Example 1

[0036] This embodiment proposes a stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching. Figure 1 As shown, the method comprises the following steps:

[0037] S101: Preprocessing and data augmentation of original stereo images for training

[0038] The original images of the left and right views are cut into 220 32×32 non-overlapping image blocks, and the quality score of each image block is the true quality score of the original image. The left and right view image blocks are converted to the frequency domain using FFT and decomposed into high and low frequencies, and the high-frequency and low-frequency image blocks of the left and right views are restored using IFFT;

[0039] S102: Training a convolutional neural network based on dual-frequency interaction enhancement and binocular matching

[0040] A convolutional neural network based on dual-frequency interaction enhancement and binocular matching is constructed. The network consists of three sub-networks: left view dual-frequency interaction feature enhancement sub-network, right view dual-frequency interaction feature enhancement sub-network and binocular matching fusion sub-network. Specifically, it includes the following contents:

[0041] Since the left and right view sub-networks are symmetrical, only the implementation of the left view dual-frequency interactive feature enhancement sub-network is introduced here. In the left view dual-frequency interactive feature enhancement sub-network, a set of convolutions with a kernel size of 3×3 and pooling operations are used to extract primary feature maps for the high-frequency and low-frequency image blocks of the left view, and the dual-frequency interactive attention module is used to calculate the attention weights of the high-frequency and low-frequency signals, and the high- and low-frequency features are interactively weighted, and then three sets of convolutions with a kernel size of 3×3 are used to further extract features, and finally a dual-frequency reorganization module is used to obtain the high- and low-frequency weighted feature maps;

[0042] In the binocular matching fusion subnetwork, a set of convolution and pooling operations are used to extract the full-frequency primary feature map of the image blocks of the left and right views. The high- and low-frequency information of the dual-frequency interactive attention module is used to enhance the full-frequency primary feature map to obtain the full-frequency enhanced feature map. Then, the enhanced full-frequency features of the left and right views are sequentially subjected to low-frequency binocular registration and high-frequency binocular registration. Finally, the binocular competition selection module is used to perform feature screening in the spatial dimension and channel dimension to obtain the binocular fusion feature map, and the features are further extracted through two sets of convolutions.

[0043] The binocular fusion feature map is concatenated with the high- and low-frequency weighted feature maps of the left and right views and then passed through a fully connected layer to obtain the objective prediction score. During the training process, the loss function of the network model uses L1 loss to measure the difference between the image quality score predicted by the network and the actual image quality score, and guides the optimization of the model.

[0044] S103: Use the trained network to evaluate stereo image quality

[0045] The stereo image to be evaluated is sliced ​​and input into the network. The quality scores of all image blocks of each stereo image output by the network are averaged to obtain the quality score of the entire image.

[0046] Embodiment 2:

[0047] Based on Example 1, but different in that, the solution in Example 1 is further introduced below in combination with specific calculation formulas and example data. Since the left and right view sub-networks are symmetrical, only the implementation of one sub-network is introduced here. See the following description for details:

[0048] Step 1: Collect EEG signals and preprocess them

[0049] The original images of the left and right views are cut into 220 non-overlapping image blocks of 32×32 size, and the quality score of each image block is the true quality score of the original image. Use FFT to convert the image block I to the frequency domain according to the following formula (1) to perform high- and low-frequency decomposition, and use IFFT to restore the high-frequency and low-frequency image blocks:

[0050]

[0051] Where z represents the frequency component of the sample; t(·; r) represents a threshold function that separates the low-frequency and high-frequency components from z according to the hyperparameter radius r.

[0052] To simplify the process, we first consider decomposing a single-channel image into high-frequency and low-frequency. For the three-channel image used in this paper, this operation is performed independently on each channel. Define a single-channel image as I∈N n×n , which is represented in the frequency domain as z∈C n×n , where C represents a complex number. Function z lf ,z hf =t(z;r) is defined as:

[0053]

[0054] Among them, z(i,j) represents the value of position (i,j), {c i ,c j} represents the centroid, and d(·,·) is the Euclidean distance.

[0055] Step 2: Train a convolutional neural network based on dual-frequency interaction enhancement and binocular matching

[0056] This embodiment constructs a convolutional neural network based on dual-frequency interaction enhancement and binocular matching. The network consists of three sub-networks: a left view dual-frequency interaction feature enhancement sub-network, a right view dual-frequency interaction feature enhancement sub-network, and a binocular matching fusion sub-network. The specific method is as follows:

[0057] Since the left and right view sub-networks are symmetrical, only the implementation of the left view dual-frequency interaction feature enhancement sub-network is introduced here.

[0058] In the left view dual frequency interaction feature enhancement sub-network, high-frequency and low-frequency image blocks are fed into a set of convolutions and pooling with a convolution kernel size of 3×3 to extract primary features. The dual frequency interaction attention module DFIA is used to calculate the attention weights of high-frequency and low-frequency signals, and the high and low frequency features are interactively weighted. Then, three sets of convolutions with a convolution kernel size of 3×3 are used to further extract features. Finally, a dual frequency reorganization module DFR is used to obtain the high and low frequency weighted feature map.

[0059] Dual frequency interactive attention module DFIA Figure 1 As shown, the primary features are first After three convolution layers with 1×1 convolution kernels, two sets of feature maps are obtained. and Then the dimensions of the two sets of feature maps K and Q are transformed into Where N = H × W. After that, matrix multiplication is performed on the transpose of Q and K, and a SoftMax layer is applied to calculate the spatial attention map, resulting in {Q lf ,K lf} and {Q hf ,K hf}The calculated spatial attention map Att lf With Att hf Finally, the spatial attention map Att lf (Att hf ) and feature map V hf (V lf ) to perform matrix multiplication to achieve interactive weighting of high and low frequency features. The whole process can be expressed as:

[0060]

[0061] in, Indicates matrix multiplication, {IF hf ,IF lf} are the high-frequency and low-frequency features after interactive enhancement.

[0062] Dual frequency reconfiguration module DFR Figure 1 As shown in Figure 1, it consists of a ResBlock, a GAP layer, and a fully connected layer in cascade. ResBlock contains two convolutional layers and a residual. The difference and sum of the interactively weighted low-frequency and high-frequency high-level features are cascaded and sent to the DFR module to obtain the weight W hf With W lf .

[0063] Finally, the obtained weights are normalized and the low-frequency and high-frequency high-level features are weighted. The feature weighting is expressed by the following formula:

[0064]

[0065] in, is the input feature of DFR, is the final single view frequency reorganization feature.

[0066] In the binocular matching fusion subnetwork, a set of convolution and pooling operations are used to extract the full-frequency primary feature maps of the image patches of the left and right views. In order to highlight the contours and details for better registration, the high- and low-frequency information of the dual-frequency interactive attention module is used to enhance the full-frequency primary feature map, that is, using Slf With S hf Respectively with V ff Perform matrix multiplication to obtain the full frequency enhancement feature map. The specific implementation process is expressed by the following formula:

[0067]

[0068] Then, the enhanced full-frequency features of the left and right views are sequentially subjected to low-frequency binocular registration and high-frequency binocular registration. Finally, the binocular competition selection module is used to screen features in the spatial dimension and channel dimension to obtain a binocular fusion feature map, and further extract features through two sets of convolutions. The binocular feature matching fusion module consists of a binocular progressive registration module and a binocular competition selection module. The specific implementation process is as follows:

[0069] ① Such as Figure 2 As shown, the binocular progressive registration module is composed of a low-frequency binocular registration unit and a high-frequency binocular registration unit;

[0070] In the low-frequency binocular registration unit, the low-frequency primary features of the left and right views are cascaded with their differential features, and then sent to a convolution layer with a convolution kernel size of 1×1 to obtain the low-frequency offset. The full-frequency primary features of the left and right views and the low-frequency offset are input into the deformable convolution to obtain the features of the left and right views registered at the low-frequency scale.

[0071] In the high-frequency binocular registration unit, the high-frequency primary features of the left and right views are cascaded with their differential features, and then sent to a convolution layer with a convolution kernel size of 1×1 to obtain the high-frequency offset. The features of the left and right views after registration at the low-frequency scale and the high-frequency offset are input into the deformable convolution to obtain the features of the left and right views registered at both the high and low frequency scales.

[0072] The specific implementation process is expressed by the formula:

[0073]

[0074] Among them, Conv represents the convolution with a convolution kernel size of 1×1, O lf and O hf represents the deformable convolution offset, DeConv represents the deformable convolution operation, and AF left and AF right are the features after alignment.

[0075] ② If Figure 3 As shown in the figure, the binocular competitive selection module consists of a spatial dimension selection block and a channel dimension selection block. The alignment features {AF left ,AF right} and full-frequency characteristics Add to get the module input {MF left ,MFright};

[0076] In the spatial dimension selection block, the stereo feature {MF left ,MF right After a convolution layer with a convolution kernel size of 1×1, we get Then F l With F r Transpose Perform matrix multiplication and pass through the SoftMax layer to obtain the attention map Then, the attention map {AM R→L ,AM L→R} and the module input {MF left ,MF right} to perform matrix multiplication:

[0077]

[0078] in is matrix multiplication, T means transpose, SF left and SF right It is the feature weighted by the disparity attention.

[0079] In the channel dimension selection block, the obtained SF left and SF right Send it to the global average pooling layer to get the channel attention map {AM left ,AM right}, and normalize the channel attention map pixel-wise:

[0080]

[0081] Then, the channel attention map is combined with SF left and SF right Perform point multiplication to obtain the fusion feature F of the binocular view b .

[0082]

[0083] Furthermore, F b Send it to two sets of convolution and pooling layers to extract the final binocular fusion feature F bino .

[0084] Finally, the binocular fusion feature F bino High and low frequency weighted features of left and right views and After splicing, the objective prediction score is obtained through the fully connected layer. During the training process, the network model uses the L1 loss function to measure the difference between the image quality score predicted by the network and the actual image quality score, and guide the optimization of the model.

[0085] 203: Use the trained network to evaluate image quality

[0086] The stereo image to be evaluated is sliced ​​and input into the network. The quality scores of all image blocks of each stereo image output by the network are averaged to obtain the quality score of the entire image.

[0087] Embodiment 3:

[0088] Based on Examples 1-2, but with the difference that the feasibility of the solutions in Examples 1 and 2 is verified in combination with specific experiments, as described below:

[0089] In order to verify the performance of the proposed method, we tested the proposed method on four datasets (LIVE I, LIVE II, WIVC I and WIVCII). In order to verify the performance of the proposed method in this paper, two common indicators were selected: Spearman rank order correlation coefficient (SROCC / SRCC) and Pearson linear correlation coefficient (PLCC). SROCC evaluates the monotonicity of the prediction of the method, and PLCC is used to describe the linear correlation between the predicted value and the true value. For PLCC and SROCC, the larger their values, the better the performance.

[0090] In order to verify the performance of the present invention, 7 mainstream no-reference stereo image quality evaluation methods were selected for comparison on the LIVE 3D database and the Waterloo IVC 3D database. These methods are 8 CNN-based evaluation methods (the methods of Fang, Liu, Yang, Zhou, Sim, Si and Chang et al.). The results are shown in Tables 1 and 2. In each column, the results of the two groups of methods with the best performance are shown in bold.

[0091] Table 1 Performance comparison of methods based on LIVE 3D image database

[0092]

[0093] Table 2 Performance comparison of methods based on Waterloo IVC 3D image database

[0094]

[0095] It can be seen from Table 1 and Table 2 that the performance indicators SROCC and PLCC of the present invention are at the leading level among all the compared methods based on the LIVE 3D image database and the Waterloo IVC 3D image database. The present invention has achieved very excellent results and has good generalization. The prediction results of the model are generally highly consistent with human subjective scores.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A stereo image quality evaluation method based on dual-frequency interactive enhancement and binocular matching, characterized in that: The following steps are involved: Step 1: Cut the left and right view original images into 32×32 non-overlapping image blocks, use FFT to convert the left and right view image blocks to the frequency domain and decompose them into high and low frequencies, and use IFFT to restore the high-frequency and low-frequency image blocks of the left and right views; Step 2: Train a convolutional neural network based on dual-frequency interaction enhancement and binocular matching. The convolutional neural network includes three subnetworks: a left view dual-frequency interaction feature enhancement subnetwork, a right view dual-frequency interaction feature enhancement subnetwork, and a binocular matching fusion subnetwork. Specifically, the following contents are included: The left and right view subnetworks are symmetrical. Taking the processing flow of the left view dual-frequency interactive feature enhancement subnetwork as an example, the specific contents include the following: In the left view dual-frequency interactive feature enhancement subnetwork, a set of convolution and pooling operations are used to extract primary feature maps of the high-frequency and low-frequency image blocks of the left view, and the dual-frequency interactive attention module is used to calculate the attention weights of the high-frequency and low-frequency signals, and the high- and low-frequency features are interactively weighted, and then three sets of convolutions are used to further extract features, and finally a dual-frequency reorganization module is used to obtain high- and low-frequency weighted feature maps; The processing flow of the binocular matching fusion sub-network specifically includes the following contents: in the binocular matching fusion sub-network, a set of convolution and pooling operations are used to extract the full-frequency primary feature map of the image blocks of the left and right views, and the high- and low-frequency information of the dual-frequency interactive attention module is used to enhance the features of the full-frequency primary feature map to obtain a full-frequency enhanced feature map; then, the enhanced full-frequency features of the left and right views are sequentially subjected to low-frequency binocular registration and high-frequency binocular registration; finally, a binocular competition selection module is used to perform feature screening in the spatial dimension and the channel dimension to obtain a binocular fusion feature map, and features are further extracted through two sets of convolutions; The binocular fusion feature map is concatenated with the high- and low-frequency weighted feature maps of the left and right views and then passed through a fully connected layer to obtain an objective prediction score. During the training process, the loss function of the network model uses L1 loss to measure the difference between the image quality score predicted by the network and the actual image quality score, and guides the optimization of the model. Step 3: Use the trained network to evaluate image quality. Slice the stereo image to be evaluated and input it into the network. Average the quality scores of all image blocks of each stereo image output by the network to obtain the quality score of the entire image.

2. The stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching according to claim 1, characterized in that: The processing flow of the dual-frequency interactive attention module described in step 2 specifically includes the following contents: The low-frequency primary features are passed through three convolutional layers with a convolution kernel of 1×1 in parallel to obtain three feature maps Q, K and V. The features Q and K are transformed in dimension and then matrix multiplication is performed, and then the low-frequency attention map is obtained through the SoftMax layer. Similarly, the same operation is performed on the high-frequency features to obtain the high-frequency attention map. Then, the low-frequency feature map V is matrix multiplied with the high-frequency attention map to obtain the low-frequency feature map after high-frequency information enhancement. The same operation is performed on the high-frequency primary features to obtain the high-frequency feature map after low-frequency information enhancement.

3. The stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching according to claim 1, characterized in that: The processing flow of the dual frequency reorganization module in step 2 specifically includes the following contents: The low-frequency and high-frequency high-level features are differentiated and summed, and the difference and sum are cascaded and sent to the residual module ResBlock composed of two convolutional layers and one residual connection to further extract features. The features are then passed through a global average pooling layer GAP and a fully connected layer to obtain adaptive weights. Finally, the weights are normalized and the low-frequency and high-frequency high-level features are weighted to obtain high- and low-frequency weighted features.

4. The stereo image quality evaluation method based on dual-frequency interaction enhancement and binocular matching according to claim 1, characterized in that: The binocular feature matching fusion module in step 2 is composed of a binocular progressive registration module and a binocular competition selection module, which specifically includes the following contents: ① The binocular progressive registration module consists of a low-frequency binocular registration unit and a high-frequency binocular registration unit; In the low-frequency binocular registration unit, the low-frequency primary features of the left and right views are cascaded with their differential features and sent to a convolution layer with a convolution kernel size of 1×1 to obtain the low-frequency offset. The full-frequency primary features of the left and right views and the low-frequency offset are sent to the deformable convolution to obtain the features of the left and right views registered at the low-frequency scale. In the high-frequency binocular registration unit, the high-frequency primary features of the left and right views are cascaded with their differential features and sent to a convolution layer with a convolution kernel size of 1×1 to obtain the high-frequency offset. The features of the left and right views after registration at the low-frequency scale and the high-frequency offset are sent to the deformable convolution to obtain the features of the left and right views registered at both the high and low frequency scales. ② The binocular competition selection module consists of a spatial dimension selection block and a channel dimension selection block. The registration features obtained by the binocular progressive registration module are added to the full-frequency primary features as the input of the module; In the spatial dimension selection block, the input left and right view features are first passed through a convolution layer with a convolution kernel size of 1×1 to obtain two sets of feature maps. Then the two sets of feature maps are transformed in dimension and matrix multiplied. Then they are passed through the SoftMax layer to obtain the disparity attention map. The disparity attention map is matrix multiplied with the left and right view features respectively and weighted in the spatial dimension. In the channel dimension selection block, the weighted features in the spatial dimension are sent to the global average pooling layer to obtain the left and right view channel attention map, and then the channel attention map is normalized and point-multiplied with the weighted features of the left and right views in the spatial dimension respectively. Finally, the weighted left and right view features in the channel dimension are added to obtain the binocular fusion feature.

Images