Comfort prediction method for stereo images combined with attention mechanism
By combining the stereoscopic image comfort prediction method with parallel dual attention mechanism, the problems of high complexity and inaccurate parallax calculation in the prior art are solved, and more accurate and efficient stereoscopic image comfort prediction is achieved.
Patent Information
- Application Number
- CN202210992815.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-18
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2042-08-18
AI Technical Summary
The existing stereo image comfort prediction methods are complex and cannot accurately provide parallax data related to visual perception, making it difficult to effectively predict the visual comfort of stereo images.
The three-dimensional image comfort prediction method combined with the parallel dual attention mechanism is adopted, and the feature maps output by multiple convolutional layers are summed and differential operations are performed, and the features are refined using the parallel dual attention module, and the comfort regression prediction is finally performed through the fully connected layer.
The process complexity is reduced, the impact of inaccurate parallax calculation is avoided, and the accuracy of comfort prediction is improved by taking into account the complex mechanisms of the human vision system and the introduction of parallel dual attention modules.
Smart Images
Figure CN115361557B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, in particular to the field of image processing technology, and specifically relates to a stereoscopic image comfort prediction method combined with an attention mechanism. Background Art
[0002] Compared with two-dimensional images, three-dimensional (3D) multimedia services can provide viewers with an immersive viewing experience and have been widely used in 3D television, immersive movies, advertising media, virtual reality and other fields. However, due to the influence of many factors, such as color mismatch between left and right images, large parallax and rapid depth change, people may feel visual discomfort when watching stereoscopic images, and experience symptoms such as eye fatigue and dizziness. These discomfort symptoms significantly affect people's visual experience and hinder the development and popularization of 3D multimedia technology. Therefore, researchers are increasingly paying attention to the research on stereoscopic image comfort prediction.
[0003] There are two main methods for predicting the comfort of stereoscopic images: subjective evaluation and objective evaluation. Among them, the subjective evaluation method needs to be carried out in a strict experimental environment and has extremely high requirements for experimental equipment. In addition, the implementation process of subjective evaluation is time-consuming and laborious. Therefore, it is necessary to develop an objective evaluation method to predict the visual comfort of stereoscopic images.
[0004] Previously, traditional stereo image comfort prediction methods mainly extracted relevant disparity features based on disparity maps, and then predicted comfort through machine learning regression algorithms. However, most algorithms for calculating disparity are very complex and cannot accurately provide disparity data related to perception. In addition, with the widespread application of deep learning in the field of image processing in recent years, traditional manual feature extraction methods are difficult to continue to obtain more effective stereo image discomfort features. Summary of the invention
[0005] Aiming at the problem that the current research on stereoscopic image comfort prediction is still immature, the present invention provides a stereoscopic image comfort prediction method combined with a parallel dual attention mechanism. The stereoscopic image comfort prediction model proposed in the present invention not only takes into account the complex mechanism of the human visual system and re-encodes the binocular images into their sum and difference information, but also uses a parallel dual attention mechanism to refine the features.
[0006] The method of the present invention firstly performs sum and difference operations on the feature maps output by multiple convolutional layers to obtain corresponding addition and subtraction feature maps, then refines the features through parallel dual attention modules to obtain attention-weighted features, and finally merges the output feature vectors of multiple sub-networks and inputs them into the fully connected layer for comfort regression prediction to obtain the final comfort prediction value.
[0007] The specific steps of the method of the present invention are as follows:
[0008] Step (1) obtains a plurality of image block pairs by performing semi-overlap sampling on the input left and right viewpoint images respectively.
[0009] Due to the small number of samples in the stereoscopic image dataset, methods such as flipping images and translation transformations used for data enhancement will change the comfort of stereoscopic images and are therefore not suitable for stereoscopic images. The image block-based data enhancement method can effectively solve the problem of insufficient number of training samples. Therefore, the semi-overlapping sampling method is used to expand the number of samples in the dataset. First, the input left view image and right view image are semi-overlapped and cropped to obtain multiple image blocks of 480×270 pixels in size. The cropped image blocks are used as output image blocks with a cropping interval of 240×135 pixels; each output image block is locally normalized to accelerate the convergence of the model during training. The number of image blocks obtained after cropping the left view image or the right view image each time Where M and N are the width and height of the input viewpoint image, respectively, and m and n are the width and height of the cropped image block. The output image blocks after local normalization of the left viewpoint image and the right viewpoint image are represented as P l k and k=1,2,…,K, the image block pair is represented as
[0010] Step (2) uses the image block pairs as model input to obtain the additive feature maps and subtractive feature maps of low-level, medium-level, and high-level features, respectively.
[0011] Studies have shown that there are binocular summation and difference channels in the visual cortex of the human brain. Therefore, in order to effectively encode binocular images, it is necessary to re-encode them into their sum and difference information. The model comprises two main subnetworks, three secondary subnetworks, and a fully connected layer network for regression prediction; the two main subnetworks input the left view image block and the right view image block respectively, and each main subnetwork is composed of the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer, the third pooling layer, the fourth convolution layer, the fourth pooling layer, the fifth convolution layer, the fifth pooling layer, the sixth convolution layer, the sixth pooling layer, and the seventh convolution layer in sequence. By performing network interaction after the second pooling layer, the fourth pooling layer, and the seventh pooling layer, low, medium, and high-level feature fusion is achieved, the output feature map of the second pooling layer is a low-level feature, the output feature map of the fourth pooling layer is a medium-level feature, and the output feature map of the seventh pooling layer is a high-level feature.
[0012] In each interaction process, the addition and subtraction feature maps of low-level, medium-level, and high-level features are first obtained through addition and subtraction operations. The calculation expression of the addition and subtraction feature maps of low-level features is: A low =Fl +F r , D low =F l -F r ; Among them, F l and F r Respectively represent the output feature maps of the left and right viewpoint image blocks after the second pooling layer, A low represents the added feature map obtained after the addition operation, D low Represents the subtraction feature map obtained after the subtraction operation. The same method is used to obtain the addition feature map A of the intermediate features. medium And subtract feature map D medium , the additive feature map A of high-level features high And subtract feature map D high .
[0013] Step (3) respectively concatenates the addition and subtraction feature maps of low-level, medium-level, and high-level features into new feature blocks, and inputs them into the parallel dual attention module for attention weighting.
[0014] Concatenate the addition and subtraction feature maps of low-level features into a new feature block F adl : F adl =(A low ,D low ), input into the parallel dual attention module; through the channel attention module in the parallel dual attention module, the feature block F is obtained adl The channel attention weight matrix Where C represents the number of input channels, is a real number domain; then, the obtained channel attention weight is used to adl Perform channel attention weighted operation to obtain channel weighted features Represents element-wise multiplication.
[0015] The feature block F is obtained by the spatial attention module in the parallel dual attention module adl The spatial attention weight matrix Use this spatial attention weight to the feature block F adl Perform spatial weighted operation to obtain the output spatial attention weighted feature F sl :
[0016] In the same way, the addition feature map and subtraction feature map of the medium and high-level features are spliced into new feature blocks, which are F adm and F adh , the channel and spatial attention weighted features of the intermediate features are obtained as F cm and F sm, the channel and spatial attention weighted features of the high-level features are F ch and F sh .
[0017] Step (4) concatenates the channel and spatial attention weighted features of the low, medium, and high-level features into new feature blocks, and inputs them into the corresponding secondary sub-networks respectively, and finally obtains the output feature vectors of the three secondary sub-networks.
[0018] The secondary sub-network A is composed of the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer, the third convolutional layer, the third pooling layer, the fourth convolutional layer, the fourth pooling layer and two fully connected layers in sequence. The secondary sub-network B is composed of the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer and two fully connected layers in sequence. The secondary sub-network C is composed of two fully connected layers.
[0019] First, the channel and spatial attention weighted features of the low-level features are concatenated into a new feature block F csl =(F cl ,F sl ), and F csl Input the secondary subnetwork A, and the output feature vector is F low ; The channel and spatial attention weighted features of the medium and high-level features are concatenated into new feature blocks, denoted as F csm and F csh , and input into the secondary sub-network B and the secondary sub-network C respectively. The output feature vector after the secondary sub-network B is F medium , the output feature vector after the secondary sub-network C is F high .
[0020] Step (5) merges the output feature vectors of the three secondary sub-networks and inputs them into the fully connected layer network for regression prediction to perform comfort regression prediction.
[0021] First, merge the output feature vectors of the secondary sub-networks A, B, and C into F all : F all =Concat(F low ,F medium ,F high ), Concat(·) means merging multiple one-dimensional feature vectors together. The fully connected layer network for regression prediction includes the first fully connected layer and the second fully connected layer.
[0022] Then, the output feature vector F all Input the fully connected layer network for regression prediction, and the final output value is the comfort score of each image block pair W1 represents the weight vector of the first fully connected layer, W2 represents the weight vector of the second fully connected layer, b1 represents the bias of the first fully connected layer, b2 represents the bias of the second fully connected layer, and g(·) represents the activation function.
[0023] Since the entire stereo image is cropped into multiple image block pairs, the comfort prediction values of all image block pairs are averaged and the average value is used as the comfort prediction value of the entire stereo image. Therefore, the comfort prediction value of the entire stereo image is Where K represents the number of image block pairs obtained for each stereo image.
[0024] During model training, the comfort mean opinion score (MOS) of the stereo image is used as the label value y for each image patch pair. k , and calculate the L1 loss as the minimization objective function during training:
[0025] Since traditional stereoscopic image comfort prediction methods are mainly based on disparity maps to extract disparity features, most algorithms for calculating disparity are very complex and cannot accurately provide disparity data related to visual perception. In the field of stereoscopic image comfort prediction research, the method of the present invention proposes a stereoscopic image comfort prediction method combined with a parallel dual attention mechanism. The method uses an end-to-end deep convolutional neural network structure, and the entire process does not require the use of disparity maps, which not only reduces the complexity of the process, but also avoids the impact of inaccurate disparity calculation. In addition, compared with previous methods, the method of the present invention not only takes into account the complex mechanism of the human visual system and re-encodes the binocular images into their sum and difference information, but also introduces a parallel dual attention module to further refine the features. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a schematic diagram of the process of the present invention;
[0027] Figure 2 The network structure of the model proposed by the method of the present invention;
[0028] Figure 3 The network structure of the channel attention module;
[0029] Figure 4 The network structure of the spatial attention module;
[0030] Figure 5 This is the loss function optimization process of the model proposed by the method of the present invention. DETAILED DESCRIPTION
[0031] The present invention is further described in detail with reference to the following specific embodiments and accompanying drawings.
[0032] like Figure 1 As shown in the figure, the stereoscopic image comfort prediction method combined with the attention mechanism is as follows:
[0033] Step (1) performs semi-overlap sampling on the input left and right images to obtain multiple image block pairs
[0034] First, the left and right input images are semi-overlapped and cropped to obtain multiple image blocks of 480×270 pixels with a cropping interval of 240×135 pixels. Each output image block is locally normalized to accelerate the convergence of the model during training. Then, they are used as the input of the proposed model. Let I l represents the left image, I r Represents the right image, the number of image blocks obtained after each stereo image is cropped Where M and N are the width and height of the input viewpoint image, respectively, and m and n are the width and height of the cropped image block. The output image blocks after local normalization of the left viewpoint image and the right viewpoint image are represented as P l k and k=1,2,…,K, the image block pair is represented as
[0035] Step (2) takes the image patch pairs as the input of the model to obtain the addition and subtraction feature maps of the low, medium and high level feature maps.
[0036] Studies have shown that there are binocular summation and difference channels in the human brain visual cortex. Therefore, in order to effectively encode binocular images, it is necessary to re-encode them into their sum and difference information. Figure 2 As shown in the figure, network interaction is performed after Conv2 / Maxpooling, Conv4 / Maxpooling and Conv7, respectively, so as to realize multi-level feature fusion. First, the addition feature map and the subtraction feature map of the feature map are calculated by addition and subtraction operations. The calculation expression of the addition feature map and the subtraction feature map is: A = F l +F r and D = F l -F r , where F l and F r They represent the feature maps output by the left and right image blocks after the convolution operation. A represents the added feature map obtained after the addition operation, and D represents the subtracted feature map obtained after the subtraction operation.
[0037] Step (3) concatenates the added and subtracted feature maps into a new feature block and inputs it into a parallel dual attention module for attention weighting.
[0038] Step (4) concatenates the channel and spatial attention weighted features of the low, medium, and high-level features into new feature blocks, and inputs them into the corresponding secondary sub-networks respectively, and finally obtains the output feature vectors of the three secondary sub-networks.
[0039] like Figure 2 As shown in Figure 1, the output addition and subtraction feature maps are spliced into a new feature block, which is then input into the parallel dual attention module. First, the channel attention weight matrix of the input feature F is generated based on the channel attention module in the parallel dual attention module. Where C represents the number of input channels, and the specific structure of the channel attention module is as follows Figure 3 Then, the obtained channel attention weights are used to perform channel weighted operations on the input features to obtain channel weighted features.
[0040] At the same time, the spatial attention weight matrix is obtained based on the spatial attention module in the parallel dual attention module The specific structure of the spatial attention module is as follows Figure 4 Finally, based on the spatial attention weight, the input feature is spatially weighted to obtain the output spatial weighted feature F s :
[0041] Finally, the obtained channel and spatial attention weighted features are concatenated into a new feature block: F cs =(F c ,F s ).
[0042] Step (5) merges the output features of the three secondary sub-networks and inputs them into the fully connected layer network for regression prediction to perform comfort regression prediction.
[0043] The output features of the three secondary sub-networks are F low ,F medium ,F high , representing low, medium, and high-level features respectively. The final input feature vector for comfort regression prediction is: F all =Concat(F low ,F medium ,F high ).
[0044] Then, the feature vector F all Input the fully connected layer network for regression prediction, and pass through two fully connected layers FC1 and FC2 (512-1) in sequence. The final output value is the comfort score of each image block pair. The comfort prediction value of the entire stereo image is:
[0045] The comfort mean opinion score (MOS) of the source stereo image is used as the label y for each image patch pair i , and calculate the L1 loss as the minimization objective function during training:
[0046]
[0047] like Figure 5 As shown in FIG. 1 , the optimization process of the loss function of the model proposed by the method of the present invention when training the IEEE-SA dataset is shown. It can be seen that the model can eventually converge well.
Claims
1. A stereo image comfort prediction method combining an attention mechanism, characterized by: Step (1) performing semi-overlap sampling on the input left and right viewpoint images respectively to obtain a plurality of image block pairs; First, the input left view image and right view image are semi-overlapped and cropped to obtain multiple image blocks of 480×270 pixels. The cropped image blocks are used as output image blocks with a cropping interval of 240×135 pixels. Each output image block is locally normalized. The number of image blocks obtained after cropping the left view image or the right view image each time is Where M and N are the width and height of the input viewpoint image, respectively, and m and n are the width and height of the cropped image block; the output image blocks after local normalization of the left viewpoint image and the right viewpoint image are represented as P l k and k=1,2,…,K, the image block pair is represented as Step (2) uses the image block pair as a model input to obtain the addition feature map and the subtraction feature map of the low, medium and high-level features respectively; the model comprises two main sub-networks, three secondary sub-networks, and a fully connected layer network for regression prediction; First, the addition and subtraction feature maps of low, medium and high-level features are obtained through addition and subtraction operations; the addition feature map A of the low-level features low =F l +F r , the subtraction and addition feature map D of low-level features low =F l -F r Among them, F l and F r Represent the low-level features in the output feature maps of the left and right viewpoint image blocks respectively; the same method is used to obtain the additive feature map A of the intermediate features medium And subtract feature map D medium , the additive feature map A of high-level features high And subtract feature map D high ; Step (3) respectively concatenates the addition and subtraction feature maps of low-level, medium-level, and high-level features into new feature blocks, and inputs them into the parallel dual attention module for attention weighting; Concatenate the addition and subtraction feature maps of low-level features into a new feature block F adl =(A low ,D low ), input into the parallel dual attention module; through the channel attention module in the parallel dual attention module, the feature block F is obtained adl The channel attention weight matrix Where C represents the number of input channels, is a real number domain; then, the obtained channel attention weight is used to adl Perform channel attention weighted operation to obtain channel weighted features represents element-wise multiplication; The feature block F is obtained by the spatial attention module in the parallel dual attention module adl The spatial attention weight matrix Use this spatial attention weight to the feature block F adl Perform spatial weighted operation to obtain the output spatial attention weighted feature F sl : In the same way, the addition feature map and subtraction feature map of the medium and high-level features are spliced into new feature blocks, which are F adm and F adh , the channel and spatial attention weighted features of the intermediate features are obtained as F cm and F sm , the channel and spatial attention weighted features of the high-level features are F ch and F sh ; Step (4) concatenates the channel and spatial attention weighted features of the low, medium, and high-level features into new feature blocks, and inputs them into the corresponding secondary sub-networks respectively, and finally obtains the output feature vectors of the three secondary sub-networks; First, the channel and spatial attention weighted features of the low-level features are concatenated into a new feature block F csl =(F cl ,F sl ), and F csl Input the secondary subnetwork A, and the output feature vector is F low ; The intermediate and advanced feature channels and spatial attention weighted features are concatenated into new feature blocks, denoted as F csm and F csh , and input into the secondary sub-network B and the secondary sub-network C respectively. The output feature vector after the secondary sub-network B is F medium , the output feature vector after the secondary sub-network C is F high ; Step (5) merging the output feature vectors of the three secondary sub-networks and inputting them into the fully connected layer network for regression prediction to perform comfort regression prediction; First, the output feature vectors of the three secondary sub-networks are merged into F all :F all =Concat(F low ,F medium ,F high ), Concat(·) means merging multiple one-dimensional feature vectors together. The fully connected layer network for regression prediction includes the first fully connected layer and the second fully connected layer; Then, the output feature vector F all Input the fully connected layer network for regression prediction, and the final output value is the comfort score of each image block pair W1 represents the weight vector of the first fully connected layer, W2 represents the weight vector of the second fully connected layer, b1 represents the bias of the first fully connected layer, b2 represents the bias of the second fully connected layer, and g(·) represents the activation function; the comfort prediction value of the entire stereo image is During model training, the comfort mean opinion score of the stereo image is used as the label value y for each image patch pair. k , and calculate the L1 loss as the minimization objective function during training:
2. The stereoscopic image comfort prediction method combined with the attention mechanism as claimed in claim 1, characterized in that: In step (2), the two main subnetworks input the left view image block and the right view image block respectively, and each main subnetwork is composed of the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer, the third pooling layer, the fourth convolution layer, the fourth pooling layer, the fifth convolution layer, the fifth pooling layer, the sixth convolution layer, the sixth pooling layer and the seventh convolution layer in sequence; by performing network interaction after the second pooling layer, the fourth pooling layer and the seventh pooling layer, low-, medium- and high-level feature fusion is realized, the output feature map of the second pooling layer is the low-level feature, the output feature map of the fourth pooling layer is the medium-level feature, and the output feature map of the seventh pooling layer is the high-level feature.
3. The stereoscopic image comfort prediction method combined with the attention mechanism as claimed in claim 1, characterized in that: In step (4), the secondary sub-network A is composed of the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer, the third convolutional layer, the third pooling layer, the fourth convolutional layer, the fourth pooling layer and two fully connected layers in sequence; the secondary sub-network B is composed of the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer and two fully connected layers in sequence; the secondary sub-network C is composed of two fully connected layers.