Virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion

Abstract

The invention discloses a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion. The method comprises the following steps: acquiring a virtual viewpoint image to be evaluated; inputting the virtual viewpoint image into a pre-trained visual residual error recovery network model constructed based on a U-Net network, and respectively extracting multi-stage degradation features and multi-stage recovery features; the virtual viewpoint image is input into a pre-trained OfficientNet-B0 network model, and multi-level content features are extracted through the model; performing adaptive fusion on the multi-scale degradation feature map, the recovery feature map and the content feature map through a multi-feature weight adaptive fusion module to generate a multi-layer fusion feature map; performing cross-channel information interaction on the multi-layer fusion feature map by using a channel attention module based on channel-level convolution to generate a fusion channel-level feature map; and mapping the fused channel-level feature map into a corresponding predicted objective quality score through a full connection layer. According to the method, the generalization ability, robustness and accuracy of the model during virtual viewpoint image quality evaluation can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual viewpoint image quality assessment, and in particular to a virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion. Background Art

[0002] With the emergence of immersive videos such as multi-viewpoint video, free viewpoint video, and multi-degree-of-freedom (DoF) video (3Dof, 3Dof+, and 6Dof), the user's sense of immersion and interactive experience have been greatly enhanced. Immersive video aims to provide users with an immersive visual experience, allowing users to freely explore every corner of the video through a panoramic view or virtual reality (VR) device. However, the transmission of immersive videos composed of a large number of viewpoints is limited by network bandwidth. Therefore, it is necessary to compress multiple viewpoints in the immersive video system. At the same time, in order to reduce the number of viewpoints transmitted, viewpoint synthesis technology is usually used to generate new viewpoints. Depth Image Based Rendering (DIBR) technology collects texture images of some viewpoints, fuses them with depth images, and synthesizes virtual viewpoints, thereby reducing the expensive cost of the camera capturing all viewpoints. However, in the virtual viewpoint rendering process, due to inaccurate depth map acquisition and imperfect rendering technology, the generated virtual viewpoint synthetic image will have some common distortion and geometric distortion different from traditional images, mainly manifested as texture distortion and structural distortion, which seriously affects people's visual experience. Therefore, how to effectively evaluate the quality of virtual viewpoint images has become one of the key issues in the development of virtual reality technology. It is crucial to establish a set of objective evaluation methods for virtual viewpoint image quality that can accurately reflect human visual perception.

[0003] At present, a series of quality assessment methods for virtual viewpoint images have been proposed. They are mainly divided into full-reference (FR) image quality assessment (IQA) methods and no-reference (NR) IQA methods according to whether the original image information is referenced. The FR method needs to obtain reference image information to evaluate the distorted virtual viewpoint image, and this type of method performs well for specific geometric distortions, but poorly for other distortions. However, the reference image of the virtual viewpoint image is difficult to obtain in practical applications. Researchers mainly use the NR method to evaluate the quality of virtual viewpoint images, which can complete the prediction of image quality by using only the image's own information.

[0004] At present, it is mainly divided into virtual viewpoint IQA methods based on manual feature extraction and virtual viewpoint IQA methods based on deep learning. The manual feature extraction model is limited by the complex distortion in the image and is not enough to measure the diversity of image features. The features extracted by the deep learning model are more similar to the visual perception of the human eye, and can integrate feature learning into network training, and output richer image texture information and structural information through a multi-layer network structure. Some existing NR-IQA methods are inspired by the free energy principle and use generative adversarial networks (GAN) to restore distorted images. The quality of the image is measured by the visual perception difference between the distorted image and its restored image. This method is limited by the small number of image samples and the complexity of image distortion, resulting in a large amount of false information in the restored image it generates. Therefore, using only the visual restoration network cannot effectively predict the quality of the synthetic image.

[0005] The applicant found that combining the visual restoration network with other feature extraction network tasks can more fully extract multi-layer features in the virtual viewpoint image. Therefore, how to further explore the NR virtual viewpoint image quality evaluation method based on multi-task deep learning on the basis of restoring the distortion-free virtual viewpoint synthetic image using the visual restoration network is an urgent problem to be solved by technicians in this field. Summary of the invention

[0006] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion, extracting multi-scale feature maps respectively through a pre-trained visual residual restoration network model and an EfficientNet-B0 network model, and then adaptively fusing the multi-scale feature maps extracted by the two networks, thereby improving the generalization ability, robustness and accuracy of the model in virtual viewpoint image quality evaluation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion includes:

[0009] S1: Obtain the virtual viewpoint image to be evaluated;

[0010] S2: Input the virtual viewpoint image into the pre-trained visual residual restoration network model based on the U-Net network, extract multi-level degradation features and multi-level restoration features through the encoding end and decoding end of the model, and generate multi-scale degradation feature maps and restoration feature maps;

[0011] S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map;

[0012] S4: Adaptively fuse the multi-scale degradation feature map, restoration feature map and content feature map through the multi-feature weight adaptive fusion module to generate a multi-layer fusion feature map;

[0013] S5: Use the channel attention module based on channel-level convolution to interact cross-channel information of multi-layer fusion feature maps to generate fusion channel-level feature maps;

[0014] S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through a fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

[0015] Preferably, in step S2, the visual residual restoration network model includes an encoder and a decoder;

[0016] The encoder includes a 3×3 convolutional layer and a number of encoding ends connected end to end in sequence with different scales, and one encoding end includes a parallel asymmetric residual network of the corresponding scale; wherein the parallel asymmetric residual network is used to capture spatial information during the forward propagation process;

[0017] The decoder includes a number of decoders connected end to end in sequence and corresponding to the scales of the encoders and a 3×3 convolution layer. A decoder includes a 3×3 deconvolution layer and a parallel asymmetric residual network of the corresponding scale; wherein the output of each encoder is jump-connected to the decoder of the corresponding scale;

[0018] The feature maps output by the encoding end of all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoding end of all scales are used as multi-scale multi-level restoration feature maps.

[0019] Preferably, the parallel asymmetric residual network includes a first group of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second group of parallel convolutions, a second batch normalization layer, and a second PReLU layer connected end to end in sequence; wherein the input feature map of the parallel asymmetric residual network is reduced in dimension by a 1×1 convolution layer, and then added to the output of the second batch normalization layer and input into the second PReLU layer;

[0020] The first and second groups of parallel convolutions input the input feature map to the 1×3 horizontal convolution layer, 3×1 vertical convolution layer and 3×3 convolution layer respectively, and then perform element-by-element addition operation on the outputs of the 1×3 horizontal convolution layer, 3×1 vertical convolution layer and 3×3 convolution layer to generate the output feature map;

[0021] The calculation formula of the parallel asymmetric residual network is expressed as:

[0022]

[0023] Where: R and I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; O1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents a 1×1 convolution; C3 represents a 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolutions respectively; Represents element-by-element addition operation; f BN and f ReLU They represent batch normalization operation and PReLU activation operation respectively.

[0024] Preferably, in step S3, the feature maps output by multiple Stage layers of the EfficientNet-B0 network model are used as multi-scale multi-level content feature maps.

[0025] Preferably, in step S4, the processing steps of the multi-feature weight adaptive fusion module include:

[0026] S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map;

[0027] S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the degradation feature map and the content feature map through a weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map;

[0028] S403: performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map;

[0029] S404: concatenating the multi-scale enhanced restoration feature map with the weighted adaptive fusion feature map of the corresponding scale and performing a global average pooling operation to obtain a multi-scale fusion feature map; cascading the multi-scale fusion feature map to generate a final multi-layer fusion feature map;

[0030] The calculation formula of the multi-layer fusion feature map is expressed as:

[0031]

[0032] Where: F fusionrepresents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1,2,3,4) scale; F2(i) represents the enhanced restoration feature map of the i-th (i=1,2,3,4) scale; GAP represents the global average pooling operation; CAT and Both represent cascade operations.

[0033] Preferably, in step S401, the feature enhancement module inputs its input feature map into three branches arranged in parallel, connects the outputs of the three branches, performs batch normalization and PReLU activation operations, and outputs the corresponding enhanced feature map; the first branch includes a 1×1 convolutional layer; the second branch includes a 3×3 convolutional layer; the third branch includes two 3×3 convolutional layers connected end to end;

[0034] The calculation formula of the enhanced degradation feature map is expressed as:

[0035]

[0036] The calculation formula of the enhanced content feature map is expressed as:

[0037]

[0038] Where: E i (f D )、E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) and (I) C (i) represents the degradation feature map and content feature map of the i-th (i=1, 2, 3, 4) scale of the input respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; Indicates a cascade operation.

[0039] Preferably, in step S402, the weight adaptive generation module cascades and splices the input enhanced degradation feature map and the enhanced content feature map, and then generates weight maps of two modes using two separate 3×3 convolutions and sigmoid functions; multiplies the two weight maps with the enhanced degradation feature map and the enhanced content feature map respectively to obtain corresponding weighted degradation feature maps and weighted content feature maps; cascades the weighted degradation feature map and the weighted content feature map to generate a weight adaptive fusion feature map;

[0040] The calculation formula of weight adaptive fusion feature is expressed as:

[0041]

[0042] Where: F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1, 2, 3, 4) scale; E i (f D )、E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F C (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale (i=1, 2, 3, 4), σ represents the sigmoid function, and ⊙ represents pixel-by-pixel multiplication. Indicates a cascade operation.

[0043] Preferably, in step S403, the calculation formula of the enhanced restored feature map is expressed as:

[0044]

[0045] Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th (i=1, 2, 3, 4) scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates a cascade operation.

[0046] Preferably, in step S5, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize spatial information interaction between channels, and obtain a weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain a fusion channel-level feature map;

[0047] The calculation formula for fusion channel-level features is expressed as:

[0048] F=F fusion ☉σ(Conv 3×3 (GAP(F fusion )));

[0049] Where: F represents the fused channel-level feature map; F fusion Represents a multi-layer fusion feature map; GAP represents a global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

[0050] Preferably, in step S6, the fully connected layer generates a corresponding predicted objective quality score after passing the fused channel-level feature map through a three-layer fully connected neural network and two PReLU activation functions.

[0051] Compared with the prior art, the virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion in the present invention has the following beneficial effects:

[0052] The present invention extracts multi-scale feature maps respectively through a pre-trained visual residual restoration network model and an EfficientNet-B0 network model. On the one hand, the visual residual restoration network follows the free energy principle, reflects the degradation degree of the distorted image through the degradation features of different layers of the encoding end, reflects the recovery degree of the distorted image by using the gradually restored restored image features, and effectively extracts the multi-level degradation features and multi-level restoration features of the image by simulating the brain to repair the distorted image. On the other hand, the EfficientNet-B0 network is trained through a large sample database, and the traditional distortion existing in ordinary images can be learned to extract the multi-level content features of the virtual viewpoint image, so that the network can extract more comprehensive distortion information. Among them, the visual residual restoration network model mainly focuses on the synthetic distortion of the image, while the EfficientNet-B0 network model is more sensitive to ordinary distortion. The multi-scale feature maps extracted by the two networks have large differences and can also complement information. By fusing the multi-scale feature maps extracted by the two networks, it can cope with the complex geometric distortion on the virtual viewpoint image, thereby improving the generalization ability and robustness of the model in the virtual viewpoint image quality evaluation.

[0053] Aiming at the differences between the feature maps extracted by the pre-trained visual residual restoration network model and the EfficientNet-B0 network model, the present invention proposes a multi-feature weight adaptive fusion module to adaptively fuse the multi-scale degradation feature map, restoration feature map and content feature map. The multi-feature weight adaptive fusion module can adaptively adjust the weights of the multi-layer and multi-scale features extracted from two different networks, assign optimal weights to the two types of features with large differences, adaptively adjust the weights of different distortions through weights, and fuse the distorted image fusion features with the multi-level restoration features at the decoding end to achieve better results for the model, thereby improving the accuracy of the model in the virtual viewpoint image quality evaluation.

[0054] In order to further improve the information interaction capability between channels, the present invention proposes a channel attention module to realize information interaction between different channels. By obtaining channel-level features that are more relevant to image quality, features that are more relevant to image quality are obtained in the quality prediction network, thereby improving the accuracy of virtual viewpoint image quality evaluation. Experiments show that compared with existing IQA methods, the method of the present invention achieves the best performance on two virtual viewpoint image databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0056] Figure 1 It is a logical block diagram of a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion;

[0057] Figure 2 This is the structural diagram of the parallel asymmetric residual network;

[0058] Figure 3 It is the structural diagram of the multi-feature weight adaptive fusion module;

[0059] Figure 4 This is the structural diagram of the channel attention module based on channel-level convolution;

[0060] Figure 5 It is a multi-scale degradation feature map and content feature map;

[0061] Figure 6 It is a multi-scale restored feature map. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0063] The following is a further detailed description through specific implementation methods:

[0064] Example:

[0065] This embodiment discloses a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion.

[0066] like Figure 1 As shown, the virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion includes:

[0067] S1: Obtain the virtual viewpoint image to be evaluated;

[0068] S2: Input the virtual viewpoint image into the pre-trained visual residual restoration network model based on the U-Net network, extract multi-level degradation features and multi-level restoration features through the encoding end and decoding end of the model, and generate multi-scale degradation feature maps and restoration feature maps;

[0069] S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map;

[0070] S4: Adaptively fuse the multi-scale degradation feature map, restoration feature map and content feature map through the Multi-feature Weight Adaptive Fusion Block (MWFB) to generate a multi-layer fusion feature map;

[0071] S5: Use the channel attention module based on channel-level convolution to interact cross-channel information of multi-layer fusion feature maps to generate fusion channel-level feature maps;

[0072] S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through a fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

[0073] The present invention extracts multi-scale feature maps respectively through a pre-trained visual residual restoration network model and an EfficientNet-B0 network model. On the one hand, the visual residual restoration network follows the free energy principle, reflects the degradation degree of the distorted image through the degradation features of different layers of the encoding end, reflects the recovery degree of the distorted image by using the gradually restored restored image features, and effectively extracts the multi-level degradation features and multi-level restoration features of the image by simulating the brain to repair the distorted image. On the other hand, the EfficientNet-B0 network is trained through a large sample database, and the traditional distortion existing in ordinary images can be learned to extract the multi-level content features of the virtual viewpoint image, so that the network can extract more comprehensive distortion information. Among them, the visual residual restoration network model mainly focuses on the synthetic distortion of the image, while the EfficientNet-B0 network model is more sensitive to ordinary distortion. The multi-scale feature maps extracted by the two networks have large differences and can also complement information. By fusing the multi-scale feature maps extracted by the two networks, it can cope with the complex geometric distortion on the virtual viewpoint image, thereby improving the generalization ability and robustness of the model in the virtual viewpoint image quality evaluation.

[0074] Aiming at the differences between the feature maps extracted by the pre-trained visual residual restoration network model and the EfficientNet-B0 network model, the present invention proposes a multi-feature weight adaptive fusion module to adaptively fuse the multi-scale degradation feature map, restoration feature map and content feature map. The multi-feature weight adaptive fusion module can adaptively adjust the weights of the multi-layer and multi-scale features extracted from two different networks, assign optimal weights to the two types of features with large differences, adaptively adjust the weights of different distortions through weights, and fuse the distorted image fusion features with the multi-level restoration features at the decoding end to achieve better results for the model, thereby improving the accuracy of the model in the virtual viewpoint image quality evaluation.

[0075] In order to further improve the information interaction capability between channels, the present invention proposes a channel attention module to realize information interaction between different channels. By obtaining channel-level features that are more relevant to image quality, features that are more relevant to image quality are obtained in the quality prediction network, thereby improving the accuracy of virtual viewpoint image quality evaluation. Experiments show that compared with existing IQA methods, the method of the present invention achieves the best performance on two virtual viewpoint image databases.

[0076] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0077] 1. Visual Residual Restoration Network Model

[0078] In this embodiment, the visual residual restoration network model is built based on the U-Net network. In order to solve the problem of a large amount of pseudo information when restoring distorted images using the improved U-Net network, this embodiment is based on the free energy principle, that is, when the human visual system (Human Visual System, HVS) quantizes the distorted virtual viewpoint image, it usually needs to compare it with the undistorted original image in the mind to perceive the quality difference. A visual residual restoration network is designed, which uses the improved U-Net network to restore the distorted image to the undistorted image. Its overall structure is as follows Figure 1 (a) shown.

[0079] Specifically, the visual residual restoration network model includes an encoder and a decoder; the encoder includes a 3×3 convolutional layer and several encoding ends connected end to end in sequence and with different scales, and one encoding end includes a parallel asymmetric residual network of the corresponding scale; wherein the parallel asymmetric residual network is used to capture spatial information during the forward propagation process; the decoder includes several decoding ends connected end to end in sequence and corresponding to the scales of the encoding ends and a 3×3 convolutional layer, and one decoding end includes a 3×3 deconvolution layer and a parallel asymmetric residual network of the corresponding scale; wherein the output of each encoding end is jump-connected to the decoding end of the corresponding scale; the feature maps output by the encoding ends of all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoding ends of all scales are used as multi-scale multi-level restoration feature maps.

[0080] When using the visual residual restoration network for image feature extraction, it is necessary to retain the important edge texture structure in the image. The ordinary convolution layer cannot extract the correct features after image preprocessing. Therefore, this embodiment designs a parallel asymmetric residual network, which replaces each ordinary 3×3 convolution with a parallel convolution of 1×3 horizontal convolution kernel, 3×1 vertical convolution kernel and 3×3 convolution kernel, thereby improving the robustness of feature extraction of the entire model.

[0081] Combination Figure 2 As shown, the parallel asymmetric residual network includes a first group of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second group of parallel convolutions, a second batch normalization layer, and a second PReLU layer connected end to end in sequence; wherein, the input feature map of the parallel asymmetric residual network is reduced in dimension by a 1×1 convolution layer, and then added to the output of the second batch normalization layer and input into the second PReLU layer; the first group and the second group of parallel convolutions input the input feature map to a 1×3 horizontal convolution layer, a 3×1 vertical convolution layer, and a 3×3 convolution layer arranged in parallel, respectively, and then the outputs of the 1×3 horizontal convolution layer, the 3×1 vertical convolution layer, and the 3×3 convolution layer are added element by element to generate an output feature map;

[0082] The calculation formula of the parallel asymmetric residual network is expressed as:

[0083]

[0084] Where: R and I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; P1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents a 1×1 convolution; C3 represents a 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolutions respectively; Represents element-by-element addition operation; f BN and f ReLU They represent batch normalization operation and PReLU activation operation respectively.

[0085] The hierarchical degradation characteristics of the visual residual restoration network have the same characteristics as the perception mechanism of the human visual system (HVS). Therefore, this embodiment reflects the degradation degree of the distorted image by extracting the degradation features of different layers of the visual residual restoration network encoding end, and extracts the feature maps output by the encoding ends D1, D2, D3 and D4 as multi-scale multi-level degradation feature maps. The decoding end of the visual residual restoration network obtains a reconstructed high-quality image by jump-connecting the deconvolution with the low-quality image features. The decoding ends R1 to R4 of the decoding end are the process of gradual image restoration, and their feature maps are semantically closer and closer to high-quality images. The gradually restored restored image features can be used to reflect the degree of restoration of the distorted image, so as to better learn the relationship between the virtual viewpoint image features and the human eye quality perception. The feature maps output by the decoding ends R1, R2, R3, and R4 are extracted as multi-scale multi-level restoration feature maps.

[0086] Due to the insufficient number of samples in the virtual viewpoint image database, it is difficult to train a better restoration network model. Therefore, this embodiment pre-trains the visual residual restoration network on a normal image dataset so that it can learn richer visual scenes, thereby improving the visual restoration ability of the network model. During pre-training, the resolution of the input image is adjusted to 256×256, and then it is randomly cropped into 224×224 pixel blocks. The initial learning rate is set to 1×10-3, and the Adam optimizer is used to update the network weights for 500 epochs, and finally the pre-trained weights of the visual residual restoration network are obtained.

[0087] 2. EfficientNet-B0 network model

[0088] In this embodiment, the structure of the EfficientNet-B0 network model is shown in the following table, including 16 mobile flip bottleneck convolution (MBConv) modules, 2 convolution layers, 1 global average pooling layer and 1 classification layer. The pre-training of the EfficientNet-B0 network model is achieved by existing means.

[0089] Specifically, the feature maps output by the 2nd, 3rd, 4th, and 6th Stage layers of the EfficientNet-B0 network model are extracted as multi-scale multi-level content (perception) feature maps.

[0090]

[0091] Figure 5 The multi-scale degradation feature maps and content feature maps extracted by the visual residual recovery network model and the EfficientNet-B0 network model are Figure 5 (a) is the input distorted virtual viewpoint image, Figure 5 (b)-(e) are multi-scale degradation feature maps. Figure 5 (f)-(i) are multi-scale content feature maps. It can be seen that the visual residual restoration network mainly focuses on the synthetic distortion of the image, while the EfficientNet-B0 network model is more sensitive to ordinary distortion. The multi-scale feature maps extracted by the two networks are quite different and can also complement each other.

[0092] Figure 6 The figure shows the process of restoring the distorted virtual viewpoint image through the visual residual restoration network. Figure 6 (a) The input is a distorted virtual viewpoint image. Figure 6 (b)-(e) are multi-scale restored feature maps. It can be seen that the quality of the feature maps from R1 to R4 layers is getting closer and closer to the original image, which intuitively reflects the restoration process of the distorted image.

[0093] 3. Multi-feature weight adaptive fusion module (MWFB)

[0094] In order to solve the problem of fusing different features extracted by the multi-task network, this embodiment takes into account that the multi-scale feature maps extracted by the two networks have large differences and can also complement each other. Therefore, different weights can be assigned to the distortion features according to their importance, and a multi-feature weight adaptive fusion module is designed. The specific structure of the module is as follows: Figure 3 As shown in Figure 2, the module consists of two feature enhancement blocks (FEB) and a weight adaptive generation block (WAGB).

[0095] In this embodiment, the processing steps of the multi-feature weight adaptive fusion module include:

[0096] S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map;

[0097] S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the degradation feature map and the content feature map through a weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map;

[0098] S403: performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map;

[0099] S404: concatenating the multi-scale enhanced restored feature map with the weighted adaptive fused feature map of the corresponding scale and performing a global average pooling (GAP) operation to obtain a multi-scale fused feature map; cascading the multi-scale fused feature map to generate a final multi-layer fused feature map;

[0100] The calculation formula of the multi-layer fusion feature map is expressed as:

[0101]

[0102] Where: F fusion represents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1,2,3,4) scale; F2(i) represents the enhanced restoration feature map of the i-th (i=1,2,3,4) scale; GAP represents the global average pooling operation; CAT and Both represent cascade operations.

[0103] 1. Feature Enhancement Module (FEB)

[0104] The feature enhancement module is composed of 1×1, 3×3 and 5×5 convolutional layers in parallel. In order to reduce the computational complexity of the network and ensure the same receptive field as the 5×5 convolutional layer, two 3×3 convolutions are used to replace the 5×5 convolution. The feature enhancement module inputs its input feature map to three parallel branches respectively, connects the outputs of the three branches, performs batch normalization and PReLU activation operations, and outputs the corresponding enhanced feature map; the first branch includes a 1×1 convolutional layer; the second branch includes a 3×3 convolutional layer; the third branch includes two 3×3 convolutional layers connected head to tail; the degradation feature map and the content feature map are passed through FEB to obtain two enhanced features.

[0105] The calculation formula of the enhanced degradation feature map is expressed as:

[0106]

[0107] The calculation formula of the enhanced content feature map is expressed as:

[0108]

[0109] Where: E i (f D )、E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) and (I) C (i) represents the degradation feature map and content feature map of the i-th (i=1, 2, 3, 4) scale of the input respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; Indicates a cascade operation.

[0110] In this embodiment, the calculation formula of the enhanced restoration feature map is expressed as:

[0111]

[0112] Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th (i=1, 2, 3, 4) scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates a cascade operation.

[0113] 2. Weight Adaptive Generation Module (WAGB)

[0114] The weight adaptive generation module concatenates the input enhanced degradation feature map and enhanced content feature map, and then generates weight maps of the two modes using two separate 3×3 convolutions and sigmoid functions; the two weight maps are multiplied with the enhanced degradation feature map and the enhanced content feature map respectively to obtain the corresponding weighted degradation feature map and weighted content feature map; the weighted degradation feature map and the weighted content feature map are concatenated to generate a weight adaptive fusion feature map;

[0115] The calculation formula of weight adaptive fusion feature is expressed as:

[0116]

[0117] Where: F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1, 2, 3, 4) scale; E i (f D )、E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F C (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale (i=1, 2, 3, 4), σ represents the sigmoid function, and ⊙ represents pixel-by-pixel multiplication. Indicates a cascade operation.

[0118] 4. Channel Attention Module

[0119] In order to avoid the interference of some useless information in the fusion features on the quality prediction network, this embodiment designs a channel attention mechanism based on channel-level convolution to achieve cross-channel information interaction to obtain channel-level features that are more relevant to image quality. Figure 4 As shown in the figure, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize the spatial information interaction between channels and obtain the weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain the fusion channel-level feature map;

[0120] The calculation formula for fusion channel-level features is expressed as:

[0121] F=F fusion ☉σ(Conv 3×3 (GAP(F fusion )));

[0122] Where: F represents the fused channel-level feature map; F fusion Represents a multi-layer fusion feature map; GAP represents a global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

[0123] 5. Fully connected layer

[0124] In this embodiment, the fully connected layer generates the corresponding predicted objective quality score after fusing the channel-level feature map through three layers of fully connected neural network and two PReLU activation functions.

[0125] 6. Experimental Description

[0126] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0127] 1. Experimental description

[0128] The performance of this experiment on 84 virtual viewpoint image samples of the IRCCyN / IVC database is compared and analyzed with that of other IQA methods, and the advancement of the present invention is further demonstrated through a series of experiments.

[0129] This experiment was implemented in the Pytorch1.10 deep learning framework and python3.8 environment, and the model was trained and tested on the NVIDIA3080 GPU. When training the entire network, the insufficient number of samples in the virtual viewpoint image database and the different image sizes in the same database were taken into account. Therefore, when training the entire network, the input image size was first resized to 768×1024 pixels, and then randomly cropped into 25 224×224 pixel blocks, where the subjective quality score of each pixel block was provided by the original distorted image. In addition, the images in each database were randomly grouped, with 80% of the images used as training sets and 20% of the images used as test sets. The entire network uses Adam to optimize the network, and the learning rate is set to 2×10-4 for a total of 50 epochs. In order to accurately measure the mapping relationship between the predicted score and the subjective score, the L loss function is used to fine-tune the training set, which is defined as:

[0130]

[0131] Where N is the number of blocks of the virtual viewpoint distorted image during batch processing, Q i is the prediction score of the i-th image block, DMOS i is the subjective score of the i-th image block, and l1 represents the l1 norm.

[0132] 2. Performance evaluation

[0133] In order to accurately evaluate the performance of the entire model, this experiment used three evaluation indicators commonly used in objective quality evaluation methods proposed by the video expert group: Pearson linear correlation coefficient (PLCC), Spearman rank correlation coefficient (SROCC) and root mean square error (RMSE). PLCC represents the correlation between subjective scores and predicted scores. The higher the PLCC, the higher the correlation. SROCC reflects the monotonic correlation between subjective scores and predicted scores. The higher the SROCC, the better the model performance. RMSE represents the absolute error between the predicted score and the subjective score. The smaller the RMSE, the higher the accuracy and stability of the model. The above three evaluation indicators are used to measure the consistency between the objective quality score and the subjective score, so as to effectively judge the pros and cons of the objective quality evaluation method.

[0134] To prove the effectiveness of the proposed method, Table 1 lists the performance comparison results of the proposed method and other 17 IQA methods on the IRCCyN / IVC virtual viewpoint image database, and the best ones are in bold. There are 3 quality evaluation methods designed for natural images: PSNR, SSIM and BIQI; in addition, there are 14 quality evaluation methods designed based on virtual viewpoint images: MP-PSNR, SSPD, SEQUSS, PU-IR&DF-CS, Zhang's, APT, NIQSV, NIQSV+, MNSS, Wang's, Fang's, Sadbhawna's, GANs-NQM and Jakhetiya's. The above three performance indicators are used to judge the performance of all methods.

[0135] As can be seen from Table 1, compared with other advanced IQA methods, the performance of the present invention is the best and exceeds all FR IQA methods. It can be seen that the virtual viewpoint image quality evaluation method based on deep learning can achieve better performance than the traditional method. In addition, the SSPD, Wang's and Fang's methods that simultaneously consider image texture and structural distortion have achieved the best performance among all traditional methods, but their performance is still lower than that of the deep learning method. The PLCC, SROCC and RMSE of the present invention reached 0.9063, 0.9283 and 0.2956 respectively, which further proves the wide applicability of the method of the present invention.

[0136] Table 1 Performance of the present invention and other methods on the IRCCyN / IVC virtual viewpoint image database

[0137]

[0138]

[0139] 3. Ratio of training and test sets

[0140] In order to verify the influence of the proportion of training test samples on the performance of the method of the present invention, we randomly selected 50%, 60%, 70%, 80% and 90% of the pictures as training samples on the IRCCyN / IVC database, and the remaining pictures were used as test samples for experiments. Table 2 lists the performance results of training with different proportions of training sets. As can be seen in Table 2, when the proportion of training sets increases, the two performance indicators PLCC and SROCC also increase and tend to stabilize. Even if the proportion of training sets is only 50%, the SROCC of the present invention has reached 0.8207, and the performance is still better than most virtual viewpoint image quality evaluation methods.

[0141] Table 2 Performance results of the methods under different training set ratios

[0142]

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion, characterized in that: include: S1: Obtain the virtual viewpoint image to be evaluated; S2: Input the virtual viewpoint image into the pre-trained visual residual restoration network model based on the U-Net network, extract multi-level degradation features and multi-level restoration features through the encoding end and decoding end of the model, and generate multi-scale degradation feature maps and restoration feature maps; S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map; S4: Adaptively fuse the multi-scale degradation feature map, restoration feature map and content feature map through the multi-feature weight adaptive fusion module to generate a multi-layer fusion feature map; S5: Use the channel attention module based on channel-level convolution to interact cross-channel information of multi-layer fusion feature maps to generate fusion channel-level feature maps; S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through a fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

2. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S2, the visual residual restoration network model includes an encoder and a decoder; The encoder includes a 3×3 convolutional layer and a number of encoding ends connected end to end in sequence with different scales, and one encoding end includes a parallel asymmetric residual network of the corresponding scale; wherein the parallel asymmetric residual network is used to capture spatial information during the forward propagation process; The decoder includes a number of decoders connected end to end in sequence and corresponding to the scales of the encoders and a 3×3 convolution layer. A decoder includes a 3×3 deconvolution layer and a parallel asymmetric residual network of the corresponding scale; wherein the output of each encoder is jump-connected to the decoder of the corresponding scale; The feature maps output by the encoding end of all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoding end of all scales are used as multi-scale multi-level restoration feature maps.

3. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 2, characterized in that: The parallel asymmetric residual network includes a first group of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second group of parallel convolutions, a second batch normalization layer, and a second PReLU layer connected end to end in sequence; wherein, the input feature map of the parallel asymmetric residual network is reduced in dimension by a 1×1 convolution layer, and then added to the output of the second batch normalization layer and input into the second PReLU layer; The first and second groups of parallel convolutions input the input feature map to the 1×3 horizontal convolution layer, 3×1 vertical convolution layer and 3×3 convolution layer respectively, and then perform element-by-element addition operation on the outputs of the 1×3 horizontal convolution layer, 3×1 vertical convolution layer and 3×3 convolution layer to generate the output feature map; The calculation formula of the parallel asymmetric residual network is expressed as: Where: R and I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; P1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents a 1×1 convolution; C3 represents a 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolutions respectively; Represents element-by-element addition operation; f BN and f ReLU They represent batch normalization operation and PReLU activation operation respectively.

4. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S3, the feature maps output by multiple Stage layers of the EfficientNet-B0 network model are used as multi-scale multi-level content feature maps.

5. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S4, the processing steps of the multi-feature weight adaptive fusion module include: S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map; S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the degradation feature map and the content feature map through a weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map; S403: performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map; S404: concatenating the multi-scale enhanced restoration feature map with the weighted adaptive fusion feature map of the corresponding scale and performing a global average pooling operation to obtain a multi-scale fusion feature map; cascading the multi-scale fusion feature map to generate a final multi-layer fusion feature map; The calculation formula of the multi-layer fusion feature map is expressed as: Where: F fusion represents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1,2,3,4) scale; F2(i) represents the enhanced restoration feature map of the i-th (i=1,2,3,4) scale; GAP represents the global average pooling operation; CAT and Both represent cascade operations.

6. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S401, the feature enhancement module inputs its input feature map to three branches set in parallel, connects the outputs of the three branches, performs batch normalization and PReLU activation operations, and outputs the corresponding enhanced feature map; the first branch includes a 1×1 convolutional layer; the second branch includes a 3×3 convolutional layer; the third branch includes two 3×3 convolutional layers connected end to end; The calculation formula of the enhanced degradation feature map is expressed as: The calculation formula of the enhanced content feature map is expressed as: Where: E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) and (I) C (i) represents the degradation feature map and content feature map of the i-th (i=1, 2, 3, 4) scale of the input respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLu Represent batch normalization operation and PReLU activation operation respectively; Indicates a cascade operation.

7. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 5, characterized in that: In step S402, the weight adaptive generation module cascades and splices the input enhanced degradation feature map and enhanced content feature map, and then generates weight maps of two modes using two separate 3×3 convolutions and sigmoid functions; multiplies the two weight maps with the enhanced degradation feature map and the enhanced content feature map respectively to obtain the corresponding weighted degradation feature map and weighted content feature map; cascades the weighted degradation feature map and the weighted content feature map to generate a weight adaptive fusion feature map; The calculation formula of weight adaptive fusion feature is expressed as: Where: F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1, 2, 3, 4) scale; E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F C (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale (i=1, 2, 3, 4), σ represents the sigmoid function, and ⊙ represents pixel-by-pixel multiplication. Indicates a cascade operation.

8. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S403, the calculation formula of the enhanced restored feature map is expressed as: Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th (i=1, 2, 3, 4) scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates a cascade operation.

9. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S5, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize the spatial information interaction between the channels, and obtain the weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain a fusion channel-level feature map; The calculation formula for fusion channel-level features is expressed as: F=F fusion ⊙σ(Conv 3×3 (GAP(F fusion ))); Where: F represents the fused channel-level feature map; F fusion Represents a multi-layer fusion feature map; GAP represents a global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

10. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S6, the fully connected layer generates the corresponding predicted objective quality score after the fused channel-level feature map passes through three layers of fully connected neural network and two PReLU activation functions.

Images