Virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion

Abstract

The present invention discloses a virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion, comprising the following steps: obtaining a virtual viewpoint image to be evaluated; inputting the virtual viewpoint image into a pre-trained visual residual recovery network model constructed based on a U-Net network to extract multi-level degradation features and multi-level recovery features respectively; inputting the virtual viewpoint image into a pre-trained EfficientNet-B0 network model to extract multi-level content features through the model; adaptively fusing multi-scale degradation feature maps, recovery feature maps, and content feature maps through a multi-feature weight adaptive fusion module to generate a multi-layer fused feature map; utilizing a channel attention module based on channel-level convolution to perform cross-channel information interaction on the multi-layer fused feature maps to generate a fused channel-level feature map; and mapping the fused channel-level feature map into a corresponding predicted objective quality score through a fully connected layer. The present invention can improve the generalization ability, robustness, and accuracy of the model in virtual viewpoint image quality assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual viewpoint image quality evaluation, and in particular to a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion. Background Art

[0002] The emergence of immersive video technologies, such as multi-viewpoint video, free-viewpoint video, and multi-degree-of-freedom (DoF) video (3Dof, 3Dof+, and 6Dof), has greatly enhanced users' sense of immersion and interactive experience. Immersive video aims to provide users with an immersive visual experience, allowing them to freely explore every corner of the video through panoramic perspectives or virtual reality (VR) devices. However, transmitting immersive videos composed of a large number of viewpoints is limited by network bandwidth. Therefore, multiple viewpoints in immersive video systems need to be compressed. To reduce the number of viewpoints transmitted, viewpoint synthesis techniques are often used to generate new viewpoints. Depth Image Based Rendering (DIBR) technology collects texture images of some viewpoints and fuses them with depth images to synthesize virtual viewpoints, thereby reducing the expensive cost of capturing all viewpoints with a camera. However, during the virtual viewpoint rendering process, due to inaccurate depth map acquisition and imperfect rendering technology, the generated virtual viewpoint synthetic images will have some common distortion as well as geometric distortion different from traditional images, mainly manifested as texture distortion and structural distortion, which seriously affects people's visual experience. Therefore, how to effectively evaluate the quality of virtual viewpoint images has become a key issue in the development of virtual reality technology. It is crucial to establish a set of objective evaluation methods for virtual viewpoint image quality that can accurately reflect human visual perception.

[0003] Currently, a series of quality assessment methods for virtual viewpoint images have been proposed. They are mainly divided into full-reference (FR) image quality assessment (IQA) methods and no-reference (NR) IQA methods, depending on whether the original image information is referenced. FR methods require reference image information to evaluate distorted virtual viewpoint images. These methods perform well for specific geometric distortions but poorly for other distortions. However, reference images for virtual viewpoint images are difficult to obtain in practical applications. Researchers mainly use NR methods to evaluate the quality of virtual viewpoint images, which can complete image quality prediction using only the image's own information.

[0004] Currently, there are two main types of virtual viewpoint IQA methods: manual feature extraction-based virtual viewpoint IQA methods and deep learning-based virtual viewpoint IQA methods. Manual feature extraction models are limited by the complex distortions present in images and are insufficient to measure the diversity of image features. The features extracted by deep learning models are more similar to human visual perception, and can integrate feature learning into network training, outputting richer image texture and structural information through a multi-layer network structure. Some existing NR-IQA methods are inspired by the free energy principle and use generative adversarial networks (GANs) to restore distorted images. The quality of the image is measured by the visual perception difference between the distorted image and its restored image. However, such methods are limited by the small number of image samples and the complexity of image distortion, resulting in a large amount of false information in the restored images they generate. Therefore, using only a visual restoration network cannot effectively predict the quality of the synthesized image.

[0005] The applicants discovered that combining a visual restoration network with other feature extraction network tasks can more fully extract multi-layer features from virtual viewpoint images. Therefore, further exploring methods for evaluating NR virtual viewpoint image quality based on multi-task deep learning, based on restoring undistorted virtual viewpoint synthetic images using a visual restoration network, is an urgent challenge for those skilled in the art. Summary of the Invention

[0006] In view of the shortcomings of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion, extracting multi-scale feature maps respectively through a pre-trained visual residual restoration network model and an EfficientNet-B0 network model, and then adaptively fusing the multi-scale feature maps extracted by the two networks, thereby improving the generalization ability, robustness and accuracy of the model in virtual viewpoint image quality evaluation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion includes:

[0009] S1: Obtain the virtual viewpoint image to be evaluated;

[0010] S2: The virtual viewpoint image is input into a pre-trained visual residual restoration network model based on the U-Net network. The encoding and decoding ends of the model respectively extract multi-level degradation features and multi-level restoration features to generate multi-scale degradation feature maps and restoration feature maps.

[0011] S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map;

[0012] S4: Adaptively fuse the multi-scale degradation feature map, restoration feature map, and content feature map through the multi-feature weight adaptive fusion module to generate a multi-layer fused feature map;

[0013] S5: Use the channel attention module based on channel-level convolution to perform cross-channel information interaction on the multi-layer fusion feature maps to generate a fused channel-level feature map;

[0014] S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through the fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

[0015] Preferably, in step S2, the visual residual restoration network model includes an encoder and a decoder;

[0016] The encoder consists of a 3×3 convolutional layer and several encoding ends of different scales connected end to end. Each encoding end includes a parallel asymmetric residual network of the corresponding scale. The parallel asymmetric residual network is used to capture spatial information during the forward propagation process.

[0017] The decoder consists of several decoders connected end-to-end and corresponding to the scale of the encoder, and a 3×3 convolutional layer. A decoder consists of a 3×3 deconvolutional layer and a parallel asymmetric residual network of the corresponding scale. The output of each encoder is jump-connected to the decoder of the corresponding scale.

[0018] The feature maps output by the encoding end of all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoding end of all scales are used as multi-scale multi-level restoration feature maps.

[0019] Preferably, the parallel asymmetric residual network includes a first group of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second group of parallel convolutions, a second batch normalization layer, and a second PReLU layer connected end to end in sequence; wherein the input feature map of the parallel asymmetric residual network is subjected to dimensionality reduction through a 1×1 convolution layer, and then added to the output of the second batch normalization layer and input into the second PReLU layer;

[0020] The first and second groups of parallel convolutions input the input feature map to the parallel 1×3 horizontal convolution layer, 3×1 vertical convolution layer, and 3×3 convolution layer respectively, and then perform element-by-element addition on the outputs of the 1×3 horizontal convolution layer, 3×1 vertical convolution layer, and 3×3 convolution layer to generate the output feature map;

[0021] The calculation formula of the parallel asymmetric residual network is expressed as:

[0022]

[0023] Where: R, I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; O1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents 1×1 convolution; C3 represents 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolution respectively; Represents element-by-element addition operation; f BN and f ReLU Represent the batch normalization operation and PReLU activation operation respectively.

[0024] Preferably, in step S3, the feature maps output by multiple Stage layers of the EfficientNet-B0 network model are used as multi-scale multi-level content feature maps.

[0025] Preferably, in step S4, the processing steps of the multi-feature weight adaptive fusion module include:

[0026] S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map;

[0027] S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map;

[0028] S403: Performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map;

[0029] S404: Concatenate the multi-scale enhanced restored feature map and the weighted adaptive fusion feature map of the corresponding scale and perform a global average pooling operation to obtain a multi-scale fusion feature map; cascade the multi-scale fusion feature map to generate a final multi-layer fusion feature map;

[0030] The calculation formula of the multi-layer fusion feature map is expressed as:

[0031]

[0032] Where: F fusionrepresents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1,2,3,4) scale; F2(i) represents the enhanced recovery feature map of the i-th (i=1,2,3,4) scale; GAP represents the global average pooling operation; CAT and Both represent cascade operations.

[0033] Preferably, in step S401, the feature enhancement module inputs its input feature map into three branches arranged in parallel, connects the outputs of the three branches, performs batch normalization and PReLU activation operations, and outputs the corresponding enhanced feature map; the first branch includes a 1×1 convolutional layer; the second branch includes a 3×3 convolutional layer; the third branch includes two 3×3 convolutional layers connected end to end;

[0034] The calculation formula of the enhanced degradation feature map is expressed as:

[0035]

[0036] The calculation formula of the enhanced content feature map is expressed as:

[0037]

[0038] Where: E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) I C (i) represents the degradation feature map and content feature map of the i-th (i=1, 2, 3, 4) scale of the input respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; Indicates a cascade operation.

[0039] Preferably, in step S402, the weight adaptive generation module concatenates the input enhanced degradation feature map and the enhanced content feature map, and then generates weight maps of the two modalities using two separate 3×3 convolutions and sigmoid functions; multiplies the two weight maps with the enhanced degradation feature map and the enhanced content feature map respectively to obtain corresponding weighted degradation feature maps and weighted content feature maps; and concatenates the weighted degradation feature map and the weighted content feature map to generate a weight adaptive fusion feature map;

[0040] The calculation formula of weight adaptive fusion feature is expressed as:

[0041]

[0042] Where: F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1, 2, 3, 4) scale; E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F C (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale (i=1, 2, 3, 4), σ represents the sigmoid function, and ⊙ represents pixel-by-pixel multiplication. Indicates a cascade operation.

[0043] Preferably, in step S403, the calculation formula of the enhanced restored feature map is expressed as:

[0044]

[0045] Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th (i=1, 2, 3, 4) scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates a cascade operation.

[0046] Preferably, in step S5, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize spatial information interaction between channels, and obtain a weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain a fusion channel-level feature map;

[0047] The calculation formula for fusion channel-level features is expressed as:

[0048] F=F fusion ☉σ(Conv 3×3 (GAP(F fusion )));

[0049] Where: F represents the fusion channel-level feature map; F fusion Represents multi-layer fusion feature map; GAP represents the global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

[0050] Preferably, in step S6, the fully connected layer generates a corresponding predicted objective quality score after fusing the channel-level feature map through a three-layer fully connected neural network and two PReLU activation functions.

[0051] Compared with the prior art, the virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion in the present invention has the following beneficial effects:

[0052] The present invention extracts multi-scale feature maps using a pre-trained visual residual restoration network model and an EfficientNet-B0 network model. On the one hand, the visual residual restoration network follows the free energy principle, reflecting the degree of degradation of the distorted image through the degradation features of different layers at the encoding end, and reflecting the degree of restoration of the distorted image using the gradually restored restored image features. By simulating the brain's repair of distorted images, it effectively extracts multi-level degradation features and multi-level restoration features of the image. On the other hand, the EfficientNet-B0 network is trained on a large sample database to learn the traditional distortions present in ordinary images to extract multi-level content features of virtual viewpoint images, enabling the network to extract more comprehensive distortion information. The visual residual restoration network model focuses primarily on synthetic image distortion, while the EfficientNet-B0 network model is more sensitive to ordinary distortion. While the multi-scale feature maps extracted by the two networks differ significantly, they can also complement each other. By fusing the multi-scale feature maps extracted by the two networks, it can cope with complex geometric distortions in virtual viewpoint images, thereby improving the model's generalization and robustness in virtual viewpoint image quality assessment.

[0053] In response to the differences between the feature maps extracted by the pre-trained visual residual restoration network model and the EfficientNet-B0 network model, this paper proposes a multi-feature weight adaptive fusion module to adaptively fuse multi-scale degraded feature maps, restored feature maps, and content feature maps. The multi-feature weight adaptive fusion module can adaptively adjust the weights of the multi-layer, multi-scale features extracted from the two different networks, assigning optimal weights to the two significantly different feature types. By adaptively adjusting the weights of different distortions, the fused features of the distorted image are fused with the multi-level restored features at the decoding end to achieve better model results, thereby improving the model's accuracy in virtual viewpoint image quality assessment.

[0054] To further enhance the information exchange between channels, this paper proposes a channel attention module to enable information exchange between different channels. By acquiring channel-level features that are more relevant to image quality, the quality prediction network can obtain features that are more relevant to image quality, thereby improving the accuracy of virtual viewpoint image quality assessment. Experiments show that compared with existing IQA methods, our method achieves optimal performance on two virtual viewpoint image databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0056] Figure 1 This is a logical block diagram of a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion;

[0057] Figure 2 This is the structural diagram of the parallel asymmetric residual network;

[0058] Figure 3 This is the structural diagram of the multi-feature weight adaptive fusion module;

[0059] Figure 4 This is the structural diagram of the channel attention module based on channel-level convolution;

[0060] Figure 5 It is a multi-scale degradation feature map and content feature map;

[0061] Figure 6 is a multi-scale restored feature map. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] The following is a further detailed description through specific implementation methods:

[0064] Example:

[0065] This embodiment discloses a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion.

[0066] like Figure 1 As shown in FIG, a virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion includes:

[0067] S1: Obtain the virtual viewpoint image to be evaluated;

[0068] S2: The virtual viewpoint image is input into a pre-trained visual residual restoration network model based on the U-Net network. The encoding and decoding ends of the model respectively extract multi-level degradation features and multi-level restoration features to generate multi-scale degradation feature maps and restoration feature maps.

[0069] S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map;

[0070] S4: The multi-feature weight adaptive fusion block (MWFB) adaptively fuses the multi-scale degradation feature map, restoration feature map, and content feature map to generate a multi-layer fusion feature map.

[0071] S5: Use the channel attention module based on channel-level convolution to perform cross-channel information interaction on the multi-layer fusion feature maps to generate a fused channel-level feature map;

[0072] S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through the fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

[0073] The present invention extracts multi-scale feature maps using a pre-trained visual residual restoration network model and an EfficientNet-B0 network model. On the one hand, the visual residual restoration network follows the free energy principle, reflecting the degree of degradation of the distorted image through the degradation features of different layers at the encoding end, and reflecting the degree of restoration of the distorted image using the gradually restored restored image features. By simulating the brain's repair of distorted images, it effectively extracts multi-level degradation features and multi-level restoration features of the image. On the other hand, the EfficientNet-B0 network is trained on a large sample database to learn the traditional distortions present in ordinary images to extract multi-level content features of virtual viewpoint images, enabling the network to extract more comprehensive distortion information. The visual residual restoration network model focuses primarily on synthetic image distortion, while the EfficientNet-B0 network model is more sensitive to ordinary distortion. While the multi-scale feature maps extracted by the two networks differ significantly, they can also complement each other. By fusing the multi-scale feature maps extracted by the two networks, it can cope with complex geometric distortions in virtual viewpoint images, thereby improving the model's generalization and robustness in virtual viewpoint image quality assessment.

[0074] In response to the differences between the feature maps extracted by the pre-trained visual residual restoration network model and the EfficientNet-B0 network model, this paper proposes a multi-feature weight adaptive fusion module to adaptively fuse multi-scale degraded feature maps, restored feature maps, and content feature maps. The multi-feature weight adaptive fusion module can adaptively adjust the weights of the multi-layer, multi-scale features extracted from the two different networks, assigning optimal weights to the two significantly different feature types. By adaptively adjusting the weights of different distortions, the fused features of the distorted image are fused with the multi-level restored features at the decoding end to achieve better model results, thereby improving the model's accuracy in virtual viewpoint image quality assessment.

[0075] To further enhance the information exchange between channels, this paper proposes a channel attention module to enable information exchange between different channels. By acquiring channel-level features that are more relevant to image quality, the quality prediction network can obtain features that are more relevant to image quality, thereby improving the accuracy of virtual viewpoint image quality assessment. Experiments show that compared with existing IQA methods, our method achieves optimal performance on two virtual viewpoint image databases.

[0076] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0077] 1. Visual Residual Restoration Network Model

[0078] In this embodiment, the visual residual restoration network model is constructed based on the U-Net network. In order to solve the problem of a large amount of pseudo-information when restoring distorted images using the improved U-Net network, this embodiment is based on the free energy principle, that is, when the human visual system (HVS) quantifies the distorted virtual viewpoint image, it usually needs to compare it with the original undistorted image in the mind to perceive the quality difference. A visual residual restoration network is designed, which uses the improved U-Net network to restore the distorted image to the undistorted image. Its overall structure is as follows: Figure 1 As shown in (a).

[0079] Specifically, the visual residual restoration network model includes an encoder and a decoder; the encoder includes a 3×3 convolutional layer and several encoding ends connected end to end in sequence and with different scales, and one encoding end includes a parallel asymmetric residual network of the corresponding scale; wherein the parallel asymmetric residual network is used to capture spatial information during the forward propagation process; the decoder includes several decoding ends connected end to end in sequence and corresponding to the scales of the encoding end and a 3×3 convolutional layer, and one decoding end includes a 3×3 deconvolution layer and a parallel asymmetric residual network of the corresponding scale; wherein the output of each encoding end is jump-connected to the decoding end of the corresponding scale; the feature maps output by the encoding ends of all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoding ends of all scales are used as multi-scale multi-level restoration feature maps.

[0080] When using a visual residual restoration network for image feature extraction, it is necessary to preserve the important edge texture structure in the image. Ordinary convolutional layers cannot extract correct features after image preprocessing. Therefore, this embodiment designs a parallel asymmetric residual network, which replaces each ordinary 3×3 convolution with a parallel convolution of a 1×3 horizontal convolution kernel, a 3×1 vertical convolution kernel, and a 3×3 convolution kernel, improving the robustness of the entire model in extracting features.

[0081] Combine Figure 2 As shown, the parallel asymmetric residual network includes a first group of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second group of parallel convolutions, a second batch normalization layer, and a second PReLU layer connected end to end in sequence; wherein, the input feature map of the parallel asymmetric residual network is reduced in dimension by a 1×1 convolution layer, and then added to the output of the second batch normalization layer and input into the second PReLU layer; the first and second groups of parallel convolutions input the input feature map into a 1×3 horizontal convolution layer, a 3×1 vertical convolution layer, and a 3×3 convolution layer set in parallel, respectively, and then the outputs of the 1×3 horizontal convolution layer, the 3×1 vertical convolution layer, and the 3×3 convolution layer are element-wise added to generate an output feature map;

[0082] The calculation formula of the parallel asymmetric residual network is expressed as:

[0083]

[0084] Where: R, I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; P1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents 1×1 convolution; C3 represents 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolution respectively; Represents element-by-element addition operation; f BN and f ReLU Represent the batch normalization operation and PReLU activation operation respectively.

[0085] The hierarchical degradation characteristics of the visual residual restoration network have the same characteristics as the perception mechanism of the human visual system (HVS). Therefore, this embodiment reflects the degradation degree of the distorted image by extracting the degradation features of different layers of the visual residual restoration network encoding end, and extracts the feature maps output by the encoding end D1, D2, D3 and D4 as multi-scale multi-level degradation feature maps. The decoding end of the visual residual restoration network obtains a reconstructed high-quality image by jump-connecting deconvolution with low-quality image features. The decoding ends R1 to R4 of the decoding end are the process of gradual image restoration, and their feature maps are semantically closer and closer to high-quality images. The gradually restored restored image features can be used to reflect the degree of restoration of the distorted image, thereby better learning the relationship between virtual viewpoint image features and human eye quality perception. The feature maps output by the decoding ends R1, R2, R3, and R4 are extracted as multi-scale multi-level restoration feature maps.

[0086] Due to the insufficient number of samples in the virtual viewpoint image database, it is difficult to train a good restoration network model. Therefore, this embodiment pre-trains the visual residual restoration network on a general image dataset, allowing it to learn richer visual scenes, thereby improving the visual restoration ability of the network model. During pre-training, the resolution of the input image is adjusted to 256×256 and then randomly cropped into 224×224 pixel blocks. The initial learning rate is set to 1×10-3, and the network weights are updated for 500 epochs using the Adam optimizer, ultimately obtaining the pre-trained weights of the visual residual restoration network.

[0087] 2. EfficientNet-B0 Network Model

[0088] In this embodiment, the structure of the EfficientNet-B0 network model is shown in the table below. It includes 16 moving-flip bottleneck convolution (MBConv) modules, 2 convolutional layers, 1 global average pooling layer, and 1 classification layer. Pre-training of the EfficientNet-B0 network model is achieved using existing methods.

[0089] Specifically, the feature maps output by the 2nd, 3rd, 4th, and 6th Stage layers of the EfficientNet-B0 network model are extracted as multi-scale multi-level content (perception) feature maps.

[0090]

[0091] Figure 5 The multi-scale degradation feature maps and content feature maps extracted by the visual residual recovery network model and the EfficientNet-B0 network model are Figure 5 (a) is the input distorted virtual viewpoint image, Figure 5 (b)-(e) are multi-scale degradation feature maps. Figure 5 (f)-(i) are multi-scale content feature maps. It can be seen that the visual residual restoration network focuses on synthetic image distortion, while the EfficientNet-B0 network model is more sensitive to general distortion. While the multi-scale feature maps extracted by the two networks differ significantly, they also complement each other.

[0092] Figure 6 The figure shows the process of restoring the distorted virtual viewpoint image through the visual residual restoration network. Figure 6 (a) The input is a distorted virtual viewpoint image. Figure 6 (b)-(e) are multi-scale restored feature maps. It can be seen that the quality of the feature maps from R1 to R4 layers is getting closer and closer to the original image, which intuitively reflects the restoration process of the distorted image.

[0093] 3. Multi-feature weight adaptive fusion module (MWFB)

[0094] In order to solve the problem of fusing different features extracted by multi-task networks, this embodiment takes into account the large differences between the multi-scale feature maps extracted by the two networks while also being able to complement each other. Therefore, different weights can be assigned to the distortion features according to their importance, and a multi-feature weight adaptive fusion module is designed. The specific structure of the module is as follows: Figure 3 As shown in Figure 2, the module consists of two feature enhancement blocks (FEB) and a weight adaptive generation block (WAGB).

[0095] In this embodiment, the processing steps of the multi-feature weight adaptive fusion module include:

[0096] S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map;

[0097] S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map;

[0098] S403: Performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map;

[0099] S404: Concatenate the multi-scale enhanced restored feature map with the weighted adaptive fusion feature map of the corresponding scale and perform a global average pooling (GAP) operation to obtain a multi-scale fusion feature map; perform a cascade operation on the multi-scale fusion feature map to generate a final multi-layer fusion feature map;

[0100] The calculation formula of the multi-layer fusion feature map is expressed as:

[0101]

[0102] Where: F fusion represents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1,2,3,4) scale; F2(i) represents the enhanced recovery feature map of the i-th (i=1,2,3,4) scale; GAP represents the global average pooling operation; CAT and Both represent cascade operations.

[0103] 1. Feature Enhancement Module (FEB)

[0104] The feature enhancement module consists of 1×1, 3×3, and 5×5 convolutional layers connected in parallel. To reduce the computational complexity of the network while maintaining the same receptive field as the 5×5 convolutional layer, two 3×3 convolutions are used instead of the 5×5 convolution. The feature enhancement module feeds its input feature map into three parallel branches. The outputs of the three branches are concatenated and then batch normalized and activated with Pre-ReLU (Pre-ReLU) to output the corresponding enhanced feature map. The first branch consists of a 1×1 convolutional layer; the second branch consists of a 3×3 convolutional layer; and the third branch consists of two 3×3 convolutional layers connected end-to-end. The degradation feature map and the content feature map are passed through FEB to obtain two enhanced features.

[0105] The calculation formula of the enhanced degradation feature map is expressed as:

[0106]

[0107] The calculation formula of the enhanced content feature map is expressed as:

[0108]

[0109] Where: E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) I C (i) represents the degradation feature map and content feature map of the i-th (i=1, 2, 3, 4) scale of the input respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; Indicates a cascade operation.

[0110] In this embodiment, the calculation formula of the enhanced restoration feature map is expressed as:

[0111]

[0112] Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th (i=1, 2, 3, 4) scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates a cascade operation.

[0113] 2. Weight Adaptive Generation Module (WAGB)

[0114] The weight adaptive generation module concatenates the input enhanced degradation feature map and enhanced content feature map, and then generates weight maps of the two modalities using two separate 3×3 convolutions and sigmoid functions. The two weight maps are multiplied by the enhanced degradation feature map and the enhanced content feature map respectively to obtain the corresponding weighted degradation feature map and weighted content feature map. The weighted degradation feature map and the weighted content feature map are concatenated to generate a weight adaptive fusion feature map.

[0115] The calculation formula of weight adaptive fusion feature is expressed as:

[0116]

[0117] Where: F1(i) represents the weighted adaptive fusion feature map of the i-th (i=1, 2, 3, 4) scale; E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F C (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale (i=1, 2, 3, 4), σ represents the sigmoid function, and ⊙ represents pixel-by-pixel multiplication. Indicates a cascade operation.

[0118] 4. Channel Attention Module

[0119] In order to avoid interference of some useless information in the fusion features on the quality prediction network, this embodiment designs a channel attention mechanism based on channel-level convolution to achieve cross-channel information interaction to obtain channel-level features that are more relevant to image quality. Figure 4 As shown in the figure, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize the spatial information interaction between each channel and obtain the weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain the fusion channel-level feature map;

[0120] The calculation formula for fusion channel-level features is expressed as:

[0121] F=F fusion ☉σ(Conv 3×3 (GAP(F fusion )));

[0122] Where: F represents the fusion channel-level feature map; F fusion Represents multi-layer fusion feature map; GAP represents the global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

[0123] 5. Fully Connected Layer

[0124] In this embodiment, the fully connected layer generates the corresponding predicted objective quality score after fusing the channel-level feature map through a three-layer fully connected neural network and two PReLU activation functions.

[0125] 6. Experimental Description

[0126] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiments.

[0127] 1. Experimental description

[0128] The performance of this experiment on 84 virtual viewpoint image samples from the IRCCyN / IVC database is compared and analyzed with that of other IQA methods, and the advanced nature of the present invention is further demonstrated through a series of experiments.

[0129] This experiment was implemented in the Pytorch1.10 deep learning framework and python3.8 environment, and the model was trained and tested on the NVIDIA3080GPU. When training the entire network, the insufficient number of samples in the virtual viewpoint image database and the different image sizes in the same database were taken into consideration. Therefore, when training the entire network, the input image was first resized to 768×1024 pixels, and then randomly cropped into 25 224×224 pixel blocks, where the subjective quality score of each pixel block was provided by the original distorted image. In addition, the images in each database were randomly grouped, with 80% of the images used as training sets and 20% of the images used as test sets. The entire network used Adam to optimize the network, and the learning rate was set to 2×10-4 for a total of 50 epochs. In order to accurately measure the mapping relationship between the predicted score and the subjective score, the L loss function was used to fine-tune the training set, which is defined as:

[0130]

[0131] Among them, N is the number of blocks of virtual viewpoint distorted image during batch processing, Q i is the prediction score of the i-th image block, DMOS i is the subjective score of the i-th image block, and l1 represents the l1 norm.

[0132] 2. Performance Evaluation

[0133] To accurately evaluate the performance of the entire model, this experiment used three commonly used evaluation metrics for objective quality evaluation methods proposed by a video expert panel: the Pearson Linear Correlation Coefficient (PLCC), the Spearman Rank Correlation Coefficient (SROCC), and the Root Mean Square Error (RMSE). The PLCC represents the correlation between the subjective score and the predicted score; a higher PLCC indicates a higher correlation. The SROCC reflects the monotonic correlation between the subjective score and the predicted score; a higher SROCC indicates better model performance. The RMSE represents the absolute error between the predicted score and the subjective score; a smaller RMSE indicates greater model accuracy and stability. These three evaluation metrics are used to measure the consistency between the objective quality score and the subjective score, effectively judging the quality of the objective quality evaluation method.

[0134] To demonstrate the effectiveness of our method, Table 1 compares the performance of our method with 17 other IQA methods on the IRCCyN / IVC virtual viewpoint image database, with the best methods indicated in bold. These methods include three quality assessment methods designed for natural images: PSNR, SSIM, and BIQI. In addition, 14 quality assessment methods designed for virtual viewpoint images are included: MP-PSNR, SSPD, SEQUSS, PU-IR & DF-CS, Zhang's, APT, NIQSV, NIQSV+, MNSS, Wang's, Fang's, Sadbhawna's, GANs-NQM, and Jakhetiya's. The three performance metrics above were used to evaluate the performance of all methods.

[0135] As shown in Table 1, compared with other advanced IQA methods, the proposed method achieves the best performance and surpasses all FR IQA methods. This shows that the deep learning-based virtual viewpoint image quality assessment method can achieve better performance than traditional methods. Furthermore, the SSPD, Wang's, and Fang's methods, which simultaneously consider image texture and structural distortion, achieve the best performance among all traditional methods, but their performance is still lower than that of deep learning methods. The proposed method achieves PLCC, SROCC, and RMSE of 0.9063, 0.9283, and 0.2956, respectively, further demonstrating its broad applicability.

[0136] Table 1 Performance of the present invention and other methods on the IRCCyN / IVC virtual viewpoint image database

[0137]

[0138]

[0139] 3. Ratio of training and test sets

[0140] To verify the impact of the training-test sample ratio on the performance of the proposed method, we randomly selected 50%, 60%, 70%, 80%, and 90% of the images from the IRCCyN / IVC database as training samples, and the remaining images as test samples for separate experiments. Table 2 lists the performance results of training with different training set ratios. As can be seen in Table 2, when the training set ratio increases, the PLCC and SROCC performance indicators also increase and tend to stabilize. Even with a training set ratio of only 50%, the proposed method's SROCC reached 0.8207, which is still superior to most virtual viewpoint image quality evaluation methods.

[0141] Table 2 Performance results of the methods under different training set ratios

[0142]

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the technical solutions. Those skilled in the art should understand that modifications or equivalent replacements of the technical solutions of the present invention that do not depart from the purpose and scope of the technical solutions of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A virtual viewpoint image quality evaluation method based on residual recovery and multi-feature fusion, characterized by: include: S1: Obtain the virtual viewpoint image to be evaluated; S2: The virtual viewpoint image is input into a pre-trained visual residual restoration network model based on the U-Net network. The encoding and decoding ends of the model respectively extract multi-level degradation features and multi-level restoration features to generate multi-scale degradation feature maps and restoration feature maps. In step S2, the visual residual restoration network model includes an encoder and a decoder; The encoder consists of a 3×3 convolutional layer and several encoding ends of different scales connected end to end. Each encoding end includes a parallel asymmetric residual network of the corresponding scale. The parallel asymmetric residual network is used to capture spatial information during the forward propagation process. The decoder consists of several decoders connected end-to-end and corresponding to the scale of the encoder, and a 3×3 convolutional layer. A decoder consists of a 3×3 deconvolutional layer and a parallel asymmetric residual network of the corresponding scale. The output of each encoder is jump-connected to the decoder of the corresponding scale. The feature maps output by the encoder at all scales are used as multi-scale multi-level degradation feature maps; the features output by the decoder at all scales are used as multi-scale multi-level restoration feature maps; The parallel asymmetric residual network includes a first set of parallel convolutions, a first batch normalization layer, a first PReLU layer, a second set of parallel convolutions, a second batch normalization layer, and a second PReLU layer, which are connected end to end. The input feature map of the parallel asymmetric residual network is reduced in dimension by a 1×1 convolution layer, and then added to the output of the second batch normalization layer before being input into the second PReLU layer. The first and second groups of parallel convolutions input the input feature map to the parallel 1×3 horizontal convolution layer, 3×1 vertical convolution layer, and 3×3 convolution layer respectively, and then perform element-by-element addition on the outputs of the 1×3 horizontal convolution layer, 3×1 vertical convolution layer, and 3×3 convolution layer to generate the output feature map; The calculation formula of the parallel asymmetric residual network is expressed as: Where: R, I represent the output feature map and input feature map of the parallel asymmetric residual network respectively; P1 and P2 represent the outputs of two sets of parallel convolutions; C1 represents 1×1 convolution; C3 represents 3×3 convolution with a step size of 1; C 1×3 and C 3×1 Represent 1×3 and 3×1 convolution respectively; Represents element-by-element addition operation; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; S3: Input the virtual viewpoint image into the pre-trained EfficientNet-B0 network model, extract multi-level content features through the model, and generate a multi-scale content feature map; S4: Adaptively fuse the multi-scale degradation feature map, restoration feature map, and content feature map through the multi-feature weight adaptive fusion module to generate a multi-layer fused feature map; S5: Use the channel attention module based on channel-level convolution to perform cross-channel information interaction on the multi-layer fusion feature maps to generate a fused channel-level feature map; S6: Map the fused channel-level feature map to the corresponding predicted objective quality score through the fully connected layer, and output it as the image quality evaluation result of the virtual viewpoint image.

2. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S3, the feature maps output by multiple Stage layers of the EfficientNet-B0 network model are used as multi-scale multi-level content feature maps.

3. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S4, the processing steps of the multi-feature weight adaptive fusion module include: S401: performing feature enhancement on the multi-scale degradation feature map and the restoration feature map respectively through a feature enhancement module to generate a multi-scale enhanced degradation feature map and an enhanced content feature map; S402: assigning different weights to the degradation feature map and the content feature map according to the importance of the features in the weight adaptive generation module; weighting the multi-scale enhanced degradation feature map and the enhanced content feature map according to the assigned weights to generate a multi-scale weight adaptive fusion feature map; S403: Performing feature enhancement on the multi-scale restored feature map through a feature enhancement module to generate a multi-scale enhanced restored feature map; S404: Concatenate the multi-scale enhanced restored feature map and the weighted adaptive fusion feature map of the corresponding scale and perform a global average pooling operation to obtain a multi-scale fusion feature map; cascade the multi-scale fusion feature map to generate a final multi-layer fusion feature map; The calculation formula of the multi-layer fusion feature map is expressed as: Where: F fusion Represents a multi-layer fusion feature map; F1(i) represents the weighted adaptive fusion feature map of the i-th scale; F2(i) represents the enhanced recovery feature map of the i-th scale; GAP represents the global average pooling operation; CAT and All represent cascade operations; i=1,2,3,4.

4. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 3, characterized in that: In step S401, the feature enhancement module inputs its input feature map into three parallel branches, concatenates the outputs of the three branches, performs batch normalization and PReLU activation, and outputs the corresponding enhanced feature map; the first branch includes a 1×1 convolutional layer; the second branch includes a 3×3 convolutional layer; and the third branch includes two 3×3 convolutional layers connected end to end; The calculation formula of the enhanced degradation feature map is expressed as: The calculation formula of the enhanced content feature map is expressed as: Where: E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; I D (i) I C (i) represents the degradation feature map and content feature map of the input at the i-th scale respectively; represents a 1×1 convolution with a stride of 2; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; represents a 1×1 convolution with a stride of 8; represents a 3×3 convolution with a stride of 8; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; Indicates cascade operation; i = 1, 2, 3, 4.

5. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 3, characterized in that: In step S402, the weight adaptive generation module concatenates the input enhanced degradation feature map and enhanced content feature map, and then generates weight maps of the two modalities using two separate 3×3 convolutions and sigmoid functions; multiplies the two weight maps with the enhanced degradation feature map and the enhanced content feature map respectively to obtain corresponding weighted degradation feature map and weighted content feature map; and concatenates the weighted degradation feature map and the weighted content feature map to generate a weight adaptive fusion feature map. The calculation formula of weight adaptive fusion feature is expressed as: Where: F1(i) represents the weighted adaptive fusion feature map of the i-th scale; E i (f D ), E i (f C ) represent the enhanced degradation feature map and the enhanced content feature map respectively; E i (f) represents the enhanced feature map after the enhanced degradation feature map and the enhanced content feature map are cascaded; C3 represents a 3×3 convolution with a step size of 1; F D (i) and F c (i) represents the weighted degradation feature map and weighted content feature map of the i-th scale respectively; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication; Indicates cascade operation; i = 1, 2, 3, 4.

6. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 3, characterized in that: In step S403, the calculation formula of the enhanced restored feature map is expressed as: Where: F2(i) represents the enhanced restoration feature map; I R (i) represents the restored feature map of the i-th scale of the input; represents a 3×3 convolution with a stride of 2; C3 represents a 3×3 convolution with a stride of 1; f BN and f ReLU Represent batch normalization operation and PReLU activation operation respectively; represents a 1×1 convolution with a stride of 2; Indicates cascade operation; i = 1, 2, 3, 4.

7. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S5, the channel attention module based on channel-level convolution performs a global average pooling operation on the input multi-layer fusion feature map, and then performs a 3×3 one-dimensional convolution and a sigmoid function to realize the spatial information interaction between each channel, and obtain the weight value of each channel; the weight value is multiplied by the input fusion feature map to obtain the fusion channel-level feature map; The calculation formula for fusion channel-level features is expressed as: F=F fusion ⊙σ(Conv 3×3 (Conv(F fusion ))); Where: F represents the fusion channel-level feature map; F fusion Represents multi-layer fusion feature map; GAP represents the global average pooling operation; Conv 3×3 represents a 1-dimensional 3×3 convolution operation; σ represents the sigmoid function; ⊙ represents pixel-by-pixel multiplication.

8. The virtual viewpoint image quality assessment method based on residual recovery and multi-feature fusion according to claim 1, characterized in that: In step S6, the fully connected layer generates the corresponding predicted objective quality score after the fused channel-level feature map passes through a three-layer fully connected neural network and two PReLU activation functions.

Images