A virtual view quality enhancement method based on asymmetric streaming

Abstract

The present invention provides a method for enhancing virtual view quality based on asymmetric streaming, comprising the following steps: S1: extracting shallow features from a low-quality input virtual view; S2: inputting the shallow features into a hybrid codec to convert them into deep features; S3: adaptively fusing the features output by each MSACB module at the decoding end of the hybrid codec; and S4: reconstructing a quality-enhanced virtual view based on the low-quality virtual view and the fused features. This method, based on asymmetric streaming, addresses the current problems of compression distortion and DIBR distortion in 3D video transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual view processing, and more particularly, to a virtual view quality enhancement method based on asymmetric streaming. Background Art

[0002] In recent years, information technology has advanced rapidly, and multimedia technologies have made significant progress. Applications using audio, images, and video as media have become an integral part of people's lives. The emergence of virtual reality (VR), based on immersive media, has injected new vitality into the multimedia field. VR-related technologies have become the core of the development of the next generation of digital media technologies. Furthermore, the advent of the 5G era has further promoted the implementation of immersive media-related applications. Consequently, extensive research has been conducted on immersive media. Among these, generating more immersive video content and enhancing the user's sense of presence are pressing challenges in immersive media and are a hot topic of concern in both academia and industry.

[0003] Among them, three-dimensional video (3DV) technology is widely used in various applications, such as 3D movies and games, and military simulation training, becoming a cutting-edge research hotspot in fields such as image signal processing and computer vision. With the popularity of the concept of the metaverse, 3D video technology that can provide viewers with an immersive and passive experience has regained public attention. Typically, before a client user views a 3D video, the transmission of 3D video includes multiple consecutive stages, such as acquisition, compression, transmission, rendering, and display. Currently, depth image-based rendering (DIBR) technology is often introduced into the rendering stage to synthesize virtual views from decoded existing texture and depth image views to provide users with an arbitrary viewpoint viewing experience. These stages will introduce several distortions, especially compression distortion and DIBR distortion in the compression and rendering stages, which greatly affect the visual quality of the synthesized view and reduce the user's visual experience in 3D video applications. Summary of the Invention

[0004] The present invention aims to overcome the technical defects of compression distortion and DIBR distortion in the transmission stage of current 3D videos and provides a virtual view quality enhancement method based on asymmetric streaming.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A method for enhancing virtual view quality based on asymmetric streaming includes the following steps:

[0007] S1: shallow feature extraction of low-quality input virtual views;

[0008] S2: Input shallow features into the hybrid encoder-decoder to convert them into deep features;

[0009] The hybrid codec has a U-shaped network structure, including multiple MSACB modules, each of which includes multiple DHAB modules. In the DHAB module, shallow features are asymmetrically divided into multi-head self-attention branches and CNN branches to extract global information and local information respectively;

[0010] S3: Adaptively fuse the features output by each MSACB module at the decoding end of the hybrid codec;

[0011] S4: The quality-enhanced virtual view is reconstructed based on the low-quality virtual view and the fused features.

[0012] In this solution, shallow features of low-quality virtual views are first extracted. Then, the ability of multi-head self-attention to model global information and the ability of CNN to capture local texture information are combined. A hybrid codec with a U-shaped network structure is used to further learn multi-level hierarchical features, adaptively fuse information at different feature scales, and finally obtain a quality-enhanced virtual view based on the fused features. This can effectively eliminate compression distortion and DIBR distortion existing in the 3D video transmission stage and improve the quality enhancement efficiency of virtual views.

[0013] Preferably, the low-quality virtual view of the input is subjected to 3×3 convolution and ReLU nonlinear activation function to extract shallow features F e :

[0014]

[0015] Where σ(·) represents the ReLU nonlinear activation function; represents 3×3 convolution; I i Represents a low-quality virtual view of the input.

[0016] Preferably, the hybrid codec is a four-layer symmetrical U-type hybrid codec.

[0017] On the encoding side, each MSACB module performs feature downsampling through 3×3 convolution and Pixelunshuffle operations. On the decoding side, each MSACB module performs feature upsampling through 3×3 convolution and Pixelshuffle operations. The features output by the MSACB module on the encoding side are transmitted to the MSACB module on the decoding side through skip connections.

[0018] Deep feature F d =F u (F e );

[0019] Among them, Fu (·) represents the processing of the U-type hybrid codec.

[0020] Preferably, the process of asymmetrically dividing shallow features into the multi-head self-attention branch and the CNN branch is:

[0021]

[0022] Among them, the shallow feature F e Through 1×1 convolution Get feature F s ; F g Represents the features flowing into the multi-head self-attention branch; F l Represents the features flowing into the CNN branch; Represents the information flow division unit IFDU operation; α is the contribution factor, which is used to adjust the contribution ratio of the information flow flowing into the two branches. α gradually decreases during the feature downsampling process and gradually increases during the feature upsampling process; C g represents the number of channels flowing into the multi-head self-attention branch; C represents the number of channels flowing into the DHAB module; C l Indicates the number of channels flowing into the CNN branch.

[0023] Preferably, the process of extracting global information from the multi-head self-attention branch is:

[0024] Given F g , first through the layer normalization operation, and then use 1×1 convolution and 3×3 depth convolution in the MDTA module of the multi-head self-attention branch to encode the channel-level spatial context;

[0025] The process of the MDTA module is as follows:

[0026]

[0027] in, Represents dimension reshaping; represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; Represents the matrix multiplication operation; F MDTA represents the output of the MDTA module; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represents the query tensor feature; Represents the key tensor feature; Represents the value tensor feature;

[0028] Perform channel-wise multiplication:

[0029] X1=W1*F MDTA ⊕Fg

[0030] Where X1 represents the output feature after DLM and element-wise addition; ⊕ represents element-wise addition; W1 represents the trainable weight generated by the first DLM in the multi-head self-attention branch;

[0031] Replace the feedforward network with 1×1 convolution and simple gating, as shown below:

[0032]

[0033] Among them, Chunk(·) represents a tensor chunking operation; ⊙ represents an element-wise multiplication operation; Y1 represents the first branch feature output by the tensor chunking operation; Y2 represents the second branch feature output by the tensor chunking operation; Y1 and Y2 have the same feature dimension size; SG(·) represents a simple gating operation; W2 represents the trainable weight generated by the second DLM in the multi-head self-attention branch; Represents the output of the multi-head self-attention branch.

[0034] Preferably, the process of extracting local information by the CNN branch is:

[0035]

[0036] in, represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; represents the output of SG; SG(·) represents a simple gating operation; GAP(·) represents a global average pooling operation; W represents the trainable weight generated by DLM in the CNN branch; SCA(·) represents a simplified channel attention operation; Represents the output of the CNN branch GLRB.

[0037] Preferably, the process of the DHAB module is as follows:

[0038]

[0039] An MSACB module includes multiple DHAB modules, output F MSACB The process is shown as follows:

[0040]

[0041] in, Represents the output of the multi-head self-attention branch; represents the output of CNN branch GLRB; F g represents the information flowing into the multi-head self-attention branch; F l Represents the features flowing into the CNN branch; F GCCTB(·) represents the operation of the entire multi-head self-attention branch; F GLRB (·) represents the operation of the entire CNN branch; F DHAB represents the features output by the DHAB module; Concat(·) represents the concatenation operation; ⊕ represents the element addition operation; F MSACB Represents the characteristics of the MSACB module output; Represents the operation of the first DHAB module in the MSACB module; Represents the operation of the second DHAB module in the MSACB module; Represents the operation of the nth DHAB module in the MSACB module.

[0042] Preferably, at the decoding end, the features of different resolutions are unified by 3×3 convolution and bilinear upsampling, and then input into the multi-path feature fusion module for adaptive fusion. The fused features are then subjected to 3×3 convolution to obtain F t , F t Added to the low-quality virtual view element level to obtain the quality enhanced virtual view I h , the process is shown as follows:

[0043]

[0044] Where Bilinear(·) represents the bilinear upsampling operation; Represents the features output by the nth MSACB module; express The features output by 3×3 convolution and bilinear upsampling operations; MPFM(·) represents the processing process of the multi-path feature fusion module; F t Represents the fused features; Represents the features output by the fourth MSACB module after 3×3 convolution and bilinear upsampling operations; Represents the features output by the fifth MSACB module after 3×3 convolution and bilinear upsampling operations; Represents the features output by the sixth MSACB module after 3×3 convolution and bilinear upsampling operations.

[0045] Preferably, the total loss function is:

[0046] L total =L ps (I h ,I g )+λL edge (I h ,I g )

[0047] Among them, I hRepresents the virtual view after quality enhancement; I g represents the original high-quality virtual view; L ps represents a hybrid loss function including PSNR loss function and SSIM loss function; L edge represents edge loss;

[0048]

[0049]

[0050] Where SSIM(·) represents the SSIM loss function; PSNR(·) represents the PSNR loss function; Δ represents the Laplace operator; λ, ξ, and ∈ are different constants.

[0051] Preferably, λ is 0.05, ξ is 0.005, and ∈ is 0.001.

[0052] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0053] The present invention provides a virtual view quality enhancement method based on asymmetric splitting. It first extracts shallow features of low-quality virtual views, then combines the ability of multi-head self-attention to model global information and the ability of CNN to capture local texture information, and uses a hybrid codec with a U-shaped network structure to further learn multi-level hierarchical features. Information at different feature scales is adaptively fused, and finally, a quality-enhanced virtual view is obtained based on the fused features. This method can effectively eliminate compression distortion and DIBR distortion existing in the 3D video transmission stage and improve the quality enhancement efficiency of virtual views. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of the steps for implementing the technical solution of the present invention;

[0055] Figure 2 This is a schematic diagram of the overall network model framework of the present invention;

[0056] Figure 3 Schematic diagram of the structure of the MSACB module in the present invention;

[0057] Figure 4 Schematic diagram of the structure of the DHAB module in the present invention;

[0058] Figure 5 This is a schematic diagram of the framework of the multi-head self-attention branch and the CNN branch in the present invention. DETAILED DESCRIPTION

[0059] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0060] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0061] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0062] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0063] Example 1

[0064] like Figure 1-3 As shown, a virtual view quality enhancement method based on asymmetric splitting includes the following steps:

[0065] S1: shallow feature extraction of low-quality input virtual views;

[0066] S2: Input shallow features into the hybrid encoder-decoder to convert them into deep features;

[0067] The hybrid codec has a U-shaped network structure, including multiple MSACB modules, each of which includes multiple DHAB modules (dynamic hybrid attention modules). In the DHAB module, shallow features are asymmetrically divided into multi-head self-attention branches and CNN branches to extract global information and local information respectively.

[0068] S3: Adaptively fuse the features output by each MSACB module at the decoding end of the hybrid codec;

[0069] S4: The quality-enhanced virtual view is reconstructed based on the low-quality virtual view and the fused features.

[0070] In the specific implementation process, the shallow features of the low-quality virtual view are first extracted. Then, the ability of multi-head self-attention to model global information and the ability of CNN to capture local texture information are combined. The hybrid codec with a U-shaped network structure is used to further learn multi-level hierarchical features, and information at different feature scales is adaptively fused. Finally, the quality-enhanced virtual view is obtained based on the fused features. This can effectively eliminate the compression distortion and DIBR distortion existing in the 3D video transmission stage, and improve the quality enhancement efficiency of the virtual view.

[0071] Example 2

[0072] More specifically, the low-quality virtual view of the input is subjected to 3×3 convolution and ReLU nonlinear activation function to extract shallow features F e :

[0073]

[0074] Where σ(·) represents the ReLU nonlinear activation function; represents 3×3 convolution; I l Represents a low-quality virtual view of the input.

[0075] More specifically, the hybrid codec is a four-layer symmetrical U-type hybrid codec.

[0076] On the encoding side, each MSACB module performs feature downsampling through 3×3 convolution and Pixelunshuffle operations. On the decoding side, each MSACB module performs feature upsampling through 3×3 convolution and Pixelshuffle operations. The features output by the MSACB module on the encoding side are transmitted to the MSACB module on the decoding side through skip connections.

[0077] Deep feature F d =F u (F e );

[0078] Among them, F u (·) represents the processing of the U-type hybrid codec.

[0079] More specifically, Figure 4-5 As shown in Figure 2, the process of asymmetrically dividing shallow features into the multi-head self-attention branch (GCCTB) and the CNN branch (GLRB) is as follows:

[0080]

[0081] Among them, the shallow feature F e Through 1×1 convolution Get feature F s ; F g Represents the features flowing into the multi-head self-attention branch; F l Represents the features flowing into the CNN branch; Represents the information flow division unit IFDU operation; α is the contribution factor, which is used to adjust the contribution ratio of the information flow flowing into the two branches. α gradually decreases during the feature downsampling process and gradually increases during the feature upsampling process; C g represents the number of channels flowing into the multi-head self-attention branch; C represents the number of channels flowing into the DHAB module; C l Indicates the number of channels flowing into the CNN branch.

[0082] In its implementation, the information flow division unit (IFDU) is proposed to selectively allocate channels to the two branches by considering the hierarchical nature of the U-shaped network structure, namely the differences in the representation of global semantic information and local texture information at different levels. The IFDU generates different weight pairs (α, 1-α) based on the feature scale, allocating channels to the CNN branch according to the contribution factor α and to the multi-head self-attention branch according to the ratio 1-α, thereby simultaneously learning multi-scale contextual features. The IFDU determines the ratio of channels flowing into the two branches based on the feature scale. When the feature scale is small, more channels flow into the multi-head self-attention branch, thereby achieving efficient feature learning.

[0083] In addition, an adaptive feature fusion unit (AFU) is proposed at the fusion end. The AFU dynamically fuses the feature information from the multi-head self-attention branch and the CNN branch based on the contextual information of the two branches, thereby enhancing the feature representation capability of the codec. It can effectively eliminate the DIBR distortion and compression distortion in the virtual view and improve the quality enhancement efficiency.

[0084] In the DHAB module, the features are first asymmetrically divided into the multi-head self-attention branch and the CNN branch through 1×1 convolution and information flow division unit (IFDU), and then the features of the two branches are fused through the adaptive feature fusion unit (AFU), and finally the features are aggregated through 1×1 convolution.

[0085] In the specific implementation process, GCCTB includes MDTA module and SimpleGate module, and adds layer normalization and dynamic learnable modulator in self-attention module and forward feedback module to stabilize the training process and help the model learn features better. g First, layer normalization LN and MDTA are used to model long-distance pixel dependencies. MDTA encodes local information through three-branch 1×1 convolution and 3×3 depth-separable convolution, and generates query (Q), key (K) and value (V) tensors. Then, the dimension is converted through the dimension conversion operation. Q and K are converted through matrix multiplication and Softmax function to generate transposed attention features. V is then multiplied with the attention feature matrix and subjected to dimension conversion and 1×1 convolution to obtain the output features of MDTA, which are then multiplied channel-wise with the trainable weights generated by DLM and added to the feature F. g The obtained features are then passed through layer normalization LN, 1×1 convolution, SimpleGate and 1×1 convolution modules to further learn features. Among them, the SimpleGate module contains the Chunk operation that divides the features into blocks and the element-level multiplication operator. GLRB includes operations such as 3×3 depth-separable convolution, SimpleGate module and global average pooling. The feature F entering GLRBl First, local context is encoded through layer normalization LN, 1×1 convolution and 3×3 depth-wise separable convolution. Secondly, some redundant information is removed through the SimpleGate gating mechanism to provide nonlinear expression capabilities. Then, channel-level features are selected by global average pooling GAP and 1×1 convolution. Finally, the features are modified by the trainable weights generated by 1×1 convolution and DLM.

[0086] More specifically, the process of extracting global information from the multi-head self-attention branch is as follows:

[0087] Given First, the layer normalization operation is performed, and then 1×1 convolution and 3×3 depth convolution are used in the MDTA module of the multi-head self-attention branch to encode the channel-level spatial context; next, the query and key projection are reconstructed by dot product multiplication to generate a transposed attention map

[0088] The process of the MDTA module is as follows:

[0089]

[0090] in, Represents dimension reshaping; represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; Represents the matrix multiplication operation; F MDTA represents the output of the MDTA module; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represents the query tensor feature; Represents the key tensor feature; Represents the value tensor feature;

[0091] Perform channel-wise multiplication:

[0092] X1=W1*F MDTA ⊕F g

[0093] Among them, X1 represents the output feature after DLM and element addition operation; ⊕ represents element addition operation; Represents the trainable weights generated by the first DLM in the multi-head self-attention branch;

[0094] In actual implementation, DLM is a generator of trainable weight tensors. During the model iteration process, the generated weight tensors will also be iterated at the same time.

[0095] The traditional feedforward network is replaced by 1×1 convolution and simple gating (SG), as shown in the following formula:

[0096]

[0097] Among them, Chunk(·) represents a tensor chunking operation; ⊙ represents an element-wise multiplication operation; Y1 represents the first branch feature output by the tensor chunking operation; Y2 represents the second branch feature output by the tensor chunking operation; Y1 and Y2 have the same feature dimension size; SG(·) represents a simple gating operation; W2 represents the trainable weight generated by the second DLM in the multi-head self-attention branch; Represents the output of the multi-head self-attention branch.

[0098] During the specific implementation process, based on the powerful global modeling capability of multi-head self-attention, a multi-head self-attention module with a cross-channel self-attention score calculation and a nonlinear gating mechanism was proposed. This module has linear computational complexity and efficient global context modeling capabilities, which improves the applicability of the virtual view quality enhancement algorithm in HD / UHD scenarios.

[0099] More specifically, the process of extracting local information by the CNN branch is:

[0100]

[0101] in, represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; represents the output of SG; SG(·) represents a simple gating operation; GAP(·) represents a global average pooling operation; W represents the trainable weight generated by DLM in the CNN branch; SCA(·) represents a simplified channel attention operation; Represents the output of the CNN branch.

[0102] More specifically, the process of the DHAB module is as follows:

[0103]

[0104] An MSACB module includes multiple DHAB modules, output F MSACB The process is shown as follows:

[0105]

[0106] in, Represents the output of the multi-head self-attention branch; represents the output of CNN branch GLRB; F g represents the information flowing into the multi-head self-attention branch; F lRepresents the features flowing into the CNN branch; F GCCTB (·) represents the operation of the entire multi-head self-attention branch; F GLRB (·) represents the operation of the entire CNN branch; F DHAB represents the features output by the DHAB module; Concat(·) represents the concatenation operation; ⊕ represents the element addition operation; F MSACB Represents the characteristics of the MSACB module output; Represents the operation of the first DHAB module in the MSACB module; Represents the operation of the second DHAB module in the MSACB module; Represents the operation of the nth DHAB module in the MSACB module.

[0107] In its implementation, the DHAB module (Dynamic Hybrid Attention Module) achieves global context modeling and local texture detail acquisition capabilities by combining a multi-head self-attention branch and a CNN branch in parallel. The multi-head self-attention branch calculates attention scores at the channel level, making its computational complexity linear with the spatial resolution of the input image. Compared to the traditional multi-head self-attention mechanism, which has quadratic complexity, this approach significantly improves its applicability in HD / UHD applications.

[0108] More specifically, at the decoding end, 3×3 convolution and bilinear upsampling are used to unify features of different resolutions, and then input into the multi-path feature fusion module (MPFM) for adaptive fusion. The fused features are then subjected to 3×3 convolution to obtain F t , F t Added to the low-quality virtual view element level to obtain the quality enhanced virtual view I h , the process is shown as follows:

[0109]

[0110] Where Bilinear(·) represents the bilinear upsampling operation; Represents the features output by the nth MSACB module; express The features output by 3×3 convolution and bilinear upsampling operations; MPFM(·) represents the processing process of the multi-path feature fusion module; F t Represents the fused features; Represents the features output by the fourth MSACB module after 3×3 convolution and bilinear upsampling operations; Represents the features output by the fifth MSACB module after 3×3 convolution and bilinear upsampling operations; Represents the features output by the sixth MSACB module after 3×3 convolution and bilinear upsampling operations.

[0111] In the specific implementation process, the multi-path feature fusion module can reduce the redundancy of information on the decoder side and efficiently interact with features of different resolutions, dynamically fuse information from different feature scales, and help the network learn more contextual information.

[0112] Example 3

[0113] More specifically, the total loss function is:

[0114] L total =L ps (I h , I g )+λL edge (I h , I g )

[0115] Among them, I h Represents the virtual view after quality enhancement; I g represents the original high-quality virtual view; L ps represents a hybrid loss function including PSNR loss function and SSIM loss function; L edge represents edge loss;

[0116] L ps It consists of PSNR loss function and SSIM loss function:

[0117]

[0118] Margin loss L edge As shown in the following formula:

[0119]

[0120] Where SSIM(·) represents the SSIM loss function; PSNR(·) represents the PSNR loss function; Δ represents the Laplace operator; λ, ξ, and ∈ are different constants.

[0121] More specifically, λ is 0.05, ξ is 0.005, and ∈ is 0.001.

[0122] In the specific implementation process, a loss function that considers human visual perception and image quality evaluation index metrics is introduced for optimization, where the edge loss L edge It can consider local texture details and help the model pay attention to more edge and texture information.

[0123] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A virtual view quality enhancement method based on asymmetric streaming, characterized in that: The following steps are involved: S1: shallow feature extraction of low-quality input virtual views; S2: Input shallow features into the hybrid encoder-decoder to convert them into deep features; The hybrid codec has a U-shaped network structure, including multiple MSACB modules, each of which includes multiple DHAB modules. In the DHAB module, shallow features are asymmetrically divided into multi-head self-attention branches and CNN branches to extract global information and local information respectively; The process of the DHAB module is as follows: An MSACB module includes multiple DHAB modules, output F MSACB The process is shown as follows: in, Represents the output of the multi-head self-attention branch; represents the output of CNN branch GLRB; F g represents the information flowing into the multi-head self-attention branch; F l Represents the features flowing into the CNN branch; F GCCTB (·) represents the operation of the entire multi-head self-attention branch; F GLRB (·) represents the operation of the entire CNN branch; F DHAB Represents the features output by the DHAB module; Concat(·) represents the concatenation operation; Indicates element addition operation; F MSACB Represents the characteristics of the MSACB module output; Represents the operation of the first DHAB module in the MSACB module; Represents the operation of the second DHAB module in the MSACB module; Represents the operation of the nth DHAB module in the MSACB module; At the decoding end, 3×3 convolution and bilinear upsampling are used to unify the features of different resolutions, and then input into the multi-path feature fusion module for adaptive fusion. The fused features are then subjected to 3×3 convolution to obtain F t , F t Added to the low-quality virtual view element level to obtain the quality enhanced virtual view I h , the process is shown as follows: Where Bilinear(·) represents the bilinear upsampling operation; Represents the features output by the nth MSACB module; express The features output by 3×3 convolution and bilinear upsampling operations; MPFM(·) represents the processing process of the multi-path feature fusion module; F t Represents the fused features; Represents the features of the fourth MSACB output after 3×3 convolution and bilinear upsampling operation; Represents the features of the fifth MSACB output after 3×3 convolution and bilinear upsampling operations; Represents the features of the sixth MSACB output after 3×3 convolution and bilinear upsampling operations; S3: Adaptively fuse the features output by each MSACB module at the decoding end of the hybrid codec; S4: The quality-enhanced virtual view is reconstructed based on the low-quality virtual view and the fused features.

2. The method for enhancing virtual view quality based on asymmetric streaming according to claim 1, wherein: The low-quality virtual view of the input is extracted through 3×3 convolution and ReLU nonlinear activation function to extract shallow features F e : Where σ(·) represents the ReLU nonlinear activation function; represents 3×3 convolution; I l Represents a low-quality virtual view of the input.

3. The method for enhancing virtual view quality based on asymmetric streaming according to claim 1, wherein: The hybrid codec is a four-layer symmetrical U-type hybrid codec. On the encoding side, each MSACB module performs feature downsampling through 3×3 convolution and Pixelunshuffle operations. On the decoding side, each MSACB module performs feature upsampling through 3×3 convolution and Pixelshuffle operations. The features output by the MSACB module on the encoding side are transmitted to the MSACB module on the decoding side through skip connections. Deep feature F d =F u (F e ); Among them, F u (·) represents the processing of the U-type hybrid codec.

4. The method for enhancing virtual view quality based on asymmetric streaming according to claim 1, wherein: The process of asymmetrically dividing shallow features into the multi-head self-attention branch and the CNN branch is: Among them, the shallow feature F e Through 1×1 convolution Get feature F s ; F g Represents the features flowing into the multi-head self-attention branch; F l Represents the features flowing into the CNN branch; Represents the information flow division unit IFDU operation; α is the contribution factor, which is used to adjust the contribution ratio of the information flow flowing into the two branches. α gradually decreases during the feature downsampling process and gradually increases during the feature upsampling process; C g represents the number of channels flowing into the multi-head self-attention branch; C represents the number of channels flowing into the DHAB module; C l Indicates the number of channels flowing into the CNN branch.

5. The method for enhancing virtual view quality based on asymmetric streaming according to claim 4, wherein: The process of extracting global information from the multi-head self-attention branch is: Given F g , first through the layer normalization operation, and then use 1×1 convolution and 3×3 depth convolution in the MDTA module of the multi-head self-attention branch to encode the channel-level spatial context; The process of the MDTA module is as follows: in, Represents dimension reshaping; represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; Represents the matrix multiplication operation; F MDTA represents the output of the MDTA module; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represents the query tensor feature; Represents the key tensor feature; Represents the value tensor feature; Perform channel-wise multiplication: Among them, X1 represents the output feature after DLM and element addition operation; Represents the element addition operation; W1 represents the trainable weight generated by the first DLM in the multi-head self-attention branch; Replace the feedforward network with 1×1 convolution and simple gating, as shown below: Where Chunk(·) represents a tensor chunking operation; ⊙ represents an element-wise multiplication operation; Y1 represents the first branch feature output by the tensor chunking operation; Y2 represents the second branch feature output by the tensor chunking operation; Y1 and Y2 have the same feature dimension size; SG(·) represents a simple gating operation; W2 represents the trainable weight generated by the second DLM in the multi-head self-attention branch; Represents the output of the multi-head self-attention branch.

6. The method for enhancing virtual view quality based on asymmetric streaming according to claim 4, wherein: The process of extracting local information by the CNN branch is: in, represents a 3×3 depth-wise separable convolution operation; LN(·) represents a layer normalization operation; represents the output of SG; SG(·) represents a simple gating operation; GAP(·) represents a global average pooling operation; W represents the trainable weight generated by DLM in the CNN branch; SCA(·) represents a simplified channel attention operation; Represents the output of the CNN branch GLRB.

7. The method for enhancing virtual view quality based on asymmetric streaming according to claim 1, wherein: The total loss function is: THE total =L ps (THE h ,THE g )+λL edge (THE h ,THE g ) Among them, I h Represents the virtual view after quality enhancement; I g represents the original high-quality virtual view; L ps represents a hybrid loss function including PSNR loss function and SSIM loss function; L edge represents edge loss; Where SSIM(·) represents the SSIM loss function; PSNR(·) represents the PSNR loss function; Δ represents the Laplace operator; λ, ξ, and ∈ are different constants.

8. The method for enhancing virtual view quality based on asymmetric streaming according to claim 7, wherein: λ is 0.05, ξ is 0.005, and ∈ is 0.001.

Images