Immersive video enhancement method and device based on frequency domain boundary collaborative optimization

Abstract

The invention discloses an immersive video enhancement method and device based on frequency domain boundary collaborative optimization, and relates to the field of video processing, and the method comprises the steps: obtaining a to-be-reconstructed compressed multi-view texture and depth video sequence, and inputting the sequence into a trained immersive video enhancement model; a current to-be-enhanced video frame passes through a feature extraction module, and high-frequency features and low-frequency features are extracted respectively; the high-frequency features and the low-frequency features pass through a frequency domain enhancement module to obtain a frequency domain enhanced image; inputting the frequency domain enhancement image and the current video frame to be enhanced into a boundary enhancement module to obtain a fused image; and inputting the fused image and an adjacent video frame of the current to-be-enhanced video frame into a space-time deformable convolution module to obtain an aligned fused image, and performing prediction on the aligned fused image through a quality enhancement module to obtain an enhanced residual error and generate a corresponding reconstructed video. According to the method, the problems of compression artifacts, boundary artifacts, low quality of immersive videos and the like are solved.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to an immersive video enhancement method and device based on frequency domain boundary collaborative optimization. Background Art

[0002] Immersive video is a type of emerging media video that can provide a high degree of freedom and can provide users with a highly immersive and realistic visual experience. By integrating multi-viewpoint texture and depth information, immersive video supports users to watch three-dimensional scenes with 6 degrees of freedom, allowing users to perceive the spatial layering and three-dimensional effects of the scene from any perspective. However, the amount of data in immersive video is huge, involving ultra-high resolution and complex multi-viewpoint processing. It is easy to introduce compression distortion, artifacts and other problems during the process of acquisition, compression, transmission and rendering, thus affecting the user's viewing experience. This puts higher requirements on video quality improvement technology and is also a key challenge in promoting the development of immersive video applications.

[0003] Multiview Video plus Depth (MVD) is a mainstream immersive video data format, which consists of texture videos collected from multiple viewpoints and corresponding depth videos obtained through depth estimation or depth cameras. Due to the complexity and diversity of MVD videos in space, time and viewpoint dimensions, various compression distortions such as blur and artifacts in compressed MVD videos are more significant, seriously affecting the user's immersive experience. Therefore, it is particularly important to propose an algorithm that conforms to the visual characteristics of the human eye and can accurately and quickly enhance the quality of immersive videos.

[0004] At present, the research on immersive video quality enhancement is mostly focused on single-frame or single-viewpoint video reconstruction, ignoring the role of multi-viewpoint fusion and inter-frame information in quality recovery. In addition, current research generally lacks sufficient consideration of the unique characteristics of immersive video. Therefore, designing a video quality enhancement algorithm that combines human visual characteristics with immersive video characteristics can not only provide theoretical support for the quality improvement of immersive video, but also has significant practical application value, which is of great significance for promoting the popularization and industrialization of immersive video technology. Summary of the invention

[0005] The purpose of this application is to propose an immersive video enhancement method and device based on frequency domain boundary collaborative optimization in response to the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides an immersive video enhancement method based on frequency domain boundary collaborative optimization, comprising the following steps:

[0007] An immersive video enhancement model is constructed and trained to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module connected in sequence;

[0008] A compressed multi-view texture plus depth video sequence to be reconstructed is obtained and input into a trained immersive video enhancement model; the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by a feature extraction module to extract high-frequency features and low-frequency features respectively; the high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by a frequency domain enhancement module and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; the frequency domain enhanced image and the current video frame to be enhanced are input into a boundary enhancement module to obtain a fused image; the fused image and adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into a spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image; the aligned fused image is passed through a quality enhancement module to predict an enhanced residual and generate a corresponding reconstructed video.

[0009] Preferably, the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by a feature extraction module to respectively extract high-frequency features and low-frequency features, specifically including:

[0010] For the video sequence to be enhanced , Indicates the current video frame to be enhanced. Indicates the number of frames before and after the current video frame to be enhanced. frame, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames;

[0011] The current video frame to be enhanced is subjected to Fourier transform to obtain the corresponding frequency domain features, as shown in the following formula:

[0012] ;

[0013] in, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of and Respectively represent the width and height of the current video frame to be enhanced, represents Fourier transform;

[0014] The low-frequency features and high-frequency features are separated from the frequency domain features by Gaussian filtering, as shown in the following formula:

[0015] ;

[0016] ;

[0017] in, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The low frequency characteristics of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced high frequency characteristics.

[0018] Preferably, the high-frequency features and the low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module, and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image, which specifically includes:

[0019] For low-frequency features, the low-frequency weight is calculated using the following formula:

[0020] ;

[0021] in, Indicates the coordinates in the frequency domain The low frequency weight, is the low frequency enhancement factor, , is the smoothing term, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of Indicates the coordinates in the frequency domain The viewpoint importance weight is expressed as follows:

[0022] ;

[0023] in, The frequency domain coordinates representing the center frequency of the user's viewpoint area, The parameter representing the user's viewpoint area is expressed as: , represents the experience factor, Indicates the maximum value of the frequency domain feature located at any frequency domain coordinate corresponding to the current video frame to be enhanced;

[0024] For high-frequency features, the high-frequency weight is calculated using the following formula:

[0025] ;

[0026] in, Located in the frequency domain The high frequency weights, is the high frequency retention factor, , Indicates the coordinates in the frequency domain The motion saliency weight of is expressed as follows:

[0027] ;

[0028] in, Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude is Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude, max means taking the maximum value;

[0029] The low-frequency weight and high-frequency weight are used to weight the low-frequency features and high-frequency features respectively to obtain the enhanced frequency domain features, as shown in the following formula:

[0030] ;

[0031] in, Indicates the enhanced frequency domain features corresponding to the current video frame to be enhanced;

[0032] The enhanced frequency domain features are returned to the spatial domain through inverse Fourier transform to obtain the frequency domain enhanced image, as shown in the following formula:

[0033] ;

[0034] in, Indicates the pixel coordinates in the frequency domain enhanced image corresponding to the current video frame to be enhanced The pixel value of and Respectively represent the width and height of the current video frame to be enhanced.

[0035] Preferably, the frequency domain enhanced image and the current video frame to be enhanced are input into a boundary enhancement module to obtain a fused image, which specifically includes:

[0036] The Canny edge detection algorithm is used to extract the significant boundary area in the frequency domain enhanced image and obtain the boundary features, as shown in the following formula:

[0037] ;

[0038] in, Indicates the pixel coordinates corresponding to the current video frame to be enhanced The boundary characteristics of Represents the Canny edge detection algorithm;

[0039] Calculate the pixel interpolation of the neighboring area in the current video frame to be enhanced, and estimate the artifact intensity, as shown in the following formula:

[0040] ;

[0041] in, Indicates the pixel coordinates in the current video frame to be enhanced The artifact intensity, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the pixel coordinates in the current video frame to be enhanced The smoothed estimate of the neighboring pixels is expressed as follows:

[0042] ;

[0043] in, Represents pixel coordinates Around Neighborhood, The pixel coordinates in the current video frame to be enhanced are The neighboring pixel coordinates The pixel value of It means taking the median of all pixel values ​​in the neighborhood;

[0044] The boundary features and artifact strength are weighted to calculate the boundary weight, as shown in the following formula:

[0045] ;

[0046] in, is the boundary enhancement factor, represents the artifact impact factor;

[0047] The boundary weight is multiplied by the frequency domain enhanced image corresponding to the current video frame to be enhanced to obtain a fused image, as shown in the following formula:

[0048] ;

[0049] in, Indicates the pixel coordinates in the fused image corresponding to the current video frame to be enhanced The pixel value of .

[0050] Preferably, the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned by using the complementary relationship between the frames to obtain the aligned fused image. The aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video, which specifically includes:

[0051] The image features are passed through the spatiotemporal deformable convolution module to obtain the aligned fused image, as shown in the following formula:

[0052]

[0053] in, represents the size of the convolution kernel, represents the convolution kernel channels, represents the aligned fused image corresponding to the current video frame to be enhanced, represents the fused image corresponding to the current video frame to be enhanced, represents any spatial position, represents the convolution kernel The conventional sampling offset corresponding to each channel is Indicates that the time position is the Tth frame and the spatial position is The corresponding deformable offset is extracted by combining the offset fields of all time positions and spatial positions in the current video frame to be enhanced. The expression is as follows:

[0054]

[0055] in, represents the U-net offset prediction network, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames;

[0056] The aligned fused image is input into the quality enhancement module consisting of multi-level residual dense channel attention blocks to predict the enhanced residual , which will enhance the residual Add the current video frame to be enhanced element by element to generate a reconstructed video frame , as shown below:

[0057] ;

[0058] in, It represents element-by-element addition, and all reconstructed video frames constitute a reconstructed video sequence and synthesize the reconstructed video.

[0059] Preferably, the total loss function used in the training process of the immersive video enhancement model is a weighted result of the reconstruction loss and the boundary loss, as shown in the following formula:

[0060] ;

[0061] in, represents the total loss function, and Represent the weight of reconstruction loss and the weight of boundary loss respectively, and denote reconstruction loss and boundary loss respectively;

[0062] The reconstruction loss is calculated as follows:

[0063] ;

[0064] in, Represents the total number of pixels, and denote the pixel value of the i-th pixel of the reconstructed video frame and the pixel value of the i-th pixel of the original image respectively;

[0065] The calculation formula of boundary loss is as follows:

[0066] ;

[0067] in, Represents the gradient, obtained by the Sobel edge detection operator, and The gradient value of the i-th pixel of the video frame and the gradient value of the i-th pixel of the original image are reconstructed respectively.

[0068] In a second aspect, the present invention provides an immersive video enhancement device based on frequency domain boundary collaborative optimization, comprising:

[0069] A model building module is configured to build and train an immersive video enhancement model to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module connected in sequence;

[0070] The reconstruction module is configured to obtain a compressed multi-view texture plus depth video sequence to be reconstructed and input it into a trained immersive video enhancement model; the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to extract high-frequency features and low-frequency features respectively; the high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; the frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image; the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image, and the aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video.

[0071] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0072] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0073] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] (1) The immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in the present invention constructs an immersive video enhancement model to dynamically integrate frequency domain features, boundary features and spatiotemporal information, thereby effectively improving the quality of immersive video.

[0076] (2) The immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in the present invention utilizes the frequency domain enhancement module to extract important information related to immersive video compression distortion from high-frequency features and low-frequency features, optimizes global structure perception and motion details, and effectively reduces compression artifacts. It utilizes edge detection and gradient information in the boundary enhancement module to focus on processing the boundary area between foreground and background, enhances the reconstruction capability of geometric boundary details, and effectively suppresses boundary artifacts.

[0077] (3) The immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in the present invention realizes collaborative enhancement of the frequency domain and the boundary by dynamically aggregating the frequency enhancement image and boundary weights. It combines the inter-frame complementary relationship with the viewpoint weighting strategy to optimize the spatiotemporal consistency and detail restoration of multi-viewpoint immersive videos, comprehensively improve the video quality and visual immersion, and effectively improves the immersive video quality enhancement performance and the overall quality of the video through the deep fusion of viewpoint information and the dynamic alignment of the inter-frame complementary relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A schematic diagram of a flow chart of an immersive video enhancement method based on frequency domain boundary collaborative optimization according to an embodiment of the present application;

[0080] Figure 2 A schematic diagram of an immersive video enhancement model of an immersive video enhancement method based on frequency domain boundary collaborative optimization according to an embodiment of the present application;

[0081] Figure 3 A schematic diagram of an immersive video enhancement device based on frequency domain boundary collaborative optimization according to an embodiment of the present application;

[0082] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Figure 1 An immersive video enhancement method based on frequency domain boundary collaborative optimization provided by an embodiment of the present application is shown, comprising the following steps:

[0085] S1, constructing and training an immersive video enhancement model to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module which are connected in sequence.

[0086] Specifically, the immersive video enhancement model proposed in the embodiment of the present application is composed of a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module and a quality enhancement module. First, the obtained compressed multi-view texture plus depth video (MVD) sequence is transformed in the frequency domain by the feature extraction module to extract high-frequency features and low-frequency features respectively; then, in the frequency domain enhancement module, according to the immersive video distortion characteristics, the low-frequency features are optimized based on the viewpoint weighting to enhance the global structure perception, and the high-frequency weights corresponding to the high-frequency features are dynamically adjusted in combination with the motion region significance, and finally the frequency domain enhanced image is obtained. In the boundary enhancement module, the boundary features are enhanced to strengthen the detail recovery of the foreground and background junction area, and the frequency domain enhanced image and the boundary features are dynamically aggregated to generate a fused image. The fused image is further input into the spatiotemporal deformable module to align the time information by using the complementary relationship between frames to generate an aligned fused image; finally, the aligned fused image is input into the quality enhancement module to predict the enhanced residual, and the high-quality enhanced reconstructed video corresponding to the compressed multi-view texture plus depth video (MVD) sequence is output. The spatiotemporal deformable convolution module and the quality enhancement module are existing modules, and their specific calculation process and principles are not repeated here.

[0087] In a specific embodiment, the total loss function used in the training process of the immersive video enhancement model is a weighted result of the reconstruction loss and the boundary loss, as shown in the following formula:

[0088] ;

[0089] in, represents the total loss function, and Represent the weight of reconstruction loss and the weight of boundary loss respectively, and denote reconstruction loss and boundary loss respectively;

[0090] The reconstruction loss is calculated as follows:

[0091] ;

[0092] in, Represents the total number of pixels, and denote the pixel value of the i-th pixel of the reconstructed video frame and the pixel value of the i-th pixel of the original image respectively;

[0093] The calculation formula of boundary loss is as follows:

[0094] ;

[0095] in, Represents the gradient, obtained by the Sobel edge detection operator, and The gradient value of the i-th pixel of the video frame and the gradient value of the i-th pixel of the original image are reconstructed respectively.

[0096] Specifically, in order to guide the immersive video enhancement model to optimize the video quality during the training process, a total loss function that comprehensively considers the reconstruction loss and the boundary loss is designed to restore more detailed information during the training process. Among them, the reconstruction loss function is designed by measuring the pixel-level difference between the reconstructed video frame and the original video frame, and the boundary loss function is designed by the pixel-level difference at the edge area of ​​the reconstructed video frame and the original video frame. A trained immersive video enhancement model is obtained based on the total loss function training.

[0097] S2, obtain the compressed multi-view texture plus depth video sequence to be reconstructed and input it into the trained immersive video enhancement model; the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to extract high-frequency features and low-frequency features respectively; the high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; the frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image; the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image, and the aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video.

[0098] Specifically, the trained immersive video enhancement model is deployed, and the compressed multi-view texture plus depth video sequence to be reconstructed can be input into the trained immersive video enhancement model for video reconstruction.

[0099] In a specific embodiment, the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by a feature extraction module to respectively extract high-frequency features and low-frequency features, specifically including:

[0100] For the video sequence to be enhanced , Indicates the current video frame to be enhanced. Indicates the number of frames before and after the current video frame to be enhanced. frame, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames;

[0101] The current video frame to be enhanced is subjected to Fourier transform to obtain the corresponding frequency domain features, as shown in the following formula:

[0102] ;

[0103] in, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of and Respectively represent the width and height of the current video frame to be enhanced, represents Fourier transform;

[0104] The low-frequency features and high-frequency features are separated from the frequency domain features by Gaussian filtering, as shown in the following formula:

[0105] ;

[0106] ;

[0107] in, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The low frequency characteristics of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced high frequency characteristics.

[0108] Specifically, refer to Figure 2 , if there are M video sequences to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed, each video sequence to be enhanced is composed of the current video frame to be enhanced and the previous and next R frames. In the feature extraction module, the current video frame to be enhanced is subjected to Fourier transform (FFT) to obtain the corresponding frequency domain features, and the low-frequency features representing global structural information (such as smooth areas) and the high-frequency features representing local detail information (such as edges and textures) are separated by Gaussian filtering.

[0109] In a specific embodiment, the high-frequency features and the low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image, which specifically includes:

[0110] For low-frequency features, the low-frequency weight is calculated using the following formula:

[0111] ;

[0112] in, Indicates the coordinates in the frequency domain The low frequency weight, is the low frequency enhancement factor, , is the smoothing term, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of Indicates the coordinates in the frequency domain The viewpoint importance weight is expressed as follows:

[0113] ;

[0114] in, The frequency domain coordinates representing the center frequency of the user's viewpoint area, The parameter representing the user's viewpoint area is expressed as: , represents the experience factor, Indicates the maximum value of the frequency domain feature located at any frequency domain coordinate corresponding to the current video frame to be enhanced;

[0115] For high-frequency features, the high-frequency weight is calculated using the following formula:

[0116] ;

[0117] in, Located in the frequency domain The high frequency weights, is the high frequency retention factor, , Indicates the coordinates in the frequency domain The motion saliency weight of is expressed as follows:

[0118] ;

[0119] in, Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude is Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude, max means taking the maximum value;

[0120] The low-frequency weight and high-frequency weight are used to weight the low-frequency features and high-frequency features respectively to obtain the enhanced frequency domain features, as shown in the following formula:

[0121] ;

[0122] in, Indicates the enhanced frequency domain features corresponding to the current video frame to be enhanced;

[0123] The enhanced frequency domain features are returned to the spatial domain through inverse Fourier transform to obtain the frequency domain enhanced image, as shown in the following formula:

[0124] ;

[0125] in, Indicates the pixel coordinates in the frequency domain enhanced image corresponding to the current video frame to be enhanced The pixel value of and Respectively represent the width and height of the current video frame to be enhanced.

[0126] Specifically, in the frequency domain enhancement module, low-frequency weights of different importance are assigned to low-frequency features and high-frequency features according to the visual characteristics of the human eye and the characteristics of immersive video. and high frequency weights For low-frequency features, in order to better adapt to the global characteristics of immersive video distortion, the viewpoint importance weight is introduced A higher weight is given to the user's viewpoint area that is currently being focused on. The viewpoint importance weight represents the weight distribution of low-frequency features in the user's viewpoint area, which can be generated by a Gaussian model of the user's viewpoint area. For high-frequency features, since moving objects in immersive videos are usually accompanied by significant changes in high-frequency details, motion saliency weights are introduced. The weights are dynamically adjusted in combination with motion information, and the low-frequency features and high-frequency features are weighted and fused through low-frequency weights and high-frequency weights to obtain the corresponding enhanced frequency domain features. The enhanced frequency domain features are returned to the spatial domain through inverse Fourier transform (IFFT) to obtain a frequency domain enhanced image.

[0127] In a specific embodiment, the frequency domain enhanced image and the current video frame to be enhanced are input into a boundary enhancement module to obtain a fused image, which specifically includes:

[0128] The Canny edge detection algorithm is used to extract the significant boundary area in the frequency domain enhanced image and obtain the boundary features, as shown in the following formula:

[0129] ;

[0130] in, Indicates the pixel coordinates corresponding to the current video frame to be enhanced The boundary characteristics of Represents the Canny edge detection algorithm;

[0131] Calculate the pixel interpolation of the neighboring area in the current video frame to be enhanced, and estimate the artifact intensity, as shown in the following formula:

[0132] ;

[0133] in, Indicates the pixel coordinates in the current video frame to be enhanced The artifact intensity, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the pixel coordinates in the current video frame to be enhanced The smoothed estimate of the neighboring pixels is expressed as follows:

[0134] ;

[0135] in, Represents pixel coordinates Around Neighborhood, The pixel coordinates in the current video frame to be enhanced are The neighboring pixel coordinates The pixel value of It means taking the median of all pixel values ​​in the neighborhood;

[0136] The boundary features and artifact strength are weighted to calculate the boundary weight, as shown in the following formula:

[0137] ;

[0138] in, is the boundary enhancement factor, represents the artifact impact factor;

[0139] The boundary weight is multiplied by the frequency domain enhanced image corresponding to the current video frame to be enhanced to obtain a fused image, as shown in the following formula:

[0140] ;

[0141] in, Indicates the pixel coordinates in the fused image corresponding to the current video frame to be enhanced The pixel value of .

[0142] Specifically, in the boundary enhancement module, the Canny edge detection algorithm is used to extract the significant boundary area in the frequency domain enhanced image and obtain the boundary features. The boundary features are dynamically weighted to give the boundary area a higher degree of attention. The frequency enhanced image and the boundary weight are fused to obtain the fused image.

[0143] In a specific embodiment, the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned by using the complementary relationship between the frames to obtain an aligned fused image. The aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video, which specifically includes:

[0144] The image features are passed through the spatiotemporal deformable convolution module to obtain the aligned fused image, as shown in the following formula:

[0145]

[0146] in, represents the size of the convolution kernel, represents the convolution kernel channels, represents the aligned fused image corresponding to the current video frame to be enhanced, represents the fused image corresponding to the current video frame to be enhanced, represents any spatial position, represents the convolution kernel The conventional sampling offset corresponding to each channel is Indicates that the time position is the Tth frame and the spatial position is The corresponding deformable offset is extracted by combining the offset fields of all time positions and spatial positions in the current video frame to be enhanced. The expression is as follows:

[0147]

[0148] in, represents the U-net offset prediction network, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames;

[0149] The aligned fused image is input into the quality enhancement module consisting of multi-level residual dense channel attention blocks to predict the enhanced residual , which will enhance the residual Add the current video frame to be enhanced element by element to generate a reconstructed video frame , as shown below:

[0150] ;

[0151] in, It represents element-by-element addition, and all reconstructed video frames constitute a reconstructed video sequence and synthesize the reconstructed video.

[0152] Specifically, to obtain temporal motion compensation, the video sequence is input into the spatiotemporal deformable convolution module (STDC) to fuse the adjacent frame information, aggregate the frequency domain features according to the variability offset, and obtain the aligned fused image. . The aligned fused image The quality enhancement module composed of multi-level residual dense channel attention blocks is input to obtain the predicted enhancement residual, and then the enhancement residual is added element by element to the current video frame to be enhanced to generate a reconstructed video frame. When all the video frames to be enhanced generate corresponding reconstructed video frames in the above manner, a reconstructed video sequence can be obtained, and finally the reconstructed video is synthesized.

[0153] The following is an explanation through specific experiments.

[0154] The experiment uses the MVD dataset provided by the MIV standard. The dataset contains 16 multi-view test sequences with different resolutions, including 10 computer-generated content (CG) and 6 natural content (NC). The texture video format of the video sequence is YUV 4:2:0 10 bits, and the depth video format is YUV 4:2:0 16 bits. The projection format includes two types: Equirectangular Projection (ERP) and Perspective Projection. Table 1 lists the detailed video parameters of the test sequence. The QP parameter combination specified by the MPEG standard test conditions is also given, and the specific numerical settings are shown in Table 2.

[0155] Table 1 Experimental sequence information:

[0156]

[0157] Table 2 Video sequence quantization parameter setting table:

[0158]

[0159] The experimental data set uses 16 multi-viewpoint MVD sequences improved by MPEG, each sequence has 9 to 25 viewpoints, and each viewpoint video contains 300 frames. In the experiment, 12 sequences are used as training sets, and two training sequences (P, T) are selected as additional test sequences except the last 100 frames of each sequence, and all videos of the remaining 4 sequences (A, E, G, O) are used as test sets. The experiment compares the enhancement effects of MFQE1.0, MFQE2.0, STDF, RFDA and the immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in this application under two sets of quantization parameter combinations, QP3 and QP4. The objective image quality evaluation indicators used are video peak signal-to-noise ratio (PSNR) and immersive video peak signal-to-noise ratio (IV-PSNR).

[0160] Table 3 Compression parameters when QP3 PSNR (dB) and IV-PSNR (dB) comparison results:

[0161]

[0162] As shown in Table 3, the enhancement results when the compression parameter is QP3, in all test scenarios, the IV-PSNR value increment of the immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in this application is significantly better than the existing methods. For low-quality (such as sequence O and P) scenes, the immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in this application achieves a large degree of detail recovery and artifact suppression by dynamically fusing visual saliency information, and the IV-PSNR increment is improved by about 0.07 to 0.10 dB. In other high-quality scenarios, the immersive video enhancement method based on frequency domain boundary collaborative optimization proposed in this application can still ensure that the enhancement effect is better than the existing methods by optimizing boundary details and aligning spatiotemporal information.

[0163] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an immersive video enhancement device based on frequency domain boundary collaborative optimization. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0164] The embodiment of the present application provides an immersive video enhancement device based on frequency domain boundary collaborative optimization, including:

[0165] A model building module 1 is configured to build and train an immersive video enhancement model to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module connected in sequence;

[0166] Reconstruction module 2 is configured to obtain a compressed multi-view texture plus depth video sequence to be reconstructed and input it into a trained immersive video enhancement model; the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to extract high-frequency features and low-frequency features respectively; the high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; the frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image; the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image, and the aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video.

[0167] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0168] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0169] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0170] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0171] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0172] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0173] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0174] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0175] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0176] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0177] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0178] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0179] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0180] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0181] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An immersive video enhancement method based on frequency domain boundary collaborative optimization, characterized in that: The following steps are involved: An immersive video enhancement model is constructed and trained to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module connected in sequence; A compressed multi-view texture plus depth video sequence to be reconstructed is obtained and input into the trained immersive video enhancement model; a current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to respectively extract high-frequency features and low-frequency features; The high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module, and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; The frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image; the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image; the aligned fused image passes through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video.

2. The immersive video enhancement method based on frequency domain boundary collaborative optimization according to claim 1, characterized in that: The current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to respectively extract high-frequency features and low-frequency features, specifically including: For the video sequence to be enhanced , Indicates the current video frame to be enhanced. Indicates the number of frames before and after the current video frame to be enhanced. frame, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames; The current video frame to be enhanced is subjected to Fourier transform to obtain the corresponding frequency domain features, as shown in the following formula: ； in, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of and Respectively represent the width and height of the current video frame to be enhanced, represents Fourier transform; The low-frequency features and high-frequency features are separated from the frequency domain features by Gaussian filtering, as shown in the following formula: ； ； in, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The low frequency characteristics of Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced high frequency characteristics.

3. The immersive video enhancement method based on frequency domain boundary collaborative optimization according to claim 1, characterized in that: The high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module, and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image, specifically including: For the low-frequency features, the low-frequency weight is calculated using the following formula: ； in, Indicates the coordinates in the frequency domain The low frequency weight, is the low frequency enhancement factor, , is the smoothing term, represents the Gaussian filter function, Indicates the frequency domain coordinates corresponding to the current video frame to be enhanced The frequency domain characteristics of Indicates the coordinates in the frequency domain The viewpoint importance weight is expressed as follows: ； in, The frequency domain coordinates representing the center frequency of the user's viewpoint area, The parameter representing the user's viewpoint area is expressed as: , represents the experience factor, Indicates the maximum value of the frequency domain feature located at any frequency domain coordinate corresponding to the current video frame to be enhanced; For the high-frequency features, the high-frequency weight is calculated using the following formula: ； in, Located in the frequency domain The high frequency weights, is the high frequency retention factor, , Indicates the coordinates in the frequency domain The motion saliency weight of is expressed as follows: ； in, Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude is Indicates the pixel coordinates between the current video frame to be enhanced and the previous video frame The pixel change amplitude, max means taking the maximum value; The low-frequency weight and high-frequency weight are used to weight the low-frequency features and high-frequency features respectively to obtain the enhanced frequency domain features, as shown in the following formula: ； in, Indicates the enhanced frequency domain features corresponding to the current video frame to be enhanced; The enhanced frequency domain features are returned to the spatial domain through inverse Fourier transform to obtain a frequency domain enhanced image, as shown in the following formula: ； in, Indicates the pixel coordinates in the frequency domain enhanced image corresponding to the current video frame to be enhanced The pixel value of and Respectively represent the width and height of the current video frame to be enhanced.

4. The immersive video enhancement method based on frequency domain boundary collaborative optimization according to claim 1, characterized in that: The frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image, which specifically includes: The Canny edge detection algorithm is used to extract the significant boundary area in the frequency domain enhanced image to obtain the boundary features, as shown in the following formula: ； in, Indicates the pixel coordinates corresponding to the current video frame to be enhanced The boundary characteristics of Represents the Canny edge detection algorithm; Calculate the pixel interpolation of the neighboring area in the current video frame to be enhanced, and estimate the artifact intensity, as shown in the following formula: ； in, Indicates the pixel coordinates in the current video frame to be enhanced The artifact intensity, Indicates the pixel coordinates in the current video frame to be enhanced The pixel value of Indicates the pixel coordinates in the current video frame to be enhanced The smoothed estimate of the neighboring pixels is expressed as follows: ； in, Represents pixel coordinates Around Neighborhood, The pixel coordinates in the current video frame to be enhanced are The neighboring pixel coordinates The pixel value of It means taking the median of all pixel values ​​in the neighborhood; The boundary feature and the artifact strength are weighted to calculate the boundary weight, as shown in the following formula: ； in, is the boundary enhancement factor, represents the artifact impact factor; The boundary weight is multiplied by the frequency domain enhanced image corresponding to the current video frame to be enhanced to obtain a fused image, as shown in the following formula: ； in, Indicates the pixel coordinates in the fused image corresponding to the current video frame to be enhanced The pixel value of .

5. The immersive video enhancement method based on frequency domain boundary collaborative optimization according to claim 1, characterized in that: The fused image and adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned by using the complementary relationship between frames to obtain an aligned fused image. The aligned fused image is passed through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video, specifically including: The image features are passed through the spatiotemporal deformable convolution module to obtain an aligned fused image, as shown in the following formula: ； in, represents the size of the convolution kernel, represents the convolution kernel channels, represents the aligned fused image corresponding to the current video frame to be enhanced, represents the fused image corresponding to the current video frame to be enhanced, represents any spatial position, represents the convolution kernel The conventional sampling offset corresponding to each channel is Indicates that the time position is the Tth frame and the spatial position is The corresponding deformable offset is extracted by combining the offset fields of all time positions and spatial positions in the current video frame to be enhanced. The expression is as follows: ； in, represents the U-net offset prediction network, Indicates the number of frames before the current video frame to be enhanced. Frames are adjacent video frames, Indicates the number of frames after the current video frame to be enhanced. Frame adjacent video frames; The aligned fused image is input into a quality enhancement module composed of multi-level residual dense channel attention blocks to predict the enhanced residual , which will enhance the residual Add the current video frame to be enhanced element by element to generate a reconstructed video frame , as shown below: ； in, It represents element-by-element addition, and all reconstructed video frames constitute a reconstructed video sequence and synthesize the reconstructed video.

6. The immersive video enhancement method based on frequency domain boundary collaborative optimization according to claim 1, characterized in that: The total loss function used in the training process of the immersive video enhancement model is the weighted result of the reconstruction loss and the boundary loss, as shown in the following formula: ； in, represents the total loss function, and Represent the weight of reconstruction loss and the weight of boundary loss respectively, and denote reconstruction loss and boundary loss respectively; The calculation formula of the reconstruction loss is as follows: ； in, Represents the total number of pixels, and denote the pixel value of the i-th pixel of the reconstructed video frame and the pixel value of the i-th pixel of the original image respectively; The calculation formula of the boundary loss is as follows: ； in, Represents the gradient, obtained by the Sobel edge detection operator, and The gradient value of the i-th pixel of the video frame and the gradient value of the i-th pixel of the original image are reconstructed respectively.

7. An immersive video enhancement device based on frequency domain boundary collaborative optimization, characterized in that: include: A model building module is configured to build and train an immersive video enhancement model to obtain a trained immersive video enhancement model, wherein the immersive video enhancement model includes a feature extraction module, a frequency domain enhancement module, a boundary enhancement module, a spatiotemporal deformable convolution module, and a quality enhancement module connected in sequence; The reconstruction module is configured to obtain a compressed multi-view texture plus depth video sequence to be reconstructed and input it into the trained immersive video enhancement model; the current video frame to be enhanced in each video sequence to be enhanced in the compressed multi-view texture plus depth video sequence to be reconstructed is first subjected to frequency domain transformation and feature extraction by the feature extraction module to respectively extract high-frequency features and low-frequency features; The high-frequency features and low-frequency features are respectively assigned corresponding high-frequency weights and low-frequency weights by the frequency domain enhancement module, and weighted fusion and inverse transformation are performed to obtain a frequency domain enhanced image; The frequency domain enhanced image and the current video frame to be enhanced are input into the boundary enhancement module to obtain a fused image; the fused image and the adjacent video frames of the current video frame to be enhanced in the video sequence to be enhanced are input into the spatiotemporal deformable convolution module, and the time information is aligned using the complementary relationship between frames to obtain an aligned fused image; the aligned fused image passes through the quality enhancement module to predict the enhanced residual and generate the corresponding reconstructed video.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images