Multi-view video synthesis method based on double-stream re-parameterization

Abstract

The invention provides a multi-view video synthesis method based on double-flow re-parameterization, and the method comprises the steps: building a double-flow re-parameterization refocusing mixed attention network model, training the double-flow re-parameterization refocusing mixed attention network model, optimizing the parameters, and synthesizing a multi-view video based on double-flow re-parameterization. The double-flow re-parameterization refocusing mixed attention network comprises a feature extraction stage and a feature reconstruction stage, a double-flow structure of a local feature pyramid attention information flow and a global multi-scale residual information flow is adopted in the feature extraction stage, the local feature pyramid attention information flow is used for extracting local texture information of an image, and the global multi-scale residual information flow is used for extracting local texture information of the image. The global multi-scale residual information flow is used for acquiring global context information in a synthetic view to ensure comprehensive grasp of detail and overall information, and the feature reconstruction stage is to integrate and output global and local information from a double-flow structure. According to the method, the problems of artifacts, cracks and holes in view synthesis can be effectively solved, and the image visual quality is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video encoding and quality enhancement, and in particular to a multi-view video synthesis method based on dual-stream re-parameterization. Background Art

[0002] With the rapid development of video technology, video content is evolving towards high-definition and three-dimensional video. Three-dimensional video (3DV) has gradually attracted widespread attention in emerging applications such as virtual reality (VR), augmented reality (AR), and six-degree-of-freedom (6DoF) video systems because it can provide an immersive stereoscopic visual experience. As a typical form of three-dimensional video, multi-view video captures the same scene from different angles through multiple cameras, allowing users to achieve interactive experiences such as free perspective selection and scene roaming. However, the amount of data in multi-view video is significantly increased compared to traditional single-view video, resulting in huge challenges in storage and transmission. To solve this problem, the International Telecommunication Union (ITU-T) and the International Organization for Standardization (ISO / IEC) jointly launched the 3D High Efficiency Video Coding (3D-HEVC) standard, which aims to effectively compress multi-view video data. 3D-HEVC is an extension of the two-dimensional video coding standard HEVC, specifically designed for efficient encoding of videos in the multi-view plus depth (MVD) format. The MVD format includes texture maps of multiple perspectives, corresponding depth maps, and camera parameters. Using the deep image rendering (DIBR) technology, the decoder can generate virtual views from existing views.

[0003] Virtual view synthesis technology relies on the accuracy of depth maps and the precision of rendering algorithms, but in practical applications, DIBR technology faces many challenges. First, after multi-view video encoding, the depth image usually degrades due to compression loss and transmission noise, resulting in inaccurate depth information in the rendered virtual view. Second, during the three-dimensional mapping process of the virtual view, due to errors in depth information, pixel interpolation rounding errors, and object occlusion problems, the synthesized view often has artifacts, cracks, and holes. These visual defects appear mainly at the junction of the foreground and background, seriously affecting the overall quality of the synthesized view and reducing the user's visual experience. In addition, the current DIBR technology does not adequately handle complex local texture details, making it difficult to achieve a balance between multi-scale information, especially in high dynamic ranges and complex scenes, and the effect is still unsatisfactory.

[0004] In order to solve the above problems, researchers have tried to enhance the quality of virtual views by improving the encoding and decoding technology of depth maps and introducing post-processing algorithms. However, the inventors of this application have found that these traditional methods are often limited in effect when processing complex scenes or large-scale multi-view data, and cannot fully address the various distortion problems generated during the virtual view synthesis process. Therefore, in recent years, advanced technologies based on deep learning have gradually been introduced into the field of DIBR virtual view synthesis, and it is very necessary to automatically extract and learn global and local information in view synthesis through deep neural networks to significantly improve the visual quality of synthetic views. Summary of the invention

[0005] In view of the technical problems that artifacts, cracks and holes often appear in the existing multi-view video synthesis, the present invention provides a multi-view video synthesis method based on dual-stream reparameterization. The method is based on a dual-stream reparameterization refocusing hybrid attention network. The network combines the global multi-scale residual information flow and the local feature pyramid attention information flow to adaptively extract global context information and local texture details at different scales, effectively solve the problems of artifacts, cracks and holes in view synthesis, and significantly improve the image visual quality.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A multi-view video synthesis method based on dual-stream re-parameterization comprises the following steps:

[0008] S1. Construction of dual-stream re-parameterized re-focused hybrid attention network model:

[0009] S11. The dual-stream re-parameterized refocused hybrid attention network includes two stages: feature extraction and feature reconstruction. The feature extraction stage adopts a dual-stream structure of a local feature pyramid attention information flow and a global multi-scale residual information flow. The local feature pyramid attention information flow is used to extract local texture information of the image, and the global multi-scale residual information flow is used to obtain global context information in the synthetic view to ensure a comprehensive grasp of details and overall information; the feature reconstruction stage integrates and outputs the global and local information from the local feature pyramid attention information flow and the global multi-scale residual information flow;

[0010] S12, the local feature pyramid attention information flow structure includes a convolution layer for extracting low-level features and a four-layer symmetrical encoder-decoder structure, the encoder structure includes four re-parameterized refocusing attention modules and three down-sampling modules located behind the low-level feature extraction convolution layer, a re-parameterized refocusing attention module and a down-sampling module are arranged alternately in sequence, and two convolution layers are set behind the last re-parameterized refocusing attention module, the decoder structure includes three convolution layers, four re-parameterized refocusing attention modules and three up-sampling modules located behind the two convolution layers, a convolution layer, a re-parameterized refocusing attention module and an up-sampling module are arranged alternately in sequence, and the last re-parameterized refocusing attention module is used as the output;

[0011] The global multi-scale residual information flow structure includes a pixel inverse recombination layer, a first convolutional layer, a first refocusing attention module, twelve residual blocks connected in series, a second refocusing attention module, a second convolutional layer and a pixel recombination layer arranged in sequence;

[0012] The feature fusion module is used for integration in the feature reconstruction stage. The feature fusion module includes two efficient feature fusion modules and two convolutional layers. A convolutional layer is set behind each efficient feature fusion module and they are arranged alternately in sequence.

[0013] S2. Dual-stream re-parameterization and re-focusing hybrid attention network model training and parameter optimization:

[0014] S21. Dataset establishment and preprocessing: 11 video sequences with different resolutions are used as experimental datasets, of which 9 sequences are used for model training and 2 sequences are used for testing. Each video sequence contains 100 frames of images, and the data is composed of low-quality synthetic view images paired with their corresponding reference images. To ensure the generalization ability of the model, the scenes in the training and test datasets are completely non-overlapping, ensuring that the scenes in the test data do not appear in the training dataset;

[0015] All video sequences are jointly encoded according to different quality levels under public test conditions, including four distortion levels of 30-39, 35-42, 40-45 and 45-48;

[0016] Finally, Mixup data shuffling is used to enhance the data of the training image data samples in the training data set, and random horizontal flipping and random rotation are used to improve the robustness and generalization ability of the model.

[0017] S22, taking the 64×64 image pair as the input of the two-stream re-parameterized refocused hybrid attention network, and generating a feature vector through network forward calculation;

[0018] S23, model training configuration: In model training, the optimizer uses AdamW, and sets the exponential decay rate β1 of the first-order momentum estimate to 0.9, the exponential decay rate β2 of the second-order momentum estimate to 0.999, and the weight decay value to 0.02; during the entire training process, the learning rate is increased from 3e -4 Start with the cosine annealing algorithm and gradually reduce the learning rate to 1e -6 ; The loss function uses PSELoss to effectively capture the feature differences of synthetic images, which is defined as follows:

[0019]

[0020] Among them, L SSIM represents the SSIM loss, L PSNR represents the PSNR loss, I h represents the predicted synthetic view, I g represents the corresponding undistorted reference view, ξ and λ are constants, ε is used as a penalty coefficient to help the network effectively handle outliers, Δ represents the Laplacian operator, Δ(I h ) indicates I h The Laplace operator, Δ(I g ) indicates I g The Laplace operator of

[0021] S24, using adaptive moment estimation gradient descent algorithm to optimize the objective function L PSE , use error back propagation to update the parameters of the two-stream re-parameterized refocused hybrid attention network model to obtain the optimal network model parameters;

[0022] S3. Multi-view video synthesis based on dual-stream re-parameterization:

[0023] Using the optimal network model parameters learned, a multi-view video synthesis network based on dual-stream re-parameterized and refocused mixed attention is built. Low-quality synthetic view images are input into the network for multi-view video synthesis, and high-quality synthetic view images are output for quality improvement.

[0024] Further, in step S12, each re-parameterized refocusing attention module includes a multi-scale convolution and a refocusing attention module arranged in sequence starting from the input end, and the output of the refocusing attention module is cascaded with the input of the input end, and the cascaded output is connected to a 3×3 convolution layer.

[0025] Furthermore, the multi-scale convolution includes, starting from the input end, a 3×3 convolution layer and a 3×3 dilated convolution layer arranged in parallel, a PReLU activation function arranged after the 3×3 convolution layer and the 3×3 dilated convolution layer, respectively, an element addition arranged after the activation function to be added to the input end input, a cascade for processing the addition result of the two elements, and a 1×1 convolution layer arranged after the cascade.

[0026] Furthermore, the refocusing attention module includes a multi-scale feature extraction module and a feature refocusing module arranged in sequence. The multi-scale feature extraction module has two branches arranged in parallel starting from the input end, the first branch includes a maximum pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence, and the second branch includes an average pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence, and the output of the last 1×1 convolution layer of the two branches is added as the input of the feature refocusing module; the feature refocusing module includes a sigmoid activation function and a dot product arranged in sequence starting from the input end, the output of the dot product is connected in parallel to take the maximum value and the average value, the output of the maximum value and the average value is connected in cascade, and the output of the cascade is connected in sequence to the basic convolution, the sigmoid activation function and the dot product.

[0027] Furthermore, the basic convolution includes a re-parameterized convolution, normalization, and a PReLU activation function that are sequentially arranged starting from the input end.

[0028] Further, in step S12, each residual block includes a 3×3 convolutional layer, a PReLU activation function, a 3×3 convolutional layer, and element-wise addition of the results of two 3×3 convolutional layers, which are arranged in sequence starting from the input end.

[0029] Furthermore, in step S12, each efficient feature fusion module consists of two branches: a global average pooling branch and a reparameterized convolution branch. The global average pooling branch focuses on modeling cross-channel information and integrating spatial position information into the channel attention map to enhance feature expression capabilities; the reparameterized convolution branch focuses on providing multi-scale spatial structure information and combining this information with the features of the global average pooling branch.

[0030] Furthermore, the network forward calculation in step S22 includes convolution operation, batch normalization and nonlinear excitation.

[0031] Furthermore, in the convolution operation, the output feature map Z corresponding to any convolution kernel is i The following formula is used for calculation:

[0032]

[0033] Where f represents the nonlinear activation function, b i represents the bias term corresponding to the i-th convolution kernel, r represents the input channel index number, k represents the number of input channels, and W ir represents the rth channel weight matrix of the i-th convolution kernel, is the convolution loss, X r Represents the rth input channel image.

[0034] Furthermore, the nonlinear excitation uses the rectified linear unit PRELU as the feature map Z i The nonlinear activation function is used to transform the feature map Z i Each normalized value is transformed nonlinearly, and the rectified linear unit PRELU is defined as follows:

[0035]

[0036] Here, x is an input value and σ() is a nonlinear activation function.

[0037] Compared with the prior art, the multi-view video synthesis method based on dual-stream reparameterization provided by the present invention has the following advantages: (1) by building a dual-stream reparameterized refocused hybrid attention network (TRRHA) model, the TRRHA model extracts local texture details and global context information through the dual-stream structure of local feature pyramid attention information flow (LFPA) and global multi-scale residual information flow (GMR), effectively improving the details and structural consistency of the synthesized image, realizing efficient fusion of local and global information, and effectively solving the problems of artifacts, cracks and holes in view synthesis; (2) using the efficient feature fusion module (EFFM) to fuse the global average pooling and multi-scale reparameterized convolution (RC) branches to ensure the accuracy of the spatial structure and details of the reconstructed image, and generate a self- (3) Pixel Unshuffle and Pixel Shuffle operations are used to expand the receptive field while reducing information loss, lowering computational complexity and improving the efficiency of the model in high-resolution tasks; (4) Through data augmentation such as Mixup data shuffling and random rotation as well as strict separation of training and test data, the model still performs well in unseen scenes and has excellent generalization ability; (5) The model can maintain high performance under different levels of distortion, especially in the case of low-quality input, and still generate high-quality output, showing strong anti-distortion ability and reducing visual defects in DIBR technology in 3D video, virtual reality and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the dual-stream re-parameterized refocused hybrid attention network structure provided by the present invention.

[0039] Figure 2 It is a schematic diagram of the structure of the re-parameterized refocusing attention module provided by the present invention.

[0040] Figure 3 It is a schematic diagram of the residual block structure provided by the present invention.

[0041] Figure 4 It is a schematic diagram of the structure of the attention refocusing module provided by the present invention.

[0042] Figure 5 It is a schematic diagram of the structure of the efficient feature fusion module provided by the present invention. DETAILED DESCRIPTION

[0043] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0044] Please refer to Figure 1 As shown, the present invention provides a multi-view video synthesis method based on dual-stream re-parameterization, comprising the following steps:

[0045] S1. Construction of dual-stream re-parameterized re-focused hybrid attention network model:

[0046] S11. The dual-stream re-parameterized refocused hybrid attention network includes two stages: feature extraction and feature reconstruction. The feature extraction stage adopts a dual-stream structure of local feature pyramid attention information flow (LFPA) and global multi-scale residual information flow (GMR). The local feature pyramid attention information flow is used to extract local texture information of the image, and the global multi-scale residual information flow is used to obtain global context information in the synthetic view to ensure a comprehensive grasp of details and overall information; the feature reconstruction stage integrates and outputs the global and local information from the local feature pyramid attention information flow and the global multi-scale residual information flow.

[0047] S12. The local feature pyramid attention information flow structure includes a 3×3 convolutional layer for extracting low-level features and a four-layer symmetrical encoder-decoder structure. The encoder structure includes four re-parameterized refocusing attention modules (RRAMs) and three down-sampling modules located behind the low-level feature extraction convolutional layer. A re-parameterized refocusing attention module and a down-sampling are arranged alternately in sequence, that is, a down-sampling operation is set behind each re-parameterized refocusing attention module, and two convolutional layers are set behind the last re-parameterized refocusing attention module. The decoder structure includes three convolutional layers, four re-parameterized refocusing attention modules and three up-sampling modules located behind two convolutional layers. A convolutional layer, a re-parameterized refocusing attention module and an up-sampling are arranged alternately in sequence, and the last re-parameterized refocusing attention module is used as the output. Specifically, in the feature extraction LFPA stage, the model takes a low-quality synthetic view image as input, which consists of 3 channels and has a size of 64×64. The input image first passes through a 3×3 convolutional layer to extract low-level features, and then is further processed through a four-layer symmetrical encoder-decoder structure. The encoder part includes a re-parameterized refocused attention module and a downsampling operation, while the decoder part uses a 3×3 convolutional layer, RRAM blocks, and upsampling technology for processing. Downsampling is achieved through a convolution operation with a stride of 2, and upsampling is performed through ConvTranspose2d. The main goal of this stage is to obtain important information in the view through feature extraction modules of different scales.

[0048] As a specific example, please refer to Figure 2 As shown, each re-parameterized refocused attention module (RRAM) includes a multi-scale convolution (RC) and a refocused attention module (RAM) arranged in sequence starting from the input end. The output of the refocused attention module is cascaded with the input of the input end, and the output of the cascade is connected to a 3×3 convolution layer.

[0049] As a preferred embodiment, please refer to Figure 2 As shown, the multi-scale convolution (RC) includes a 3×3 convolution layer and a 3×3 dilated convolution layer arranged in parallel from the input end, a PReLU activation function arranged after the 3×3 convolution layer and the 3×3 dilated convolution layer, an element addition arranged after the activation function and the input end input, a cascade for processing the result of the addition of the two elements, and a 1×1 convolution layer arranged after the cascade. The specific LFPA network parameters are as follows.

[0050] Table 1 LFPA network parameters

[0051]

[0052]

[0053] The global multi-scale residual information flow structure includes a sequentially arranged pixel unshuffle layer, a first convolutional layer, a first refocusing attention module, twelve serially connected residual blocks, a second refocusing attention module, a second convolutional layer, and a pixel unshuffle layer. Specifically, in the global multi-scale residual information flow (GMR), the initial low-quality synthetic image is first downsampled by a scaling factor of 2 through PixelUnshuffle, and further processed by a RAM module (refocusing attention module), and then the global context features are captured by serially connecting 12 residual blocks (RB), and finally upsampled by a Pixelshuffle operation with a scaling factor of 2 to restore the image. PixelUnshuffle and Pixelshuffle are a set of symmetric operations that can effectively expand the receptive field of the network during downsampling and upsampling, thereby ensuring that feature information is not lost in the process.

[0054] In the entire model, the global multi-scale residual information flow mainly focuses on the acquisition of global information, especially the understanding of the overall structure and multi-scale features of the scene, while LFPA plays an important role in capturing local details and contextual information. By combining these two information flows, the model can efficiently extract and fuse local and global information, ultimately improving the quality and realism of the synthesized view.

[0055] As a specific example, please refer to Figure 4 As shown in the figure, in the global multi-scale residual information flow structure, the refocusing attention module includes a multi-scale feature extraction module (Multi-scale feature extraction) and a feature refocusing module (Features Refocusing) arranged in sequence, and the multi-scale feature extraction module has two branches arranged in parallel starting from the input end, the first branch includes a maximum pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence, and the second branch includes an average pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence, and the output of the last 1×1 convolution layer of the two branches is added as the input of the feature refocusing module; the feature refocusing module includes a sigmoid activation function and a dot product arranged in sequence starting from the input end, the output of the dot product is connected in parallel to take the maximum value and the average value, the output of the maximum value and the average value is connected in cascade, and the output of the cascade is connected in sequence to the basic convolution, the sigmoid activation function and the dot product.

[0056] As a specific example, please refer to Figure 4 As shown, the basic convolution includes a re-parameterized convolution, normalization, and PReLU activation function set in sequence starting from the input end.

[0057] As a specific example, please refer to Figure 3 As shown in the figure, in the global multi-scale residual information flow structure, each residual block (RB) includes a 3×3 convolutional layer, a PReLU activation function, a 3×3 convolutional layer, and an element-wise addition of the results of two 3×3 convolutional layers starting from the input end. The specific GMR network parameters are as follows.

[0058] Table 2 GMR network parameters

[0059]

[0060] The feature reconstruction stage uses a feature fusion module (FFM) for integration. The feature fusion module includes two efficient feature fusion modules (EFFM) and two convolutional layers. A convolutional layer is set behind each efficient feature fusion module and arranged alternately in sequence. Specifically in the feature reconstruction stage, the core component is the efficient feature fusion module (EFFM), which is responsible for integrating the local information from the LFPA and the global information from the GMR. The EFFM can be likened to a high-quality image editor that fuses different levels of information (local texture and global context) into a seamless final image. This integration ensures that local details and global structures can be effectively expressed in the final synthetic view.

[0061] Specifically, please refer to Figure 5 As shown in the figure, each efficient feature fusion module (EFFM) consists of two branches: the global average pooling branch and the reparameterized convolution branch. The global average pooling branch focuses on modeling cross-channel information and integrates spatial position information into the channel attention map to enhance feature expression capabilities. By capturing global information, the model can better understand the overall structure of the input image, thereby improving the expressiveness of complex scenes; the reparameterized convolution branch focuses on providing multi-scale spatial structure information and combining this information with the features of the global average pooling branch. Through this integration of multi-scale features, EFFM can fully understand the spatial structure of the input features, thereby significantly improving the model's expressiveness when processing complex tasks.

[0062] Through the synergy of these two branches, EFFM achieves full fusion of local and global features, so that the final generated synthetic view not only has rich detail expression but also retains global consistency. This efficient feature fusion strategy demonstrates excellent reconstruction capabilities in complex scenes and ensures high-quality output of synthetic images. The specific FFM network parameters are as follows.

[0063] Table 3 FFM network parameters

[0064]

[0065] S2. Dual-stream re-parameterization and re-focusing hybrid attention network model training and parameter optimization:

[0066] S21. Dataset establishment and preprocessing: In the network model training phase, the present invention adopts a carefully designed experimental setting to ensure the excellent performance of the model in processing multi-view video synthesis. The experiment uses 11 video sequences of different resolutions as the experimental data set, of which 9 sequences are used for model training and 2 sequences are used for testing. Each video sequence contains 100 frames of images, and the data is composed of low-quality synthetic view images paired with their corresponding reference images. To ensure the generalization ability of the model, the scenes in the training and test data sets are completely non-overlapping, ensuring that the scenes in the test data do not appear in the training data set, so that the model can show good generalization ability when dealing with unseen data;

[0067] During the experiment, all video sequences were jointly encoded according to different quality levels (QP pairs) under public test conditions, including four distortion levels of 30-39, 35-42, 40-45 and 45-48, and the encoding platform used HTM-16.0; Finally, 900 synthetic views were extracted from the video sequences with a resolution of 1920×1088 or 1024×768, and each synthetic view image was cropped into an image block of size 64×64 and a stride of 64, generating a total of 1,327,200 training samples;

[0068] Finally, Mixup data shuffling is used to perform data enhancement on the training image data samples in the training dataset, and random horizontal flipping and random rotation of 90°, 180°, and 270° are used to improve the robustness and generalization ability of the model.

[0069] S22. Use the 64×64 image pair as the input of the dual-stream reparameterized refocused hybrid attention network, and generate a feature vector through network forward calculation. Specifically, the network forward calculation includes convolution operation, batch normalization and nonlinear excitation.

[0070] Convolution operation: In the convolution operation, the output feature map Z corresponding to any convolution kernel is i The following formula is used for calculation:

[0071]

[0072] Where f represents the nonlinear activation function, b i represents the bias term corresponding to the i-th convolution kernel, r represents the input channel index number, k represents the number of input channels, and W ir represents the rth channel weight matrix of the i-th convolution kernel, is the convolution loss, X r Represents the rth input channel image.

[0073] Batch Normalization: Batch Normalization (BN) is a method that makes neural network training faster and more stable. It is used to calculate the mean and variance of each mini-batch and pull it back to a standard normal distribution with a mean of 0 and a variance of 1. The specific operation and calculation are well known to those skilled in the art.

[0074] Nonlinear excitation: The nonlinear excitation uses the rectified linear unit PRELU as the feature map Z i The nonlinear activation function is used to transform the feature map Z i Each normalized value is transformed nonlinearly, and the rectified linear unit PRELU is defined as follows:

[0075]

[0076] Here, x is an input value and σ() is a nonlinear activation function.

[0077] S23. Model training configuration: In terms of training configuration, the optimizer used in the training of the TRRHA model is AdamW, and the exponential decay rate β1 of the first-order momentum estimation is set to 0.9, the exponential decay rate β2 of the second-order momentum estimation (gradient squared sum moving average) is set to 0.999, and the weight decay value is 0.02; during the entire training process, the learning rate is increased from 3e -4 Start and use the cosine annealing algorithm (CosineAnnealingLR) to gradually reduce the learning rate to 1e -6 ; The loss function uses PSE Loss to effectively capture the feature differences of synthetic images, which is defined as follows:

[0078]

[0079] Among them, L SSIM represents the SSIM loss, L PSNR represents the PSNR loss, I h represents the predicted synthetic view, I g represents the corresponding undistorted reference view, ξ and λ are constants, ε is used as a penalty coefficient to help the network effectively handle outliers, Δ represents the Laplacian operator, Δ(I h ) indicates I h The Laplace operator, Δ(I g ) indicates I g As a specific implementation, the constant λ is set to 0.05, another constant ξ is set to 0.005 based on experience, and ε is set to 0.001.

[0080] The training process was conducted for a total of 100 epochs, covering all QP pairs, ensuring that the model learned data of different quality levels. In the test phase, each video sequence was tested frame by frame according to its corresponding QP pair. The entire experiment was conducted on the Ubuntu 16.04 operating system. The hardware platform included Intel i7-10900K CPU and NVIDIA 3090 GPU. The model was trained and inferred based on the PyTorch framework. This training setting and optimization strategy ensured the efficient performance of the model in multi-view video synthesis tasks.

[0081] During the model training process, the present invention proposes an optimized loss function PSE Loss, which balances the relationship between objective evaluation indicators and human visual perception, so that the generated synthetic image not only meets the objective quality evaluation standards, but also shows a higher image quality in human subjective perception, effectively avoiding the shortcomings of traditional loss functions on image quality.

[0082] S24, using adaptive moment estimation gradient descent algorithm to optimize the objective function L PSE , use error back propagation to update the two-stream reparameterized refocused hybrid attention network model parameters and obtain the optimal network model parameters.

[0083] S3. Multi-view video synthesis based on dual-stream re-parameterization:

[0084] Using the optimal network model parameters learned, a multi-view video synthesis network based on dual-stream re-parameterized and refocused mixed attention is built. Low-quality synthetic view images are input into the network for multi-view video synthesis, and high-quality synthetic view images are output for quality improvement.

[0085] Compared with the prior art, the multi-view video synthesis method based on dual-stream reparameterization provided by the present invention has the following advantages: (1) by building a dual-stream reparameterized refocused hybrid attention network (TRRHA) model, the TRRHA model extracts local texture details and global context information through the dual-stream structure of local feature pyramid attention information flow (LFPA) and global multi-scale residual information flow (GMR), effectively improving the details and structural consistency of the synthesized image, realizing efficient fusion of local and global information, and effectively solving the problems of artifacts, cracks and holes in view synthesis; (2) using the efficient feature fusion module (EFFM) to fuse the global average pooling and multi-scale reparameterized convolution (RC) branches to ensure the accuracy of the spatial structure and details of the reconstructed image, and generate a self- (3) Pixel Unshuffle and Pixel Shuffle operations are used to expand the receptive field while reducing information loss, lowering computational complexity and improving the efficiency of the model in high-resolution tasks; (4) Through data augmentation such as Mixup data shuffling and random rotation as well as strict separation of training and test data, the model still performs well in unseen scenes and has excellent generalization ability; (5) The model can maintain high performance under different levels of distortion, especially in the case of low-quality input, and still generate high-quality output, showing strong anti-distortion ability and reducing visual defects in DIBR technology in 3D video, virtual reality and other applications.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-view video synthesis method based on dual-stream re-parameterization, characterized in that: The following steps are involved: S1. Construction of dual-stream re-parameterized re-focused hybrid attention network model: S11. The dual-stream re-parameterized refocused hybrid attention network includes two stages: feature extraction and feature reconstruction. The feature extraction stage adopts a dual-stream structure of a local feature pyramid attention information flow and a global multi-scale residual information flow. The local feature pyramid attention information flow is used to extract local texture information of the image, and the global multi-scale residual information flow is used to obtain global context information in the synthetic view to ensure a comprehensive grasp of details and overall information; the feature reconstruction stage integrates and outputs the global and local information from the local feature pyramid attention information flow and the global multi-scale residual information flow; S12, the local feature pyramid attention information flow structure includes a convolution layer for extracting low-level features and a four-layer symmetrical encoder-decoder structure, the encoder structure includes four re-parameterized refocusing attention modules and three down-sampling modules located behind the low-level feature extraction convolution layer, a re-parameterized refocusing attention module and a down-sampling module are arranged alternately in sequence, and two convolution layers are set behind the last re-parameterized refocusing attention module, the decoder structure includes three convolution layers, four re-parameterized refocusing attention modules and three up-sampling modules located behind the two convolution layers, a convolution layer, a re-parameterized refocusing attention module and an up-sampling module are arranged alternately in sequence, and the last re-parameterized refocusing attention module is used as the output; The global multi-scale residual information flow structure includes a pixel inverse recombination layer, a first convolutional layer, a first refocusing attention module, twelve residual blocks connected in series, a second refocusing attention module, a second convolutional layer and a pixel recombination layer arranged in sequence; The feature fusion module is used for integration in the feature reconstruction stage. The feature fusion module includes two efficient feature fusion modules and two convolutional layers. A convolutional layer is set behind each efficient feature fusion module and they are arranged alternately in sequence. S2. Dual-stream re-parameterization and re-focusing hybrid attention network model training and parameter optimization: S21. Dataset establishment and preprocessing: 11 video sequences with different resolutions are used as experimental datasets, of which 9 sequences are used for model training and 2 sequences are used for testing. Each video sequence contains 100 frames of images, and the data is composed of low-quality synthetic view images paired with their corresponding reference images. To ensure the generalization ability of the model, the scenes in the training and test datasets are completely non-overlapping, ensuring that the scenes in the test data do not appear in the training dataset; All video sequences are jointly encoded according to different quality levels under public test conditions, including four distortion levels of 30-39, 35-42, 40-45 and 45-48; Finally, Mixup data shuffling is used to enhance the data of the training image data samples in the training data set, and random horizontal flipping and random rotation are used to improve the robustness and generalization ability of the model. S22, taking the 64×64 image pair as the input of the two-stream re-parameterized refocused hybrid attention network, and generating a feature vector through network forward calculation; S23, model training configuration: In model training, the optimizer uses AdamW, and sets the exponential decay rate β1 of the first-order momentum estimate to 0.9, the exponential decay rate β2 of the second-order momentum estimate to 0.999, and the weight decay value to 0.02; during the entire training process, the learning rate is increased from 3e -4 Start with the cosine annealing algorithm and gradually reduce the learning rate to 1e -6 ; The loss function uses PSELoss to effectively capture the feature differences of synthetic images, which is defined as follows: Among them, L SSIM represents the SSIM loss, L PSNR represents the PSNR loss, I h represents the predicted synthetic view, I g represents the corresponding undistorted reference view, ξ and λ are constants, ε is used as a penalty coefficient to help the network effectively handle outliers, Δ represents the Laplacian operator, Δ(I h ) indicates I h The Laplace operator, Δ(I g ) indicates I g The Laplace operator of S24, using adaptive moment estimation gradient descent algorithm to optimize the objective function L PSE , use error back propagation to update the parameters of the two-stream re-parameterized refocused hybrid attention network model to obtain the optimal network model parameters; S3. Multi-view video synthesis based on dual-stream re-parameterization: Using the optimal network model parameters learned, a multi-view video synthesis network based on dual-stream re-parameterized and refocused mixed attention is built. Low-quality synthetic view images are input into the network for multi-view video synthesis, and high-quality synthetic view images are output for quality improvement.

2. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 1, characterized in that: In step S12, each re-parameterized refocusing attention module includes a multi-scale convolution and a refocusing attention module arranged in sequence starting from the input end, the output of the refocusing attention module is cascaded with the input of the input end, and the cascaded output is connected to a 3×3 convolution layer.

3. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 2, characterized in that: The multi-scale convolution includes, starting from the input end, a 3×3 convolution layer and a 3×3 dilated convolution layer arranged in parallel, a PReLU activation function arranged after the 3×3 convolution layer and the 3×3 dilated convolution layer, respectively, an element addition arranged after the activation function to be added to the input end input, a cascade for processing the addition result of the two elements, and a 1×1 convolution layer arranged after the cascade.

4. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 1 or 2, characterized in that: The refocusing attention module includes a multi-scale feature extraction module and a feature refocusing module arranged in sequence. The multi-scale feature extraction module has two branches arranged in parallel starting from the input end. The first branch includes a maximum pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence. The second branch includes an average pooling layer, a 1×1 convolution layer, a PReLU activation function and a 1×1 convolution layer arranged in sequence. The output of the last 1×1 convolution layer of the two branches is added together as the input of the feature refocusing module; the feature refocusing module includes a sigmoid activation function and a dot product arranged in sequence starting from the input end. The outputs of the dot product are connected in parallel to take the maximum value and the average value. The outputs of the maximum value and the average value are connected in cascade. The output of the cascade is connected in sequence to the basic convolution, the sigmoid activation function and the dot product.

5. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 4, characterized in that: The basic convolution includes a re-parameterized convolution, normalization, and a PReLU activation function, which are set in sequence starting from the input end.

6. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 1, characterized in that: In the step S12, each residual block includes a 3×3 convolutional layer, a PReLU activation function, a 3×3 convolutional layer, and element-wise addition of the results of two 3×3 convolutional layers, which are arranged in sequence starting from the input end.

7. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 1, characterized in that: In step S12, each efficient feature fusion module consists of two branches: a global average pooling branch and a reparameterized convolution branch. The global average pooling branch focuses on modeling cross-channel information and integrates spatial position information into the channel attention map to enhance feature expression capabilities; the reparameterized convolution branch focuses on providing multi-scale spatial structure information and combining this information with the features of the global average pooling branch.

8. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 1, characterized in that: The network forward calculation in step S22 includes convolution operation, batch normalization and nonlinear excitation.

9. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 8, characterized in that: In the convolution operation, the output feature map Z corresponding to any convolution kernel i The following formula is used for calculation: Where f represents the nonlinear activation function, b i represents the bias term corresponding to the i-th convolution kernel, r represents the input channel index number, k represents the number of input channels, and W ir represents the rth channel weight matrix of the i-th convolution kernel, is the convolution loss, X r Represents the rth input channel image.

10. The multi-view video synthesis method based on dual-stream re-parameterization according to claim 8, characterized in that: The nonlinear excitation uses the rectified linear unit PRELU as the feature map Z i The nonlinear activation function is used to transform the feature map Z i Each normalized value is subjected to nonlinear transformation, and the rectified linear unit PRELU is defined as follows: Here, x is an input value and σ( ) is a nonlinear activation function.

Images