Panoramic video image reconstruction method, device and equipment

Abstract

The present invention is applicable to the field of computer technology and provides a panoramic video image reconstruction method, device and equipment. The method includes: receiving a panoramic video projection image frame to be reconstructed, performing feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame, recalibrating the preliminary reconstruction features according to the panoramic video projection image frame to obtain calibrated reconstruction features, and then restoring the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction features, thereby realizing reconstruction of the Y component image of the panoramic video projection image frame, improving the reconstruction quality of the panoramic video image, and at the same time improving the reconstruction efficiency of the panoramic video image.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method, device and equipment for reconstructing panoramic video images. Background Art

[0002] In recent years, 360-degree panoramic video has attracted considerable attention, both in terms of industrial applications and standardization. To provide users with a more immersive experience, 360-degree panoramic video requires higher resolution and frame rate than traditional two-dimensional (2D) video, thus requiring more bandwidth for transmission. Consequently, effective compression methods are highly desirable for storing and transmitting such large amounts of data. A common 360-degree video acquisition approach involves simultaneously recording multi-view video around the observer using multiple cameras. Then, through a series of operations such as image stitching, projection, and mapping, the visual information from the raw 360-degree sensor is mapped from a three-dimensional (3D) sphere to a 2D plane. This can be directly input into traditional video systems for compression without modifying the core codec. However, projection and mapping can cause distortion (or distortion) in the reconstructed video content. Existing work has focused on developing projection formats and advanced tools to improve coding performance. Specifically, the Joint Collaborative Team on Video Coding (JCT-VC) and the Joint Video Experts Team (JVET) are working to standardize 360-degree video compression, with the development of High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), respectively.

[0003] Because HEVC adopts a block-based video coding framework, prediction and post-transform quantization inevitably introduce severe distortion, such as blurring, blocking, and ringing artifacts. Consequently, the quality of the reconstructed video is significantly degraded, especially in low-bandwidth scenarios. To improve the visual quality of reconstructed video, many researchers have explored loop filtering techniques. For example, to reduce blocking artifacts, a deblocking filter first analyzes the distortion of each coding unit and then adaptively filters it. Sample-adaptive compensation reduces sample distortion by classifying reconstructed samples into different categories. For each category, a compensation value is derived at the encoder, encoded and transmitted in the bitstream, and compensation is applied to each pixel at the decoder. To further remove artifacts caused by block-based processing and quantization, researchers have also proposed other types of loop filtering techniques, such as adaptive loop filtering and non-local loop filtering.

[0004] Convolutional neural networks (CNNs) have achieved great success in solving image restoration tasks, and CNN-based in-loop filtering methods have been widely studied. However, most existing in-loop filtering methods primarily process perspective images on a regular 2D grid and do not consider the characteristics of 360-degree panoramic videos. Since the projection stage of 360-degree panoramic videos is non-uniform, for example, the equirectangular projection (ERP) samples the poles more densely than the equator, the filter size or shape of the convolutional layer should adapt to the latitude. Due to the limitations of ordinary CNN models with fixed filter sizes, directly applying existing in-loop filtering models to 360-degree panoramic video encoding does not achieve good performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a panoramic video image reconstruction method, device and equipment, aiming to solve the problem that the existing technology cannot provide an effective panoramic video image reconstruction method, resulting in poor quality and low efficiency of panoramic video image reconstruction.

[0006] In one aspect, the present invention provides a method for reconstructing a panoramic video image, the method comprising the following steps:

[0007] receiving a panoramic video projection image frame to be reconstructed, and performing feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame;

[0008] recalibrating the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features;

[0009] Based on the calibrated reconstruction features, the Y component image corresponding to the panoramic video projection image frame is restored to achieve reconstruction of the Y component image.

[0010] In another aspect, the present invention provides a panoramic video image reconstruction device, comprising:

[0011] a feature extraction unit, configured to receive a panoramic video projection image frame to be reconstructed, and perform feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame;

[0012] a recalibration unit, configured to recalibrate the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features; and

[0013] An image restoration unit is used to restore the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction feature to achieve reconstruction of the Y component image.

[0014] On the other hand, the present invention further provides a computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0015] The present invention receives a panoramic video projection image frame to be reconstructed, performs feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame, recalibrates the preliminary reconstruction features according to the panoramic video projection image frame to obtain calibrated reconstruction features, and then restores the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction features, thereby realizing the reconstruction of the Y component image of the panoramic video projection image frame, thereby improving the reconstruction quality of the panoramic video image and simultaneously improving the reconstruction efficiency of the panoramic video image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of the implementation of the panoramic video image reconstruction method provided by the first embodiment of the present invention;

[0017] Figure 2 Schematic diagram of obtaining a binary mask in the panoramic video image reconstruction method provided in the first embodiment of the present invention;

[0018] Figure 3 Schematic diagram of the structure of the ResNet network in the panoramic video image reconstruction method provided in the first embodiment of the present invention;

[0019] Figure 4 1 is a schematic diagram of the structure of the residual network in the panoramic video image reconstruction method provided in the first embodiment of the present invention;

[0020] Figure 5 is a flowchart of a method for reconstructing a panoramic video image according to a second embodiment of the present invention;

[0021] Figure 6 This is a flowchart of the training implementation of the ResNet network, residual network, and convolutional network in the panoramic video image reconstruction method provided in the second embodiment of the present invention;

[0022] Figure 7 2 is a schematic structural diagram of a panoramic video image reconstruction device provided by a fourth embodiment of the present invention;

[0023] Figure 8 It is a structural diagram of the computing device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] The following describes the specific implementation of the present invention in detail with reference to specific embodiments:

[0026] Example 1:

[0027] Figure 1 The implementation process of the panoramic video image reconstruction method provided by the first embodiment of the present invention is shown. For ease of explanation, only the part related to the embodiment of the present invention is shown, which is detailed as follows:

[0028] In step S101, a panoramic video projection image frame to be reconstructed is received, and feature extraction is performed on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame;

[0029] Embodiments of the present invention are applicable to computing devices, such as computers, or user terminals suitable for specific computing functions, such as panoramic video playback devices. The panoramic video projection image frames to be reconstructed are two-dimensional planar videos obtained by projecting the original panoramic video image using various projection methods. Specifically, they can be projected using projection methods such as ERP (Equi-Rectangular Projection), EAP (Equal-Area Projection), or AEP (Adjusted Equal-area Projection).

[0030] When performing feature extraction on a panoramic video projection image frame, preferably, the panoramic video projection image frame is subjected to coding unit (CU) block information extraction to obtain a binary mask corresponding to each CU, and the binary mask is used to guide the feature extraction of the panoramic video projection image frame to obtain regional features of different regions. Finally, the regional features of different regions are subjected to feature channel merging to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame, wherein the different regions are texture-rich regions and smooth regions, so that when feature extraction is subsequently performed based on the CU, the regional features of different regions are subjected to differentiated processing to improve the feature extraction efficiency. In a specific implementation process, when obtaining the binary mask corresponding to each CU, the CU in the tree coding unit (CTU) of the panoramic video projection image frame can be divided into two groups according to the depth level. For example, for each CTU, the pixels in the CU blocks with depth levels of 0 and 1 are assigned a value of 0, and the pixels in the CU blocks with depth levels of 2 and 3 are assigned a value of 1, thereby obtaining the binary mask corresponding to the CU, such as Figure 2As shown in (a), after processing all CTUs in the panoramic video projection image frame, the corresponding binary masks can be obtained, as shown in Figure 2 (b) is shown. Figure 2 (b) It can be seen that the area near the equator contains complex local texture information (white CU size is smaller), while the area near the poles is relatively smooth (black CU size is larger). From the equator to the poles, as the latitude increases, the distribution proportion of large-size CU blocks gradually increases.

[0031] Further preferably, when using a binary mask to guide the feature extraction of the panoramic video projection image frame, the binary mask is used to guide the trained ResNet network to extract the features of the panoramic video projection image frame to obtain regional features of different regions. The ResNet network is a symmetrical ResNet network with small convolution kernels and large convolution kernels. Specifically, the following method can be used: Figure 3 The ResNet network shown in the figure includes a branch network with a small convolution kernel (denoted as SB) and a ResNet network with a large convolution kernel (denoted as LB). After the first convolution of the input image (panoramic video projection image frame) D, it is input into the SB branch network and the LB branch network respectively. Specifically, there are four convolutions (Conv) and three activation functions (ReLU functions) in the SB branch. The convolution kernel size is 3. The outputs of the first two activations are multiplied with the binary mask M to obtain the features of a specific area. The third time only contains convolution operations. The output of the third convolution is added to the output of the first convolution and then output to the next convolution. The final convolution output is multiplied with the binary mask to obtain the output of the SB branch network. There are four convolutions (Conv) and three activation functions (ReLU functions) in the LB branch network. The outputs of the first two activations are multiplied with the binary mask M to obtain the features of a specific area. The third time only contains convolution operations. The output of the third convolution is added to the output of the first convolution and then output to the next convolution. The final convolution output is multiplied with the binary mask to obtain the output of the SB branch network. (in, ) to obtain the features of a specific area. The third convolution operation only contains the convolution operation. The third convolution output is added to the first convolution output and then output to the next convolution. The final convolution output is combined with the binary mask. The output of the LB branch network is obtained by multiplication. In a specific implementation, the convolution and activation functions in the SB branch network and the LB branch network can be set according to specific requirements.

[0032] In this way, the SB branch network is guided by the binary mask M corresponding to the panoramic video projection image frame to extract the features of the smooth area in the image, and the inverse mask obtained by the binary mask M is The LB branch network is guided to extract the features of the texture area in the image, and the quality of feature extraction is improved by region-by-region processing.

[0033] In step S102, the preliminary reconstructed features are recalibrated according to the panoramic video projection image frame to obtain calibrated reconstructed features;

[0034] In an embodiment of the present invention, the preliminary reconstructed features are recalibrated according to the panoramic video projection image frame to obtain calibrated reconstructed features, thereby readjusting the spatial information of individual features and inter-channel features of the panoramic video projection image frame to adapt to the current panoramic video projection image frame.

[0035] Preferably, when recalibrating the preliminary reconstructed features according to the panoramic video projection image frames, the preliminary reconstructed features are feature separated to obtain depth-separable image features, the depth-separable image features are recalibrated to obtain image features with different weights, and finally the image features with different weights and the preliminary reconstructed features are added together to obtain calibrated reconstructed features. Furthermore, the preliminary reconstructed features are recalibrated using a cascaded, trained residual network to obtain calibrated reconstructed features. The residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer. In a specific embodiment, as Figure 4 As shown, the residual network includes a feature separation layer 1, an adaptive recalibration layer 2, and a residual feature fusion layer 3. The feature separation layer 1 consists of depth convolution and point-by-point convolution to readjust the weight of each feature along the spatial dimension. The depth convolution is applied to the input feature map X. i-1 Each channel performs independent spatial convolution, and point-by-point convolution converts the feature map output by the depth convolution into Projection to new feature map Where H represents the height of the feature map, W represents the width of the feature map, Ci represents the number of channels, and R represents a real number. The adaptive recalibration layer 2 includes a global pooling layer, four cascaded fully connected layers (FC) and a Sigmoid activation function. The recalibrated feature map pass Transformation (Scale operation) is obtained, where represents the recalibrated j-channel feature, represents the calibration factor of channel j, Represents the features of the j channel before calibration. The feature map output by the residual feature fusion layer 3 in, represents the feature image of the convolution output after the feature is calibrated by the adaptive recalibration layer 2, X i-1 Represents the input features of the current residual network.

[0036] In step S103 , based on the calibrated reconstruction features, the Y component image corresponding to the panoramic video projection image frame is restored to achieve reconstruction of the Y component image.

[0037] In an embodiment of the present invention, preferably, when restoring the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstructed features, a trained convolutional network is used to gradually learn the calibrated reconstructed features to obtain the Y component image corresponding to the panoramic video projection image frame, and the convolutional network is composed of a preset number of convolutional layers.

[0038] The embodiment of the present invention receives a panoramic video projection image frame to be reconstructed, performs feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame, recalibrates the preliminary reconstruction features according to the panoramic video projection image frame to obtain calibrated reconstruction features, and then restores the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction features, thereby realizing the reconstruction of the Y component image of the panoramic video projection image frame, thereby improving the reconstruction quality of the panoramic video image and simultaneously improving the reconstruction efficiency of the panoramic video image.

[0039] Example 2:

[0040] Figure 5 The implementation process of the panoramic video image reconstruction method provided by the second embodiment of the present invention is shown. For ease of explanation, only the part related to the embodiment of the present invention is shown, which is detailed as follows:

[0041] In step S501, the coding unit block information is extracted from the panoramic video projection image frame to obtain a binary mask corresponding to each coding unit;

[0042] In step S502, a binary mask is used to guide a trained ResNet network to extract features from the panoramic video projection image frame to obtain regional features of different regions. The ResNet network is a symmetric ResNet network with small convolution kernels and large convolution kernels.

[0043] In step S503, feature channels are merged for regional features of different regions to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame;

[0044] In step S504, the preliminary reconstructed features are recalibrated using a cascaded, trained residual network to obtain calibrated reconstructed features. The residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer.

[0045] In step S505, the trained convolutional network is used to gradually learn the calibrated reconstructed features to obtain the Y component image corresponding to the panoramic video projection image frame, and the convolutional network is composed of a preset number of convolutional layers.

[0046] In the embodiment of the present invention, the specific implementation of steps S501 to S505 can refer to the corresponding description in the first embodiment, and will not be repeated here.

[0047] In an embodiment of the present invention, a binary mask is used to guide a trained ResNet network to perform feature extraction on a panoramic video projection image frame to identify distortion in the image, and partial convolution is performed to activate regional features of different regions, and then feature channels of the regional features of different regions are merged to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame. Thereafter, a cascaded, trained residual network is used to recalibrate the preliminary reconstruction features, thereby automatically adjusting inter-channel and intra-channel features to adapt to the current panoramic video projection image frame. Finally, a trained convolutional network is used to gradually learn the calibrated reconstruction features to obtain a Y component image corresponding to the panoramic video projection image frame, thereby improving the reconstruction quality of the panoramic video image and at the same time improving the reconstruction efficiency of the panoramic video image.

[0048] Example 3:

[0049] Figure 6 The following illustrates the training implementation process of the ResNet network, residual network, and convolutional network in the panoramic video image reconstruction method provided in the second embodiment of the present invention. For ease of illustration, only the parts related to the embodiment of the present invention are shown, which are detailed as follows:

[0050] In step S601, a training panoramic video projection image frame is received, and coding unit block information is extracted from the training panoramic video projection image frame to obtain a binary mask corresponding to each coding unit;

[0051] In an embodiment of the present invention, before using the panoramic video image reconstruction method of Example 1 or Example 2 to reconstruct the panoramic video projection image frame, it is necessary to train the ResNet network, residual network and convolutional network used to obtain trained ResNet network, residual network and convolutional network.

[0052] After receiving the training panoramic video projection image frame, the coding unit block information is extracted from the training panoramic video projection image frame to obtain a binary mask corresponding to each coding unit. The specific implementation of the coding unit block information extraction for the training panoramic video projection image frame can be found in the corresponding implementation of step 101 in Example 1 and is not further described here.

[0053] In step S602, a pre-built ResNet network is guided by a binary mask to extract features from the training panoramic video projection image frame to obtain regional features of different regions. The ResNet network is a symmetric ResNet network with small convolution kernels and large convolution kernels.

[0054] In the embodiment of the present invention, a ResNet network is pre-built to extract features from the training panoramic video projection image frames to obtain regional features of different regions. The ResNet network is a symmetrical ResNet network with small convolution kernels and large convolution kernels. For example, reference may be made to FIG. Figure 3 The ResNet network shown and the corresponding description in Example 1.

[0055] In step S603, feature channels of regional features of different regions are merged to obtain preliminary reconstructed features corresponding to the training panoramic video projection image frame;

[0056] In step S604, the preliminary reconstructed features are recalibrated using a cascaded, pre-built residual network to obtain calibrated reconstructed features. The residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer.

[0057] In an embodiment of the present invention, a residual network is pre-constructed to recalibrate the preliminary reconstruction features to obtain calibrated reconstruction features. The residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer. For example, reference may be made to Figure 4 The residual network shown and the corresponding description in Example 1.

[0058] In step S605, a pre-built convolutional network is used to gradually learn the calibrated reconstructed features to obtain a Y component image corresponding to the panoramic video projection image frame, where the convolutional network is composed of a preset number of convolutional layers;

[0059] In step S606, it is determined whether the preset number of training times is reached. If yes, the parameters of the ResNet network, the residual network, and the convolutional network are set to complete the training of the ResNet network, the residual network, and the convolutional network.

[0060] In step S607, if the preset number of training times is not reached, the weight of the pixel at each pixel position of the training panoramic video projection image frame is calculated, and the reconstruction loss of the Y component image is calculated based on the reconstructed Y component image, the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame, and the weight;

[0061] In an embodiment of the present invention, if the number of training times for the ResNet network, the residual network, and the convolutional network does not reach the preset number of training times, the weight of the pixel at each pixel position of the training panoramic video projection image frame is calculated. Specifically, the weight of the pixel at each pixel position of the training panoramic video projection image frame is calculated by the formula Calculate the weight of each pixel at position (i, j), where H represents the height of the training panoramic video projection image frame. Then, calculate the reconstruction loss of the Y component image based on the reconstructed Y component image, the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame, and the weight. Specifically, use the formula Calculate the reconstruction loss of the Y component image, where N represents the number of pixels, p represents the index of the pixel, its size is BxHxW, P represents the pixel set of the image, and Y(p) represents the Y component of the uncompressed image. The Y component image of the reconstructed image, G(p) represents the weight map, which can be calculated from g(i, j). Using this formula to calculate the reconstruction loss of the Y component image provides the direction for network convergence, making the reconstructed image quality closer to the original image quality (the quality of the image before compression), thereby improving overall compression performance.

[0062] In step S608, a field of view error between the reconstructed Y component image and the Y component image of the uncompressed image is calculated based on the viewport image of the reconstructed Y component image and the viewport image of the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame;

[0063] In the embodiment of the present invention, the field of view error between the reconstructed Y component image and the Y component image of the uncompressed image is calculated based on the viewport image of the reconstructed Y component image and the viewport image of the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame. Specifically, the formula is used.

[0064] Calculate the field of view error between the reconstructed Y component image and the Y component image of the uncompressed image, where V(p) represents the field of view image of the Y component image of the uncompressed image, The field of view image represents the Y component image of the reconstructed image. Calculating the field of view error using this formula can ensure that the network can converge in the direction of improving the quality of the field of view image, thereby improving the quality of the field of view image and achieving the purpose of improving the user's viewing experience.

[0065] In step S609, the parameters of the ResNet network, the residual network, and the convolutional network are updated according to the reconstruction loss and the field of view error, and the process jumps to step S601 to receive the panoramic video projection image frame for training and continue training the ResNet network, the residual network, and the convolutional network.

[0066] In an embodiment of the present invention, after obtaining the reconstruction loss and the field of view error, the parameters of the ResNet network, the residual network, and the convolutional network are updated according to the reconstruction loss and the field of view error, thereby combining the reconstruction loss and the field of view error to update the parameters of the ResNet network, the residual network, and the convolutional network, so that the network has better adaptability and reconstruction capabilities for images of different compression qualities, thereby improving the compression performance of the codec. Specifically, after obtaining the reconstruction loss and the field of view error, the formula is used Calculate the global reconstruction error that better reflects the reconstructed image. Then use the Adam algorithm to backpropagate the global reconstruction error, update the parameters of the ResNet network, residual network, and convolutional network, and then proceed to the next round of training. λ is preferably set to 0.5 to better reflect the loss of the reconstructed image.

[0067] In step S610, if the preset number of training times is reached, the parameters of the ResNet network, the residual network, and the convolutional network are set to complete the training of the ResNet network, the residual network, and the convolutional network;

[0068] In an embodiment of the present invention, during the training process of the ResNet network, the residual network, and the convolutional network, a binary mask is used to guide the ResNet network to extract features from the panoramic video projection image frame to identify distortion in the image, and partial convolution is performed to activate regional features of different regions. Feature channels of the regional features of different regions are then merged to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame. The preliminary reconstructed features are then recalibrated using a cascaded residual network to automatically adjust inter-channel and intra-channel features to adapt to the current panoramic video projection image frame. The calibrated reconstructed features are then gradually learned using a convolutional network to obtain a Y component image corresponding to the panoramic video projection image frame. When the number of training times is not reached, the global reconstruction error is backpropagated to update the parameters of the ResNet network, the residual network, and the convolutional network to enter the next round of training. This iterative training finally completes the training of the ResNet network, the residual network, and the convolutional network. During the training process, the network continuously converges in the direction of gradient descent, and ultimately obtains a converged network structure. This network structure can effectively adapt to the reconstruction of images of different qualities and improve overall compression performance.

[0069] Example 4:

[0070] Figure 7 The structure of the panoramic video image reconstruction device provided by the fourth embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, including:

[0071] The feature extraction unit 71 is configured to receive a panoramic video projection image frame to be reconstructed, and perform feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame;

[0072] a recalibration unit 72, configured to recalibrate the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features; and

[0073] The image restoration unit 73 is configured to restore the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction feature to achieve reconstruction of the Y component image.

[0074] In the embodiment of the present invention, each unit of the panoramic video image reconstruction device can be implemented by a corresponding hardware or software unit. Each unit can be an independent software or hardware unit or integrated into one software or hardware unit, which is not intended to limit the present invention.

[0075] Embodiment 5:

[0076] Figure 8 The structure of a computing device provided by the fifth embodiment of the present invention is shown. For ease of description, only the parts related to the embodiment of the present invention are shown.

[0077] The computing device 8 of the embodiment of the present invention includes a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned various panoramic video image reconstruction method embodiments are implemented, such as Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 7 The functions of the units 71 to 73 are shown.

[0078] The computing device of the embodiment of the present invention may be a computer or a user terminal suitable for specific computing functions, such as a panoramic video playback device. The steps implemented when the processor 80 of the computing device 8 executes the computer program 82 to implement the panoramic video image reconstruction method can be referred to the description of the aforementioned method embodiment and will not be repeated here.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A panoramic video image reconstruction method, characterized in that: The method comprises the following steps: receiving a panoramic video projection image frame to be reconstructed, and performing feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame; recalibrating the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features; Restoring the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction features to achieve reconstruction of the Y component image, including: using a trained convolutional network to gradually learn the calibrated reconstruction features to obtain the Y component image corresponding to the panoramic video projection image frame; The step of extracting features from the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame includes: Extracting coding unit block information from the panoramic video projection image frame to obtain a binary mask corresponding to each coding unit; Extracting features of the panoramic video projection image frame using the binary mask to guide the trained ResNet network, thereby obtaining regional features of different regions, wherein the ResNet network is a symmetrical ResNet network having a small convolution kernel and a large convolution kernel; Feature channels are merged on the regional features of the different regions to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame.

2. The method according to claim 1, wherein The step of recalibrating the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features includes: Performing feature separation on the preliminary reconstructed features to obtain depth-separable image features; recalibrating the depth-separable image features to obtain differently weighted image features; The differently weighted image features and the preliminary reconstruction features are added together to obtain the calibrated reconstruction features.

3. The method according to claim 2, wherein The step of recalibrating the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features includes: The preliminary reconstructed features are recalibrated using a cascaded, trained residual network to obtain calibrated reconstructed features, wherein the residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer.

4. The method according to claim 1, wherein The convolutional network consists of a preset number of convolutional layers.

5. The method according to claim 1, wherein: The steps of receiving a panoramic video projection image frame to be reconstructed and performing feature extraction on the panoramic video projection image frame are specifically as follows: Extracting coding unit block information from the panoramic video projection image frame to obtain a binary mask corresponding to each coding unit; Using the binary mask to guide the trained ResNet network to perform feature extraction on the panoramic video projection image frame to obtain regional features of different regions, the ResNet network is a symmetric ResNet network with small convolution kernels and large convolution kernels; Merging feature channels of the regional features of the different regions to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame; The step of recalibrating the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features is specifically as follows: Recalibrating the preliminary reconstructed features using a cascaded, trained residual network to obtain calibrated reconstructed features, wherein the residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer; The step of restoring the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction feature includes: The calibrated reconstructed features are gradually learned using a trained convolutional network to obtain a Y component image corresponding to the panoramic video projection image frame, wherein the convolutional network is composed of a preset number of convolutional layers.

6. The method according to claim 5, wherein Before the step of receiving the panoramic video projection image frame to be reconstructed and performing feature extraction on the panoramic video projection image frame, the method further includes: receiving a training panoramic video projection image frame, extracting coding unit block information from the training panoramic video projection image frame, and obtaining a binary mask corresponding to each coding unit; Using the binary mask to guide the pre-built ResNet network to perform feature extraction on the training panoramic video projection image frame to obtain regional features of different regions, the ResNet network is a symmetric ResNet network with small convolution kernels and large convolution kernels; Merging feature channels of the regional features of the different regions to obtain preliminary reconstructed features corresponding to the training panoramic video projection image frame; recalibrating the preliminary reconstructed features using a cascaded, pre-built residual network to obtain calibrated reconstructed features, wherein the residual network is a residual network based on a channel attention mechanism, including a feature separation layer, an adaptive recalibration layer, and a residual feature fusion layer; Using a pre-built convolutional network to gradually learn the calibrated reconstructed features to obtain a Y component image corresponding to the panoramic video projection image frame, the convolutional network consisting of a preset number of convolutional layers; Determine whether a preset number of training times is reached, and if so, set the parameters of the ResNet network, the residual network, and the convolutional network to complete the training of the ResNet network, the residual network, and the convolutional network; Otherwise, calculating the weight of a pixel at each pixel position of the training panoramic video projection image frame, and calculating the reconstruction loss of the Y component image based on the reconstructed Y component image, the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame, and the weight; Calculating a field of view error between the reconstructed Y component image and the Y component image of the uncompressed image based on the reconstructed viewport image of the Y component image and the viewport image of the Y component image of the uncompressed image corresponding to the training panoramic video projection image frame; The parameters of the ResNet network, the residual network, and the convolutional network are updated according to the reconstruction loss and the field of view error, and the process jumps to the step of receiving the panoramic video projection image frame for training until the training of the ResNet network, the residual network, and the convolutional network is completed.

7. A panoramic video image reconstruction device, characterized in that: The device comprises: a feature extraction unit, configured to receive a panoramic video projection image frame to be reconstructed, and perform feature extraction on the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame; a recalibration unit, configured to recalibrate the preliminary reconstructed features according to the panoramic video projection image frame to obtain calibrated reconstructed features; and An image restoration unit is configured to restore the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction feature to achieve reconstruction of the Y component image. When restoring the Y component image corresponding to the panoramic video projection image frame based on the calibrated reconstruction feature, the image restoration unit includes: using a trained convolutional network to gradually learn the calibrated reconstruction feature to obtain the Y component image corresponding to the panoramic video projection image frame; When the feature extraction unit extracts features from the panoramic video projection image frame to obtain preliminary reconstruction features corresponding to the panoramic video projection image frame, the feature extraction unit includes: Extracting coding unit block information from the panoramic video projection image frame to obtain a binary mask corresponding to each coding unit; Extracting features of the panoramic video projection image frame using the binary mask to guide the trained ResNet network, thereby obtaining regional features of different regions, wherein the ResNet network is a symmetrical ResNet network having a small convolution kernel and a large convolution kernel; Feature channels are merged on the regional features of the different regions to obtain preliminary reconstructed features corresponding to the panoramic video projection image frame.

8. The device according to claim 7, wherein The convolutional network consists of a preset number of convolutional layers.

9. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images