360-degree video transmission method and system based on neural and multi-scale feature fusion

Through the method of fusion of neural and multi-scale features, the feature extraction and encoding of 360-degree video streams is used to fusion the neural and multi-scale feature, and the feature extraction and encoding of 360-degree video streams is solved, which is difficult to maintain the quality of video streams under low bandwidth and realizes efficient video streams Transmission and reconstruction.

CN118660186BActive Publication Date: 2024-11-19NANJING UNIV OF INFORMATION SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124016.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing 360-degree video streaming technology is difficult to maintain high quality under low bandwidth conditions, and the existing Super-Resolution method has large model size and high computational consumption, so it cannot effectively process downsampling and reconstruction of video frames, and is easily affected by packet loss. .

Method used

Using a method based on neural and multi-scale feature fusion, multi-scale features are extracted through spatiotemporal downsampling models, and the masked automatic encoder of ViT-tiny backbone network is used to encode and reconstruct low-resolution video frames, combining PixelShuffle layer and volume Upsampling of the accumulated layers reduces data redundancy and improves visual quality.

Benefits of technology

It significantly reduces data redundancy during end-to-end transmission, improves the visual quality of reconstructed video frames, reduces the need for hardware resources, and maintains high-quality video streaming under low bandwidth conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118660186B_ABST
    Figure CN118660186B_ABST
Patent Text Reader

Abstract

The present invention discloses a 360-degree video transmission method and system based on neural and multi-scale feature fusion, which relates to the technical field of video stream transmission, including: acquiring a 360-degree video, preprocessing the 360-degree video to obtain a preprocessed 360-degree video, inputting the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and outputting multi-scale features and low-resolution video frames; mask encoding the low-resolution video frames to obtain processed mask frames, and reconstructing the mask frames to obtain reconstructed LR frames; inputting the reconstructed LR frames and multi-scale features into a pre-established upsampling super-resolution model, and outputting optimized super-resolution video frames, which can significantly reduce data redundancy in the end-to-end transmission process and achieve ideal visual quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video stream transmission, and in particular to a 360-degree video transmission method and system based on neural and multi-scale feature fusion. Background Art

[0002] In recent years, 360-degree videos have been sought after by a large number of viewers. However, high-quality 360-degree video streaming services require high network bandwidth rates, and the bandwidth in most parts of the world is still unable to support such services. Although there have been many studies on adaptive 360-degree video streaming in the past few years, most of them are based on tile or viewport prediction methods, which cannot get rid of the influence of the user's viewport and it is difficult to achieve accurate viewport prediction.

[0003] Existing video streaming methods based on Super-Resolution (SR) or reconstruction are not suitable for high-quality 360-degree video streaming. This is because, first of all, it is difficult to strike a balance between the downsampling ratio and the visual quality of the reconstructed video frame. Too large a downsampling ratio will greatly reduce the video quality. Although a conservative sampling strategy can ensure the quality of the reconstructed video, the reduction in bandwidth consumption is very limited. Secondly, high-performance SR models are large in size, the inference calculation process is quite time-consuming, and the hardware resources (GPU) of the client are very demanding. We need a low-cost, lightweight SR model. Third, packet loss may occur during transmission, which is particularly common in cases of poor network bandwidth and will seriously affect the visual quality of the reconstructed video frames. Summary of the invention

[0004] In order to address the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a 360-degree video transmission method and system based on neural and multi-scale feature fusion, which can significantly reduce data redundancy in the end-to-end transmission process and achieve ideal visual quality.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a 360-degree video transmission method based on neural and multi-scale feature fusion, the method comprising the following steps:

[0006] Acquire a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames;

[0007] The low-resolution video frame is mask-encoded to obtain a processed mask frame, and the mask frame is reconstructed to obtain a reconstructed LR frame;

[0008] The reconstructed LR frames and multi-scale features are input into a pre-established upsampling super-resolution model, and the optimized super-resolution video frames are output.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the preprocessing of the 360-degree video is performed by providing a standardized data format for frames for broadcast transmission.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the pre-established spatiotemporal downsampling model first performs temporal downsampling of the preprocessed 360-degree video with a preset step size, and performs downsampling of a preset ratio in the spatial dimension; extracts multi-scale features within the preprocessed 360-degree video through a residual block, and spatially downsamples the preprocessed 360-degree video through a residual block and a two-dimensional convolutional layer to obtain a low-resolution video frame.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the low-resolution video frame is masked and encoded by inputting it into a masked autoencoder network model based on the ViT-tiny backbone network, and tubular masking is used to mask the low resolution, the mask ratio is set, and then the masked frame is obtained and encoded, and then transmitted to the client. After the mask decoder on the client receives the masked frame, it uses the spatiotemporal joint attention mechanism of the ViT backbone network to reconstruct the video frame as a reconstructed LR frame.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the pre-established upsampling super-resolution model includes multiple residual blocks, a convolution layer and a PixelShuffle layer, and the reconstructed LR frame is upsampled through the convolution layer and the PixelShuffle layer to restore the size of the original video frame, and the reconstructed LR frame is fused with the multi-scale features provided by the server, and the number of channels is restored to three RGB channels through convolution.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: calculating the loss of the video frame as follows:

[0014] (1)

[0015] in, , , , are the values ​​of the pixel at the i-th or j-th position of the original video frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and The weight for calculating the loss is n, which is the total number of pixels between the original video frame and the final reconstructed super-resolution frame, and m is the total number of pixels between the LR frame and the reconstructed LR frame.

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: training the masked autoencoder network model based on the ViT-tiny backbone network by using the smoothness L1 loss formula conduct:

[0017] (2)

[0018] in, and are the values ​​of the pixels at the i-th or j-th positions of the original video frame and the final reconstructed super-resolution frame, respectively. is the hyperparameter weight.

[0019] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a 360-degree video transmission system based on neural and multi-scale feature fusion, comprising:

[0020] A video frame extraction module is used to obtain a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames;

[0021] The video frame reconstruction module performs mask encoding on the low-resolution video frame to obtain a processed mask frame, and reconstructs the mask frame to obtain a reconstructed LR frame;

[0022] The quality enhancement module is used to input the reconstructed LR frames and multi-scale features into a pre-established upsampling super-resolution model, and output an optimized super-resolution video frame.

[0023] In conjunction with the second aspect, in certain implementations of the second aspect, the system further includes: the video frame extraction module pre-processes the 360-degree video by providing a standardized data format for frames for broadcast transmission;

[0024] The pre-established spatiotemporal downsampling model in the video frame extraction module first performs temporal downsampling of the pre-processed 360-degree video with a preset step length, and performs downsampling of a preset magnification in the spatial dimension; the multi-scale features in the pre-processed 360-degree video are extracted through the residual block, and the pre-processed 360-degree video is spatially downsampled through the residual block and the two-dimensional convolution layer to obtain a low-resolution video frame;

[0025] In the video frame reconstruction module, the low-resolution video frame is masked and encoded by inputting it into the masked autoencoder network model based on the ViT-tiny backbone network, and the low resolution is masked using pipe masking, the mask ratio is set, and then the masked frame is obtained and encoded, and then transmitted to the client. After receiving the masked frame, the mask decoder on the client reconstructs the video frame using the spatiotemporal joint attention mechanism of the ViT backbone network as the reconstructed LR frame;

[0026] The pre-established upsampling super-resolution model in the quality enhancement module contains multiple residual blocks, convolutional layers, and a PixelShuffle layer. The reconstructed LR frames are upsampled through the convolutional layers and the PixelShuffle layers to restore the size of the original video frames, and the reconstructed LR frames are fused with the multi-scale features provided by the server. The number of channels is then restored to three RGB channels through convolution.

[0027] The loss of video frames in the quality enhancement module is calculated as follows:

[0028] (1)

[0029] in, , , , are the values ​​of the pixel at the i-th or j-th position of the original video frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and The weight for calculating the loss is n, which is the total number of pixels between the original video frame and the final reconstructed super-resolution frame, and m is the total number of pixels between the LR frame and the reconstructed LR frame.

[0030] The training of the mask autoencoder network model based on the ViT-tiny backbone network in the video frame reconstruction module is done through the smoothness L1 loss formula conduct:

[0031] (2)

[0032] in, and are the values ​​of the pixels at the i-th or j-th positions of the original video frame and the final reconstructed super-resolution frame, respectively. is the hyperparameter weight.

[0033] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the 360-degree video transmission method based on neural and multi-scale feature fusion as described above is adopted.

[0034] Beneficial effects of the present invention:

[0035] On the server side, the present invention designs a spatiotemporal downsampling network including multi-scale feature extraction to process video frames, preliminarily reduce data redundancy, and transmits multi-scale features to the client to cope with the packet loss problem during transmission and improve the visual quality of reconstructed video frames. A masked autoencoder based on the Masked Autoencoder (MAE) backbone network is designed, and an extremely high pipe masking rate is introduced, thereby improving the visualization quality of video frames. A masked autoencoder based on the MAE backbone network is designed, and an extremely high ratio of pipeline masks is introduced to mask the sampled video frames, further reducing data redundancy. On the client side, the masked decoder and the lightweight super-resolution model designed by us are used to reconstruct video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a schematic flow chart of the method of the present invention;

[0038] Figure 2 It is a schematic diagram of the overall framework of the 360-degree video stream optimization solution based on neural and multi-scale feature fusion of the present invention;

[0039] Figure 3 It is a schematic diagram of the network structure of the present invention;

[0040] Figure 4 It is a schematic diagram of the pipeline mask of the present invention;

[0041] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Embodiment 1:

[0044] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0045] 360-degree video: 360-degree video is when multiple cameras simultaneously shoot objects in the same space from multiple angles, and it is a website that supports multi-angle video playback or "360-degree video playback".

[0046] Multi-scale features: Multi-scale features refer to features observed and extracted at different scales. These features can reflect the attributes and information of objects in different spatial or temporal ranges.

[0047] like Figure 1 , 2 As shown, a 360-degree video transmission method based on neural and multi-scale feature fusion includes the following steps:

[0048] S101: Acquire a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames;

[0049] Among them, pre-processing the 360-degree video is to provide a standardized data format for frames used for broadcast transmission;

[0050] Unlike image data, video data has not only spatial redundancy, but also high temporal redundancy and correlation. Our goal is to smoothly transmit high-quality 360-degree video streams under the commonly used 4G bandwidth, so we first perform temporal downsampling of the 360-degree video with a step size of 2 to initially reduce the redundancy of data transmission. Downsampling only in the temporal dimension is not enough to reduce the redundant data to a tolerable bandwidth level. Therefore, we also perform downsampling at a rate of x2 in the spatial dimension. Considering that when processing image data, residual blocks can play a role in enhancing features, while helping to improve the representativeness and generalization of the network. We deploy more residual blocks (resblocks) throughout the system. We then transfer the resulting temporal downsampled frames to the downsampling module, which contains two residual block layers of different scales and a two-dimensional convolutional layer (Conv2d). Before and after the video frames are spatially downsampled by the convolutional layer and the residual block, we use the residual block to extract multi-scale features from the video frames. These features will be used for the reconstruction and quality enhancement of the video frames deployed in the client-side upsampling super-resolution model. As Figure 3 As shown in Figure 3, we set the feature sizes of the two extractions to 32 × 32 and 64 × 64 respectively.

[0051] S102: performing mask encoding on the low-resolution video frame to obtain a processed mask frame, and reconstructing the mask frame to obtain a reconstructed LR frame;

[0052] A masked autoencoder network based on the (Vision Transformer) ViT-tiny backbone network is designed, and its encoder (MaskedEncoder) and decoder (MaskedDecoder) are deployed on the server and client respectively.

[0053] After getting the downsampled low-resolution frame, we mask some of its regions. In fact, to save bandwidth, we only transmit a few unmasked fragments and then reconstruct most of the masked fragments on the client. Regarding the mask strategy, Figure 4As shown in the figure, we use tubular masking to mask low-resolution (LR) frames. At the same time, in order to further reduce data redundancy during transmission, we set an extremely high mask ratio. The mask position is spatially the same during the playback time of each video block. The purpose of this is to promote the learning ability of the model for frame reconstruction. Because if a patch is masked at a certain position in the current frame, but not in the next frame, this may cause the model to be unable to learn to reconstruct the patch at that position. After that, we will obtain the masked frame, encode it, and then transmit it to the client. After receiving the masked frame, the mask decoder on the client will reconstruct the video frame using the spatiotemporal joint attention mechanism of the ViT backbone network. In this way, we can get the reconstructed LR frame.

[0054] S103: input the reconstructed LR frame and multi-scale features into a pre-established upsampling super-resolution model, and output an optimized super-resolution video frame.

[0055] In this stage, we upsample the LR frames to obtain super-resolution video frames. To further improve the visual quality of the final reconstructed video frames, we cleverly introduce a multi-scale feature fusion method. This method works well in most image processing and video tasks. By extracting feature information from images of different scales, it can help improve the accuracy and robustness of the results. Although multi-scale features have been widely used in other fields, it is a bold attempt for 360-degree video streaming tasks. However, we will show the effectiveness of this method in the evaluation section later.

[0056] like Figure 3 As shown in the figure, our upsampling super-resolution model contains multiple residual blocks, convolutional layers, and a PixelShuffle layer. In fact, the model is very concise, without other redundant parts, and only uses a small number of lightweight modules, which can be easily deployed on the client. We mainly upsample the video frames through convolutional layers and PixelShuffle layers to restore the size of the original video frames. Before and after this, we fuse the reconstructed video frames with the multi-scale features provided by the server to improve the visual quality of the super-resolution frames. Finally, the number of channels is restored to three RGB channels through convolution.

[0057] In order to improve the feature extraction and fusion capabilities of the entire model, as well as the reconstruction capabilities of the client-side lightweight model, we designed a two-stage training method. In the first stage, we train the models on both ends as a whole to improve the feature extraction capabilities of the server-side model and the feature fusion capabilities of the client-side model. In this stage, we use MSE as the loss function to guide the training of the model. As shown in Formula 1, we specifically consider the loss between comparing the original frame and the final reconstructed super-resolution frame, as well as the loss between the downsampled low-resolution frame and the reconstructed low-resolution frame.

[0058] (1)

[0059] in, , , , are the values ​​of the pixel at the i-th or j-th position of the original frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and The weight for calculating the loss is n, which is the total number of pixels between the original video frame and the final reconstructed super-resolution frame, and m is the total number of pixels between the LR frame and the reconstructed LR frame. The specific training process is shown in Algorithm 1.

[0060] Before training begins, we first define a learnable parameter Used for feature fusion. Then the video frame is tiled into tiles, and each tile is divided into patches. Then the original patch is obtained. Perform spatiotemporal downsampling and extract multi-scale features , . Downsampled LR clip It is sent to the mask encoder for masking and encoding, and then transmitted to the client's decoder. The mask decoder reconstructs it to obtain the reconstructed patch Then, our SR model will The multi-scale features of are fused and upsampled to obtain super-resolution patches After the first phase of training, we will save the model and parameters. , used for the subsequent second stage training.

[0061] The first stage model training algorithm:

[0062] 1: Input: original video frame ; ;

[0063] in, is the frame of the original video, and are the two weight values ​​of formula 1

[0064] 2: Output: original patch ;

[0065] in, is the weight value of feature fusion (will be used below)

[0066] 3: Define learnable parameters

[0067] 4: Convert the original video frame Split into original tiles

[0068] in, Yes Each frame is cut into 8x8=64 tiles of the same size (video slices) in equal proportions.

[0069] 5: Divide the original tile into original patches

[0070] in, is defined within the model. Based on the Divided into 14x14=196 patches of the same size.

[0071] (The above are all preprocessing before training begins, and the following two loops are the subject of the training algorithm)

[0072] 6: Loop 1, executed for each epoch

[0073] 7: Loop 2, for each batch (batch_size) execute

[0074] 8: Spatiotemporal downsampling model processing get ; ;

[0075] , , is a spatiotemporal downsampling model. The results obtained after sampling are For low resolution patches, and is the multi-scale feature extracted.

[0076] 9: Mask encoder pair Mask and encode to get masked patches

[0077] Together with and Sent to the client

[0078] 10: Mask decoder pair Reconstruction is performed

[0079] Mask decoder pair Perform visual reconstruction operations to reconstruct the occluded patch and obtain the reconstructed patch

[0080] 11:

[0081] This step is the first fusion of multi-scale feature fusion

[0082] 12: Upsampling Super Resolution Model Pair Processing to obtain

[0083] After the first fusion, the model Perform super-resolution to get high-resolution patches

[0084] 13:

[0085] This step is the second fusion of multi-scale features.

[0086] 14: Conv2d Processing to obtain

[0087] After the second fusion, the 2D convolution layer is used to Process it, restore its three channels, and get the super-resolution patch

[0088] 15: End loop 2

[0089] 16:

[0090] After each round of the cycle, the loss is calculated using Formula 1, and the loss is passed to the model using back propagation to update the model parameters.

[0091] 17: Update model and

[0092] 18: End loop 1

[0093] After all rounds are completed, the model training is completed and the model parameters are saved

[0094] 19: Save the model

[0095] 20: Return ,

[0096] Finally, we will get and Reserved for Phase 2 training

[0097] After the first phase of training, we will save the model and parameters , which is used for the subsequent second-stage training. In the second stage, we fix the server-side model and only update the client-side model to improve its visual enhancement ability for reconstructed LR frames. In order to improve the overall quality of the reconstructed video frames and the user's subjective viewing experience, we use the Smoothness L1 loss function in this stage To guide the training of the model.

[0098] (2)

[0099] in, and Same definition as above, is the hyperparameter weight. The following is the second-stage training algorithm of the model we designed:

[0100] The second stage model training algorithm:

[0101] 1: Input: ; ;

[0102] is the weight value of formula 2

[0103] 2: Output: Super-resolution patches

[0104] 3: Initialization

[0105] 4: Fix the parameters of the server-side model so that it will not be updated, and update the client-side model separately

[0106] 5: Loop 1, for each round execution

[0107] 6: Loop 2, execute for each batch

[0108] 7: Server-side model processing get , ,

[0109] 8: Mask decoder pair Reconstruction is performed

[0110] 9:

[0111] 10: Upsampling Super Resolution Model Processing to obtain

[0112] 11:

[0113] 12: Conv2d Processing to obtain

[0114] 13: End loop 2

[0115] 14:

[0116] Here, Formula 2 is used as the loss function for this stage of training.

[0117] 15: Update model and , send the model to the client

[0118] 16: End loop 1

[0119] 17: Save the model

[0120] 18: Return

[0121] Experimental setup: We set the tile size of the 360-degree video frame to 8×8, the size of each tile to 224×224, and the patch size of the ViT backbone network to 16×16. We set the downsampling factor to x2 and the occlusion rate to 90%. On the server side, we use 4 residual blocks in two residual layers, and on the client side, we use 2 residual blocks. In the first stage of training, we set the data batch size to 128, the training epochs to 300, and the learning rate is reduced from 1e-4 to 1.5e-6 according to half-period cosine decay after 20 training epochs of warm-up. Weighting parameters and All are set to 0.5, the initial value Set to 0.7. In the second stage of training, we set the batch size to 64, the training epochs to 50, the learning rate to 1.5e-5, and other conditions remain unchanged. Weighting parameters Set to 0.5. In addition, at the end we will use the existing optical flow frame interpolation method to restore the reconstructed high-resolution video to the same number of frames as the original video.

[0122] Experimental Datasets: We selected three public datasets containing a large number of 360-degree videos to evaluate our method. (1) Contains 208 360-degree short videos, each of which is about 10-20 seconds long. We selected 200 of them and divided them into training and test datasets in a ratio of 9:1. In addition, we transferred the trained model to test in (2) containing 5 360-degree long videos and (3) containing 9 360-degree long videos. These videos contain various types of scenes, such as animals, games, sports games, wilderness, parks, beaches, etc.

[0123] Experimental Results: To evaluate the objective visual quality, we compared the performance of five methods on PSNR and SSIM metrics. As shown in Table 1, EDSR and MAE were originally used to process image tasks and did not achieve ideal results in 360-degree video transmission tasks. GAN360, a method based on a generative model, cannot reconstruct most of the details of 360-degree video frames well. Both MAESR360 and Masked360 performed better than other methods, which proves that mask-based super-resolution models are more suitable for 360-degree video streams. However, our method achieved satisfactory results, with PSNR and SSIM higher than the current state-of-the-art Masked360 by 3.66 dB and 0.027 dB, respectively. Our model utilizes additional multi-scale features and a clever two-stage training method, which helps to improve the visual quality of the final super-resolution frames.

[0124] In terms of data redundancy, we also achieved the lowest level of redundancy, and the downsampling method and pipe masking we used reduced the amount of data transmitted by 95.21%, which is 1.46% higher than the highest performance of the baseline method. It should be noted that the data of Masked360 comes from its paper, and the range of redundancy depends on whether it transmits additional patches.

[0125] Table 1 Comparative experiment

[0126]

[0127] In order to test the transfer learning ability of the model, we also conducted experiments on two other 360-degree video stream datasets. We transferred the model trained on dataset (1) to (2) and (3) for testing. The results of the transfer experiment on dataset (2) are shown in Table 2a, and the results of the transfer experiment on dataset (3) are shown in Table 2b. The final experimental results show that our model has good transfer ability and can still perform well on video types that did not appear in training. Although the other four baseline methods also showed small improvements, this may be due to the better video quality and longer video length of (2) and (3). Because longer temporal correlation is conducive to the model's prediction of future video frames. Our method can perform well on these two datasets, relying on its ability to extract and utilize multi-scale features, while paying attention to the global features and local details of the video frames.

[0128] Table 2a Migration experiments on dataset (2)

[0129]

[0130] Table 2b Migration experiments on dataset (3)

[0131]

[0132] We also conduct detailed ablation studies to demonstrate the effectiveness of our proposed model and method. We compare our method and its three variants (i.e., without transmitting multi-scale features, with feature weights set to a fixed value of 0.5, and only performing the first stage of training). The comparison results are shown in Table 3. It can be seen that when we do not transmit multi-scale features, the test results are not as good as Masked360. This is because 360-degree videos may experience packet loss during streaming, which affects the final video quality. When we add multi-scale features, the final results are better even if only one stage of training or fixed feature weight values ​​are performed. This proves that our proposed operation of adding multi-scale features during transmission is effective. Similarly, our designed two-stage training method can also improve the performance of the model.

[0133] Table 3 Ablation experiment

[0134]

[0135] Embodiment 2: In the second aspect, as Figure 5 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a 360-degree video transmission system based on neural and multi-scale feature fusion, comprising:

[0136] The video frame extraction module 11 is used to obtain a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames;

[0137] The video frame reconstruction module 12 performs mask encoding on the low-resolution video frame to obtain a processed mask frame, and reconstructs the mask frame to obtain a reconstructed LR frame;

[0138] The quality improvement module 13 is used to input the reconstructed LR frames and multi-scale features into a pre-established upsampling super-resolution model, and output an optimized super-resolution video frame.

[0139] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: pre-processing the 360-degree video in the video frame extraction module 11 to provide a standardized data format for frames for broadcast transmission;

[0140] The spatiotemporal downsampling model pre-established in the video frame extraction module 11 first performs temporal downsampling of the pre-processed 360-degree video with a preset step length, and performs downsampling of a preset magnification in the spatial dimension; multi-scale features in the pre-processed 360-degree video are extracted through the residual block, and the pre-processed 360-degree video is spatially downsampled through the residual block and the two-dimensional convolution layer to obtain a low-resolution video frame;

[0141] The video frame reconstruction module 12 performs mask encoding on the low-resolution video frame and inputs it into the mask autoencoder network model based on the ViT-tiny backbone network, and uses pipe masking to mask the low resolution, sets the mask ratio, and then obtains and encodes the masked frame, and then transmits it to the client. After receiving the masked frame, the mask decoder on the client reconstructs the video frame using the spatiotemporal joint attention mechanism of the ViT backbone network as the reconstructed LR frame;

[0142] The upsampling super-resolution model pre-established in the quality enhancement module 13 includes multiple residual blocks, convolution layers and a PixelShuffle layer. The reconstructed LR frame is upsampled through the convolution layer and the PixelShuffle layer to restore the size of the original video frame, and the reconstructed LR frame is fused with the multi-scale features provided by the server, and the number of channels is restored to three RGB channels through convolution;

[0143] The loss of the video frame in the quality improvement module 13 is calculated as follows:

[0144] (1)

[0145] in, , , , are the values ​​of the pixel at the i-th or j-th position of the original video frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and The weight of the loss is calculated, where n is the total number of pixels in the original video frame and the final reconstructed super-resolution frame, and m is the total number of pixels in the LR frame and the reconstructed LR frame. The training of the mask autoencoder network model based on the ViT-tiny backbone network in the video frame reconstruction module 12 is performed through the smoothness L1 loss formula conduct:

[0146] (2)

[0147] in, and are the values ​​of the pixels at the i-th or j-th positions of the original video frame and the final reconstructed super-resolution frame, respectively. is the hyperparameter weight.

[0148] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0149] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A 360-degree video transmission method based on neural and multi-scale feature fusion, characterized in that: The method comprises the following steps: Acquire a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames; The low-resolution video frame is mask-encoded to obtain a processed mask frame, and the mask frame is reconstructed to obtain a reconstructed LR frame; The reconstructed LR frames and multi-scale features are input into a pre-established upsampling super-resolution model, and an optimized super-resolution video frame is output; The pre-established upsampling super-resolution model includes multiple residual blocks, convolution layers and a PixelShuffle layer. The reconstructed LR frame is upsampled through the convolution layer and the PixelShuffle layer to restore the size of the original video frame, and the reconstructed LR frame is fused with the multi-scale features provided by the server, and the number of channels is restored to three RGB channels through convolution.

2. The 360-degree video transmission method based on neural and multi-scale feature fusion according to claim 1, characterized in that: The 360-degree video is pre-processed by providing a standardized data format for frames for broadcast transmission.

3. The 360-degree video transmission method based on neural and multi-scale feature fusion according to claim 1, characterized in that: The pre-established spatiotemporal downsampling model first performs temporal downsampling of a preset step length on the pre-processed 360-degree video, and performs downsampling of a preset ratio in the spatial dimension; The multi-scale features in the preprocessed 360-degree video are extracted through the residual block, and the preprocessed 360-degree video is spatially downsampled through the residual block and the two-dimensional convolutional layer to obtain a low-resolution video frame.

4. The 360-degree video transmission method based on neural and multi-scale feature fusion according to claim 1, characterized in that: The low-resolution video frame is masked and encoded by inputting it into a masked autoencoder network model based on the ViT-tiny backbone network, and a pipeline masking method is introduced to mask the low resolution, set the mask ratio, and then obtain the masked frame and encode it, and then transmit it to the client. After the mask decoder on the client receives the masked frame, it uses the spatiotemporal joint attention mechanism of the ViT backbone network to reconstruct the video frame as the reconstructed LR frame.

5. The 360-degree video transmission method based on neural and multi-scale feature fusion according to claim 1, characterized in that: The loss Loss1 of the video frame is calculated as follows: (1) in, , , , are the values ​​of the pixel at the i-th or j-th position of the original video frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and The weight for calculating the loss is n, which is the total number of pixels between the original video frame and the final reconstructed super-resolution frame, and m is the total number of pixels between the LR frame and the reconstructed LR frame.

6. The 360-degree video transmission method based on neural and multi-scale feature fusion according to claim 4, characterized in that: The training of the masked autoencoder network model based on the ViT-tiny backbone network is carried out using the smoothness L1 loss formula conduct: (2) in, and are the values ​​of the pixels at the i-th or j-th positions of the original video frame and the final reconstructed super-resolution frame, respectively. is the hyperparameter weight.

7. A 360-degree video transmission system based on neural and multi-scale feature fusion, characterized in that: include: A video frame extraction module is used to obtain a 360-degree video, preprocess the 360-degree video to obtain a preprocessed 360-degree video, input the preprocessed 360-degree video into a pre-established spatiotemporal downsampling model, and output multi-scale features and low-resolution video frames; The video frame reconstruction module performs mask encoding on the low-resolution video frame to obtain a processed mask frame, and reconstructs the mask frame to obtain a reconstructed LR frame; A quality enhancement module, which is used to input the reconstructed LR frames and multi-scale features into a pre-established upsampling super-resolution model, and output an optimized super-resolution video frame; The pre-established upsampling super-resolution model in the quality enhancement module contains multiple residual blocks, convolutional layers and a PixelShuffle layer. The reconstructed LR frames are upsampled through the convolutional layers and the PixelShuffle layers to restore the size of the original video frames, and the reconstructed LR frames are fused with the multi-scale features provided by the server. The number of channels is then restored to three RGB channels through convolution.

8. The 360-degree video transmission system based on neural and multi-scale feature fusion according to claim 7, characterized in that: The video frame extraction module pre-processes the 360-degree video by providing a standardized data format for the frames for broadcast transmission; The pre-established spatiotemporal downsampling model in the video frame extraction module first performs temporal downsampling of the pre-processed 360-degree video with a preset step length, and performs downsampling of a preset ratio in the spatial dimension; The multi-scale features in the pre-processed 360-degree video are extracted through the residual block, and the pre-processed 360-degree video is spatially downsampled through the residual block and the two-dimensional convolution layer to obtain a low-resolution video frame; In the video frame reconstruction module, the low-resolution video frame is masked and encoded by inputting it into the masked autoencoder network model based on the ViT-tiny backbone network, and the low resolution is masked using pipe masking, the mask ratio is set, and then the masked frame is obtained and encoded, and then transmitted to the client. After receiving the masked frame, the mask decoder on the client reconstructs the video frame using the spatiotemporal joint attention mechanism of the ViT backbone network as the reconstructed LR frame; The loss of video frames in the quality enhancement module is calculated as follows: (1) in, , , , are the values ​​of the pixel at the i-th or j-th position of the original video frame, the final reconstructed super-resolution frame, the downsampled LR frame, and the reconstructed LR frame, respectively. yes and Calculate the weight of the loss, yes and Calculate the weight of the loss, n is the total number of pixels of the original video frame and the final reconstructed super-resolution frame, m is the total number of pixels of the LR frame and the reconstructed LR frame; The training of the mask autoencoder network model based on the ViT-tiny backbone network in the video frame reconstruction module is done through the smoothness L1 loss formula conduct: (2) in, and are the values ​​of the pixels at the i-th or j-th positions of the original video frame and the final reconstructed super-resolution frame, respectively. is the hyperparameter weight.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the 360-degree video transmission method based on neural and multi-scale feature fusion described in any one of claims 1 to 6 is adopted.