Immersive video super-resolution method and related devices

Abstract

The application provides an immersive video super-resolution method, which decodes a video file, assigns independent key frame propagation branches to key frames to obtain key frame features, so that the information recorded in the key frames can be fully utilized, and assigns forward propagation branches and backward propagation branches to the key frames, so that the non-key frames can further utilize the information recorded in the key frames; the forward propagation branches and the backward propagation branches corresponding to the non-key frames are alternately arranged, unnecessary propagation branches are reduced, and the operation amount required in panoramic video super-resolution is reduced; after the features corresponding to the key frames and the non-key frames are obtained, super-resolution is performed on the key frames and the non-key frames respectively, high-definition key frames, first high-definition non-key frames and second high-definition non-key frames are obtained, and the high-definition key frames, the first high-definition non-key frames and the second high-definition non-key frames are spliced into a high-definition immersive video file.

Description

Technical Field

[0001] This application relates to the field of video super-resolution technology, and in particular to an immersive video super-resolution method and related equipment. Background Technology

[0002] With technological advancements and social innovation, immersive video technology is gaining increasing popularity. Unlike traditional videos that capture scenes from a single perspective, immersive videos achieve comprehensive scene coverage through multi-angle shooting from the same location. Viewers can use specialized head-mounted displays to control the viewing angle with head movements, or explore the video on a computer screen using a mouse and keyboard, breaking the limitations of traditional video perspectives and providing an immersive viewing experience. Compared to traditional videos, panoramic videos typically require 4K or 8K resolution transmission to deliver a satisfactory immersive experience, which not only increases data volume but also places higher demands on network bandwidth.

[0003] In traditional video transmission, in order to reduce the server bandwidth burden and enable a wider range of users to access immersive videos, it is currently common to transmit low-quality videos over the Internet and then perform super-resolution on the client side to transmit traditional videos.

[0004] Currently, super-resolution models for traditional videos have a relatively complex structure. Directly applying them to the super-resolution operation of immersive videos will significantly increase the amount of data that the client device needs to process, resulting in excessively long super-resolution processing time for immersive videos on client devices with poor performance. Summary of the Invention

[0005] In view of this, the purpose of this application is to propose an immersive video super-resolution method and related equipment.

[0006] To achieve the above objectives, this application provides an immersive video super-resolution method, comprising: decoding a video file and determining keyframes and non-keyframes; assigning keyframe propagation branches, forward propagation branches, and backward propagation branches to keyframes; wherein, the keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe; the forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame; the backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame; assigning forward and backward propagation branches to non-keyframes, wherein each non-keyframe is assigned only one of the forward and backward propagation branches, and the propagation branches assigned to two adjacent non-keyframes are different; and performing super-resolution on the keyframes based on the keyframe features, the first feature, and the second feature corresponding to the keyframes to obtain high-definition keyframes. Based on the first feature corresponding to the non-keyframe, super-resolution is performed on the non-keyframe to obtain the first high-definition non-keyframe. Based on the second feature corresponding to the non-keyframe, super-resolution is performed on the non-keyframe to obtain the second high-definition non-keyframe. Based on the high-definition keyframe, the first high-definition non-keyframe, and the second high-definition non-keyframe, a high-definition immersive video file is obtained.

[0007] In some embodiments, the keyframe propagation branch includes a keyframe optical flow estimation module, a keyframe feature alignment module, and a keyframe residual module. Specifically, the keyframe propagation branch is configured to: input a keyframe and a previous keyframe into the keyframe optical flow estimation module, which compares the feature differences between the keyframe and the previous keyframe and outputs keyframe difference information; input the keyframe difference information and the keyframe features of the previous keyframe into the keyframe feature alignment module, which distorts the previous keyframe features to align them with the keyframe difference information and outputs keyframe alignment information; and input the keyframe and the keyframe alignment information into the keyframe residual module, which outputs keyframe features.

[0008] In some embodiments, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module, and a first residual module. Specifically, the forward propagation branch is configured to: input the current frame and the previous frame including the forward propagation branch into the first optical flow estimation module; the first optical flow estimation module compares the feature differences between the current frame and the previous frame including the forward propagation branch and outputs first difference information; input the first difference information and the first feature output by the previous forward propagation branch into the first feature alignment module; the first feature alignment module distorts the first feature output by the previous forward propagation branch to align the first feature output by the previous forward propagation branch with the first difference information and outputs first alignment information; input the current frame and the first alignment information into the first residual module; the first residual module outputs the first feature.

[0009] In some embodiments, the backpropagation branch includes a second optical flow estimation module, a second feature alignment module, and a second residual module. Specifically, the backpropagation branch is configured to: input the current frame and the next frame including the backpropagation branch into the second optical flow estimation module; the second optical flow estimation module compares the feature differences between the current frame and the next frame including the backpropagation branch and outputs second difference information; input the second difference information and the second feature output by the next backpropagation branch into the second feature alignment module; the second feature alignment module distorts the second feature output by the next backpropagation branch to align the next second feature with the second difference information and outputs second alignment information; and input the current frame and the second alignment information into the second residual module, which outputs the second feature.

[0010] In some embodiments, the keyframe is super-resolution based on the keyframe features, the first feature, and the second feature corresponding to the keyframe, specifically including: rearranging the keyframe features, the first feature, and the second feature, and super-resolution of the keyframe based on a sub-pixel convolution method.

[0011] In some embodiments, super-resolution of keyframes based on subpixel convolution further includes: calculating the difference between each point in the high-resolution image of the keyframe and the keyframe based on the Chabonier loss function, as the loss value.

[0012] In some embodiments, after decoding the video file and determining keyframes and non-keyframes, the method further includes: segmenting the keyframes to obtain segmented images. The segmented images closest to the first edge of the keyframe are then stitched to the second edge of the keyframe. The first edge and the second edge can be stitched together to form a continuous image.

[0013] In some embodiments, the system includes: a decoding module for decoding a video file and determining keyframes and non-keyframes; a propagation branch allocation module for allocating keyframe propagation branches, forward propagation branches, and backward propagation branches to keyframes. The keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame. Forward and backward propagation branches are also allocated to non-keyframes, with each non-keyframe allocated only one of the forward or backward propagation branches, and adjacent non-keyframes are assigned different propagation branches. A keyframe super-resolution module is used to perform super-resolution on the keyframes based on the keyframe features, the first feature, and the second feature corresponding to the keyframes, to obtain high-definition keyframes. The first super-resolution module performs super-resolution on non-keyframes based on a first feature corresponding to the non-keyframes, resulting in a first high-definition non-keyframe. The second super-resolution module performs super-resolution on non-keyframes based on a second feature corresponding to the non-keyframes, resulting in a second high-definition non-keyframe. The video generation module generates a high-definition immersive video file based on the high-definition keyframe, the first high-definition non-keyframe, and the second high-definition non-keyframe.

[0014] In some embodiments, this application also provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the foregoing embodiments.

[0015] In some embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods described above.

[0016] As can be seen from the above embodiments, the immersive video super-resolution method provided in this application decodes the video file and assigns independent keyframe propagation branches to keyframes to obtain keyframe features, thereby ensuring that the information recorded in the keyframes can be fully utilized; and assigns forward propagation branches and backward propagation branches to keyframes, so that non-keyframes can further utilize the information recorded in the keyframes; by alternating the forward propagation branches and backward propagation branches corresponding to non-keyframes, unnecessary propagation branches are reduced, thereby reducing the amount of computation required for panoramic video super-resolution; after obtaining the features corresponding to keyframes and non-keyframes, super-resolution is performed on keyframes and non-keyframes respectively to obtain high-definition keyframes, first high-definition non-keyframes, and second high-definition non-keyframes, and they are stitched together into a high-definition immersive video file. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an immersive video super-resolution method provided in this application embodiment;

[0019] Figure 2 A schematic diagram illustrating a propagation branch allocation method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the propagation branch structure provided in the embodiments of this application;

[0021] Figure 4 A schematic diagram of the structure of an immersive video super-resolution device provided in another embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0024] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0027] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] Immersive video, typically referring to spherical or panoramic video, is a cutting-edge multimedia presentation technique. Unlike traditional video that captures scenes from a single perspective, panoramic video achieves comprehensive scene coverage by shooting from multiple angles from the same location. Viewers can use specialized head-mounted displays to control the viewing angle through head movements, or explore the video on a computer screen using a mouse and keyboard, breaking the limitations of traditional video perspectives and providing an immersive viewing experience. With technological advancements and social innovation, panoramic video technology is gaining increasing popularity and is being used in various industries such as education, healthcare, tourism, retail, and entertainment. For example, in the tourism industry, panoramic video allows users to experience scenery from around the world from home; in the entertainment industry, panoramic video combined with motion seats can simulate experiences like roller coasters, giving users a sense of being there. The maturity of the technology and the decrease in cost have driven the popularization of panoramic video, allowing users to view panoramic content through various devices, including professional head-mounted displays, computer screens, and even mobile phone screens. However, the high data volume of panoramic video poses challenges to network transmission. Compared to traditional video, panoramic video typically needs to be transmitted at 4K or 8K resolution to provide a satisfactory immersive experience. This not only increases the amount of data but also places higher demands on network bandwidth. For example, an 8K video stream may require at least 100Mbps of bandwidth.

[0030] In the field of computer vision, super-resolution technology has attracted much attention in recent years. This technology uses algorithms to upsample low-resolution images or videos to high resolution, and is widely used in video restoration, deblurring, noise reduction, and other fields, and is gradually being extended to video transmission. By performing super-resolution processing on low-resolution videos on the client side, rather than directly transmitting high-resolution videos from the server, the bandwidth burden on the server side can be significantly reduced, allowing users to enjoy high-definition videos even with limited bandwidth.

[0031] As society's demands for panoramic video experience quality increase, super-resolution technology becomes particularly crucial in assisting users with limited resources. The value of researching panoramic video super-resolution technology lies in its ability to effectively reduce the amount of data required for panoramic video transmission. This allows users, even in ordinary network environments, to receive lower-quality video data and perform super-resolution processing locally, thereby enjoying high-quality, high-resolution panoramic videos and enhancing immersion and realism. However, it is worth noting that the time required for video super-resolution processing is also a significant factor affecting the user's viewing experience; excessively long processing times may cause users to abandon watching.

[0032] Therefore, the aim of this study is to develop a more efficient deep learning-driven panoramic video super-resolution algorithm by optimizing the video super-resolution network architecture. The algorithm aims to lower the barrier for users to enjoy immersive content, promote the continued development of the panoramic video field, and enable a wider range of users to access immersive experiences.

[0033] Currently, commonly used super-resolution models often have complex structures, which limits their practical deployment and application. Video super-resolution technology inherently involves processing massive datasets, and panoramic video, due to its high resolution requirements, requires significantly more data processing compared to traditional video formats, thus significantly increasing the task's complexity. To address this issue, this paper aims to simplify traditional video super-resolution models and proposes a super-resolution method for panoramic video that accelerates the inference process while maintaining reasonable quality.

[0034] Furthermore, current video super-resolution algorithms primarily target planar videos, with limited research on panoramic videos. However, due to the unique characteristics of panoramic videos, preprocessing is necessary before inputting the panoramic video into the model to better utilize the information within each frame.

[0035] This invention addresses the complexity of information propagation mechanisms in existing video super-resolution models by simplifying the structure and optimizing the processing flow. Specifically, this paper preprocesses panoramic videos to improve information utilization. Furthermore, it proposes a simplified bidirectional propagation strategy, which, compared to the original strategy, has fewer propagation branches, thus reducing computational load and improving the efficiency of super-resolution training and inference. Meanwhile, keyframes are crucial in video coding, containing the most complete information. This paper incorporates super-resolution of keyframes to ensure that video quality remains largely unchanged. Based on this, this research designs a deep learning-based panoramic video super-resolution method that uses incomplete bidirectional propagation and super-resolution of keyframes to accelerate super-resolution inference while maintaining super-resolution quality.

[0036] like Figure 1 As shown, this application provides an immersive video super-resolution method, including:

[0037] Step S1: Decode the video file and determine keyframes and non-keyframes.

[0038] In this embodiment, the video file is a panoramic video, which is encoded into a video file after being unfolded from a 360° panoramic video into a planar video. When playing the video file, the player decodes the panoramic video and maps the planar video onto the panoramic space, thereby enabling the playback of the panoramic video. Keyframes contain complete video information, while non-keyframes are frames similar to keyframe images during video compression. By deleting some duplicate information, the amount of information in the video is reduced. By referring to keyframes or other non-keyframes, the complete image of the current non-keyframe can be reconstructed.

[0039] Before inputting panoramic video into a super-resolution network, preprocessing is necessary. This is because super-resolution neural networks typically receive input as a sequence of images, while encoded video is not presented as images. The video encoding stream contains three frame types for compressing video data: I-frames (keyframes), P-frames (predictive frames), and B-frames (bidirectional prediction frames). I-frames are encoded independently, containing complete image data, resulting in larger file sizes, but allowing playback to begin from any I-frame. P-frames rely on preceding I-frames or P-frames, storing only difference information and reconstructing the current frame by referencing the previous frame, thus improving compression efficiency. B-frames reference information from both preceding and following frames, achieving even higher compression efficiency by storing inter-frame difference information, but the decoding process is more complex. Clearly, I-frames (keyframes) contain more complete and reliable information than other P-frames and B-frames (non-keyframes), and the decoder can easily obtain the keyframe's location information during decoding, annotating the keyframes at output based on the decoder's information.

[0040] Step S2 involves assigning a keyframe propagation branch, a forward propagation branch, and a backward propagation branch to each keyframe. The keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame that includes the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame that includes the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame.

[0041] In video super-resolution processing, propagation refers to the process of transferring and sharing information between multiple video frames. By utilizing information from previous and subsequent frames, motion and details can be captured more effectively, enhancing the quality of the current frame. This application employs three propagation branches: a forward propagation branch, a backward propagation branch, and a keyframe propagation branch. These branches have the same structure, taking the current frame, the previous frame (reference frame), and feature information from the previous propagation branch as input, and outputting the feature information obtained by the current propagation branch, which is then passed to the aggregation and upsampling module of the next propagation branch and the current frame. The forward propagation branch propagates feature information forward in time, the backward propagation branch propagates feature information backward in time, and the keyframe propagation branch propagates keyframe information forward in time.

[0042] Step S3: Assign forward propagation branches and backward propagation branches to non-key frames, and assign only one of the forward propagation branches and backward propagation branches to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different.

[0043] like Figure 2 As shown, the calculation of propagation branches is quite complex. Designing two propagation paths (forward and backward) for each frame would require numerous convolution operations, which are time-consuming and computationally demanding, hindering immersive content experiences for users with lower hardware performance. Therefore, this application assigns three propagation branches—keyframe propagation branch, forward propagation branch, and backward propagation branch—to keyframes, and sets forward and backward propagation branches for non-keyframe intervals. This reduces the computational load required for video super-resolution while maintaining video super-resolution quality. Furthermore, assigning keyframe propagation branches independent of the forward and backward propagation branches allows for better utilization of the complete information recorded in the keyframes, improving the quality of video super-resolution.

[0044] Based on this, by connecting the keyframe propagation branch, the forward propagation branch, and the backward propagation branch, and traversing all the three propagation branches corresponding to all frames in the order of keyframes and non-keyframes, the three propagation branches output the frame features of the corresponding frames respectively. Based on the frame features corresponding to each frame, super-resolution is performed on the current frame to obtain a high-definition image.

[0045] Step S4: Based on the keyframe features, the first feature, and the second feature corresponding to the keyframe, perform super-resolution on the keyframe to obtain a high-definition keyframe.

[0046] Step S5: Based on the first feature corresponding to the non-key frame, perform super-resolution on the non-key frame to obtain the first high-definition non-key frame.

[0047] Step S6: Based on the second feature corresponding to the non-key frame, perform super-resolution on the non-key frame to obtain the second high-definition non-key frame.

[0048] Step S7: Based on the high-definition keyframe, the first high-definition non-keyframe, and the second high-definition non-keyframe, a high-definition immersive video file is obtained.

[0049] This method involves encoding high-definition keyframes, a first high-definition non-keyframe, and a second high-definition non-keyframe according to the original keyframe and non-keyframe arrangement to obtain high-definition immersive video. Building upon the steps above, this method can also save video transmission bandwidth by transmitting low-resolution immersive video to a terminal device and then performing super-resolution on the low-resolution immersive video. Furthermore, it can perform super-resolution on panoramic video files with low original resolution to obtain clearer immersive video images.

[0050] In some embodiments, the keyframe propagation branch includes a keyframe optical flow estimation module, a keyframe feature alignment module, and a keyframe residual module.

[0051] In this embodiment, as Figure 3 As shown, the keyframe optical flow estimation module, the keyframe feature alignment module, and the keyframe residual module are connected in sequence. The current keyframe, the previous keyframe, and the keyframe features of the previous keyframe are used as inputs. After processing by the keyframe optical flow estimation module, the keyframe feature alignment module, and the keyframe residual module, the keyframe features of the current keyframe are output.

[0052] The keyframe propagation branch is specifically configured to: input the keyframe and the previous keyframe into the keyframe optical flow estimation module; the keyframe optical flow estimation module compares the feature differences between the keyframe and the previous keyframe and outputs the keyframe difference information.

[0053] In this application, due to motion and deformation in the video, the image content between frames analyzed by the propagation branch may be shifted and distorted. Each propagation branch in this application has the same structure as a traditional propagation branch, such as... Figure 3 As shown, x i This represents the current input frame, which will be compared with the previous frame x in the current propagation direction. i-1 Together, they are input into the keyframe optical flow estimation module, which is responsible for analyzing the differences in features between the two frames.

[0054] The keyframe difference information and the keyframe features of the previous keyframe are input into the keyframe feature alignment module. The keyframe feature alignment module distorts the previous keyframe features to align the previous keyframe features with the keyframe difference information and outputs the keyframe alignment information.

[0055] Among them, the keyframe difference information calculated by the keyframe optical flow estimation module will be combined with the keyframe features f from the previous keyframe propagation branch. i-1 The features are input together into the keyframe feature alignment module. This module uses the keyframe difference information to distort the features from the previous frame to achieve alignment with the current frame, and finally outputs the keyframe alignment information.

[0056] The keyframes and keyframe alignment information are input into the keyframe residual module, which then outputs the keyframe features.

[0057] In this process, by inputting the keyframe alignment information and the current frame together into the residual module, the output keyframe features are used for subsequent propagation and aggregation and upsampling of the current frame.

[0058] In some embodiments, such as Figure 3 As shown, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module, and a first residual module.

[0059] The forward propagation branch is specifically configured to: input the current frame and the previous frame including the forward propagation branch into the first optical flow estimation module; the first optical flow estimation module compares the feature differences between the current frame and the previous frame including the forward propagation branch and outputs the first difference information.

[0060] The first difference information and the first feature output by the previous forward propagation branch are input to the first feature alignment module. The first feature alignment module distorts the first feature output by the previous forward propagation branch so that the first feature output by the previous forward propagation branch is aligned with the first difference information, and outputs the first alignment information.

[0061] The current frame and the first alignment information are input into the first residual module, and the first residual module outputs the first feature.

[0062] In this embodiment, similar to the keyframe propagation branch, the forward propagation branch takes the current frame, the previous frame, and the first feature of the previous frame as input, and processes them through the first optical flow estimation module, the first feature alignment module, and the first residual module to output the first feature of the current frame.

[0063] It should be noted that since the forward propagation branch can be assigned to either a keyframe to output the first feature of the keyframe or a non-keyframe to output the first feature of the non-keyframe, the types of the previous frame and the current frame need to be determined according to the arrangement order of the keyframes and non-keyframes. The processing methods for the previous frame and the current frame of different types can be the same or different.

[0064] In some embodiments, such as Figure 3As shown, the backpropagation branch includes a second optical flow estimation module, a second feature alignment module, and a second residual module.

[0065] The backpropagation branch is specifically configured to: input the current frame and the next frame including the backpropagation branch into the second optical flow estimation module; the second optical flow estimation module compares the feature differences between the current frame and the next frame including the backpropagation branch and outputs the second difference information.

[0066] The second difference information and the second feature output by the next backpropagation branch are input into the second feature alignment module. The second feature alignment module distorts the second feature output by the next backpropagation branch so that the next second feature is aligned with the second difference information, and outputs the second alignment information.

[0067] The current frame and the second alignment information are input into the second residual module, and the second residual module outputs the second feature.

[0068] In this embodiment, similar to the forward propagation branch, the backward propagation branch obtains the second feature of the current frame based on the second feature of the current frame, the next frame, and the second feature of the next frame. Both the next frame and the current frame can be key frames or non-key frames.

[0069] In some embodiments, step S4 specifically includes:

[0070] Step S41: The keyframe features, the first feature, and the second feature are rearranged, and the keyframe is super-resolution based on the sub-pixel convolution method.

[0071] In this embodiment, after obtaining the keyframe features, the first feature, and the second feature, the keyframe can be super-resolutiond using a sub-pixel convolution method, namely the pixel-shuffle method. Specifically, it is necessary to generate r from the keyframe. 2 The feature maps of size H×W×C are rearranged to generate a high-resolution image of size rH×rW×C. Here, r is the upsampling magnification, H and W are the height and width of the image, respectively, and C represents the number of color channels (e.g., an RGB image has three color channels). The upsampling process can be formally described as follows:

[0072]

[0073] PS refers to Pixel-Shuffle operation, I SR and I LR These represent the generated high-resolution image and the original low-resolution image, respectively. = is used to represent I. SR In other words, how is the image restored using super-resolution generated? Looking from right to left, PS(I) LR This refers to performing Photoshop operations on low-resolution images. This refers to the keyframe generation r 2 Each channel (which can be understood as different channels performing r) 2 The feature map (sub-feature extraction) has a size of H×W×C, I rH×rW×C It refers to a high-resolution image of size rH×rW×C obtained by rearranging the feature maps.

[0074] Preferably, step S5 may further include:

[0075] Step S51: The first feature is rearranged, and the non-key frame is super-resolution based on the sub-pixel convolution method to obtain the first high-definition non-key frame.

[0076] Preferably, step S6 may further include:

[0077] Step S61: The second feature is rearranged, and super-resolution of non-keyframes is performed based on the sub-pixel convolution method.

[0078] The super-resolution operation for non-keyframes based on the Pixel-Shuffle method is the same as the super-resolution operation for keyframes described above, and will not be repeated here.

[0079] In some embodiments, step S41 further includes:

[0080] Step S411 calculates the difference between each point in the high-resolution keyframe image and the keyframe based on the Chabonier loss function, and uses it as the loss value.

[0081] In this embodiment, when converting panoramic video to planar video, to facilitate video file storage, the panoramic video is usually cut along its meridians and stretched to convert it into a rectangular image for storage. Therefore, the stretching and deformation are more severe in areas closer to the poles of the panoramic video, resulting in lower reliability of residual calculations for these areas. To avoid inaccurate residual calculations due to the stretching, weights need to be assigned to each point in the panoramic video image. The weight of a point (i, j) is defined by the following formula:

[0082]

[0083] The loss value is calculated based on the Charbonnier loss function, and the calculation formula is as follows:

[0084]

[0085] Where M and N represent the width and height of the image, respectively. I represents a point on the predicted image. i,j It represents a point on a real image.

[0086] In some embodiments, the method further includes the following steps after step S1:

[0087] Step S11: Segment the keyframes to obtain the segmented image.

[0088] Step S12: The segmented image near the first edge of the keyframe is stitched to the second edge of the keyframe.

[0089] The first edge and the second edge can be stitched together to form a continuous image.

[0090] In this embodiment, due to the panoramic video mapping, the main subject in the panoramic video frame may be located at the edge, making it impossible to effectively utilize continuous information. Therefore, appropriate image stitching is necessary. Specifically, the panoramic video frame can be divided into several equal parts, and the segmented image from one side of the frame can be copied and stitched to the other side. Super-resolution can then be applied to the stitched image. Optionally, to improve the super-resolution effect, the frame can be divided into a grid, and at least one edge of the frame can be copied and stitched to the other edge. When the frame is too complex, the number of stitched edges can be increased as appropriate. Alternatively, one corner of the frame can be stitched to a corresponding corner. The stitching method can be reasonably modified according to the actual application requirements.

[0091] As can be seen from the above embodiments of this application, this application decodes the video file and assigns independent keyframe propagation branches to keyframes to obtain keyframe features, thereby ensuring that the information recorded in the keyframes can be fully utilized; and assigns forward propagation branches and backward propagation branches to keyframes, so that non-keyframes can further utilize the information recorded in the keyframes; by alternating the forward propagation branches and backward propagation branches corresponding to non-keyframes, unnecessary propagation branches are reduced, thereby reducing the amount of computation required for panoramic video super-resolution; after obtaining the features corresponding to keyframes and non-keyframes, super-resolution is performed on keyframes and non-keyframes respectively to obtain high-definition keyframes, first high-definition non-keyframes and second high-definition non-keyframes, and they are stitched together into a high-definition immersive video file.

[0092] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0093] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] Based on the same inventive concept, and corresponding to any of the above embodiments, this application also provides an immersive video super-resolution device.

[0095] refer to Figure 4 The immersive video super-resolution device includes:

[0096] The decoding module is used to decode video files and determine keyframes and non-keyframes.

[0097] The propagation branch allocation module is used to allocate keyframe propagation branches, forward propagation branches, and backward propagation branches to keyframes. Specifically, the keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame.

[0098] Furthermore, forward propagation branches and backward propagation branches are assigned to non-key frames, and only one of the forward propagation branch and backward propagation branch is assigned to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different.

[0099] The keyframe super-resolution module is used to perform super-resolution on keyframes based on the keyframe features, first features, and second features corresponding to the keyframes, so as to obtain high-definition keyframes.

[0100] The first super-resolution module is used to perform super-resolution on non-key frames based on the first feature corresponding to the non-key frames, so as to obtain the first high-definition non-key frames.

[0101] The second super-resolution module is used to perform super-resolution on non-key frames based on the second feature corresponding to the non-key frames, so as to obtain the second high-definition non-key frames.

[0102] The video generation module is used to obtain a high-definition immersive video file based on high-definition keyframes, a first high-definition non-keyframe, and a second high-definition non-keyframe.

[0103] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0104] The apparatus described above is used to implement the corresponding immersive video super-resolution method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0105] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement an immersive video super-resolution method as described in any of the above embodiments.

[0106] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0107] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0108] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0109] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0110] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0111] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0112] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0113] The electronic devices described above are used to implement a corresponding immersive video super-resolution method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute an immersive video super-resolution method as described in any of the above embodiments.

[0115] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0116] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute an immersive video super-resolution method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0117] It should be noted that the embodiments of this application can also be further described in the following ways:

[0118] An immersive video super-resolution method includes:

[0119] Decode the video file to identify keyframes and non-keyframes.

[0120] Assign keyframe propagation branches, forward propagation branches, and backward propagation branches to keyframes. The keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame that includes the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame that includes the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame.

[0121] Assign forward and backward propagation branches to non-key frames, and assign only one of the forward and backward propagation branches to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different.

[0122] Based on the keyframe features, first feature, and second feature corresponding to the keyframe, super-resolution is performed on the keyframe to obtain high-definition keyframes.

[0123] Based on the first feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain the first high-definition non-key frame.

[0124] Based on the second feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain the second high-definition non-key frame.

[0125] A high-definition immersive video file is obtained based on high-definition keyframes, a first high-definition non-keyframe, and a second high-definition non-keyframe.

[0126] Optionally, the keyframe propagation branch includes a keyframe optical flow estimation module, a keyframe feature alignment module, and a keyframe residual module.

[0127] The keyframe propagation branch is specifically configured to: input the keyframe and the previous keyframe into the keyframe optical flow estimation module; the keyframe optical flow estimation module compares the feature differences between the keyframe and the previous keyframe and outputs the keyframe difference information.

[0128] The keyframe difference information and the keyframe features of the previous keyframe are input into the keyframe feature alignment module. The keyframe feature alignment module distorts the previous keyframe features to align the previous keyframe features with the keyframe difference information and outputs the keyframe alignment information.

[0129] The keyframes and keyframe alignment information are input into the keyframe residual module, which then outputs the keyframe features.

[0130] Optionally, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module, and a first residual module.

[0131] The forward propagation branch is specifically configured to: input the current frame and the previous frame including the forward propagation branch into the first optical flow estimation module; the first optical flow estimation module compares the feature differences between the current frame and the previous frame including the forward propagation branch and outputs the first difference information.

[0132] The first difference information and the first feature output by the previous forward propagation branch are input to the first feature alignment module. The first feature alignment module distorts the first feature output by the previous forward propagation branch so that the first feature output by the previous forward propagation branch is aligned with the first difference information, and outputs the first alignment information.

[0133] The current frame and the first alignment information are input into the first residual module, and the first residual module outputs the first feature.

[0134] Optionally, the backpropagation branch includes a second optical flow estimation module, a second feature alignment module, and a second residual module.

[0135] The backpropagation branch is specifically configured to: input the current frame and the next frame including the backpropagation branch into the second optical flow estimation module; the second optical flow estimation module compares the feature differences between the current frame and the next frame including the backpropagation branch and outputs the second difference information.

[0136] The second difference information and the second feature output by the next backpropagation branch are input into the second feature alignment module. The second feature alignment module distorts the second feature output by the next backpropagation branch so that the next second feature is aligned with the second difference information, and outputs the second alignment information.

[0137] The current frame and the second alignment information are input into the second residual module, and the second residual module outputs the second feature.

[0138] Optionally, based on the keyframe features, the first feature, and the second feature corresponding to the keyframe, super-resolution of the keyframe is performed, specifically including:

[0139] The keyframe features, the first feature, and the second feature are rearranged, and the keyframes are super-resolution based on the subpixel convolution method.

[0140] Optionally, the super-resolution of keyframes based on the sub-pixel convolution method also includes:

[0141] The difference between each point in the high-resolution keyframe image and the keyframe is calculated based on the Charbonnier loss function and used as the loss value.

[0142] Optionally, after decoding the video file and determining keyframes and non-keyframes, the process may also include:

[0143] Segment the keyframes to obtain the segmented image.

[0144] The segmented image closest to the first edge of the keyframe is stitched to the second edge of the keyframe.

[0145] The first edge and the second edge can be stitched together to form a continuous image.

[0146] This application also provides an immersive video super-resolution device, comprising:

[0147] The decoding module 601 is used to decode video files and determine key frames and non-key frames.

[0148] The propagation branch allocation module 602 is used to allocate keyframe propagation branches, forward propagation branches, and backward propagation branches to keyframes. Specifically, the keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame.

[0149] Furthermore, forward propagation branches and backward propagation branches are assigned to non-key frames, and only one of the forward propagation branch and backward propagation branch is assigned to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different.

[0150] The keyframe super-resolution module 603 is used to perform super-resolution on the keyframe based on the keyframe features, the first feature, and the second feature corresponding to the keyframe, so as to obtain a high-definition keyframe.

[0151] The first super-resolution module 604 is used to perform super-resolution on non-key frames based on the first feature corresponding to the non-key frames to obtain the first high-definition non-key frames.

[0152] The second super-resolution module 605 is used to perform super-resolution on non-key frames based on the second feature corresponding to the non-key frames to obtain second high-definition non-key frames.

[0153] The video generation module 606 is used to obtain a high-definition immersive video file based on high-definition keyframes, a first high-definition non-keyframe, and a second high-definition non-keyframe.

[0154] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above-mentioned embodiments.

[0155] This application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods described above.

[0156] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0157] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0158] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0159] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An immersive video super-resolution method, comprising: Decode the video file to identify keyframes and non-keyframes; A keyframe propagation branch, a forward propagation branch, and a backward propagation branch are assigned to the keyframe; wherein, the keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe; the forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame; the backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame; The forward propagation branch and the backward propagation branch are assigned to the non-key frames, and only one of the forward propagation branch and the backward propagation branch is assigned to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different. Based on the key frame features corresponding to the key frame, the first feature, and the second feature, super-resolution is performed on the key frame to obtain a high-definition key frame; Based on the first feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain the first high-definition non-key frame. Based on the second feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a second high-definition non-key frame. Based on the high-definition keyframe, the first high-definition non-keyframe, and the second high-definition non-keyframe, a high-definition immersive video file is obtained.

2. The immersive video super-resolution method according to claim 1, wherein, The keyframe propagation branch includes a keyframe optical flow estimation module, a keyframe feature alignment module, and a keyframe residual module; The keyframe propagation branch is specifically configured to: input the keyframe and the previous keyframe into the keyframe optical flow estimation module, and the keyframe optical flow estimation module compares the feature differences between the keyframe and the previous keyframe and outputs keyframe difference information; The keyframe difference information and the keyframe feature of the previous keyframe are input to the keyframe feature alignment module. The keyframe feature alignment module distorts the previous keyframe feature to align the previous keyframe feature with the keyframe difference information and outputs keyframe alignment information. The keyframe and its alignment information are input into the keyframe residual module, and the keyframe residual module outputs the keyframe features.

3. The immersive video super-resolution method according to claim 1, wherein, The forward propagation branch includes a first optical flow estimation module, a first feature alignment module, and a first residual module; The forward propagation branch is specifically configured to: input the current frame and the previous frame including the forward propagation branch into the first optical flow estimation module, and the first optical flow estimation module compares the feature differences between the current frame and the previous frame including the forward propagation branch, and outputs the first difference information. The first difference information and the first feature output by the previous forward propagation branch are input to the first feature alignment module. The first feature alignment module distorts the first feature output by the previous forward propagation branch so that the first feature output by the previous forward propagation branch is aligned with the first difference information, and outputs the first alignment information. The current frame and the first alignment information are input to the first residual module, and the first residual module outputs the first feature.

4. The immersive video super-resolution method according to claim 1, wherein, The backpropagation branch includes a second optical flow estimation module, a second feature alignment module, and a second residual module; The backpropagation branch is specifically configured to: input the current frame and the next frame including the backpropagation branch into the second optical flow estimation module, and the second optical flow estimation module compares the feature differences between the current frame and the next frame including the backpropagation branch and outputs second difference information; The second difference information and the second feature output by the next backpropagation branch are input to the second feature alignment module. The second feature alignment module distorts the second feature output by the next backpropagation branch so that the next second feature is aligned with the second difference information, and outputs the second alignment information. The current frame and the second alignment information are input to the second residual module, and the second residual module outputs the second feature.

5. The immersive video super-resolution method according to claim 1, wherein, The super-resolution of the keyframe based on the keyframe features corresponding to the keyframe, the first feature, and the second feature specifically includes: The keyframe features, the first feature, and the second feature are rearranged, and the keyframe is super-resolution performed based on a subpixel convolution method.

6. The immersive video super-resolution method according to claim 5, wherein, The sub-pixel convolution-based super-resolution method for the keyframes also includes: The difference between each point in the high-resolution image of the keyframe and the keyframe is calculated based on the Charbonnier loss function and used as the loss value.

7. The immersive video super-resolution method according to claim 1, wherein, After decoding the video file and determining keyframes and non-keyframes, the following steps are also included: The keyframes are segmented to obtain segmented images; The segmented image near the first edge of the keyframe is stitched to the second edge of the keyframe; The first edge and the second edge can be stitched together to form a continuous image.

8. An immersive video super-resolution device, comprising: The decoding module is used to decode video files and determine keyframes and non-keyframes; A propagation branch allocation module is used to allocate keyframe propagation branches, forward propagation branches, and backward propagation branches to the keyframes; wherein, the keyframe propagation branch is configured to calculate the keyframe features of the current keyframe based on the previous keyframe, the keyframe features of the previous keyframe, and the current keyframe; the forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame; the backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame; Furthermore, the forward propagation branch and the backward propagation branch are assigned to the non-key frames, and only one of the forward propagation branch and the backward propagation branch is assigned to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different; A keyframe super-resolution module is used to perform super-resolution on the keyframe based on the keyframe features corresponding to the keyframe, the first feature and the second feature, to obtain a high-definition keyframe. The first super-resolution module is used to perform super-resolution on the non-key frame based on the first feature corresponding to the non-key frame to obtain the first high-definition non-key frame. The second super-resolution module is used to perform super-resolution on the non-key frame based on the second feature corresponding to the non-key frame to obtain a second high-definition non-key frame. The video generation module is used to obtain a high-definition immersive video file based on the high-definition keyframe, the first high-definition non-keyframe, and the second high-definition non-keyframe.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.

Images