Video generation device and method thereof

Abstract

A video generation device obtains Laplacian pyramid levels for 360° video frames; promotes each obtained Laplacian pyramid level to restore a corresponding set of Gaussian pyramid levels for each 360° video frame; and, for each pixel in each 360° video frame, determines a restored Gaussian pyramid level from the restored set of Gaussian pyramid levels based on a live broadcast bitrate budget and a viewport area by executing a pyramid level allocation process. A 360° video with heterogeneous spatial quality is synthesized by aggregating the restored Gaussian pyramid levels determined for each pixel in each 360° video frame. Therefore, when a user views the 360° video, the bitrate can be reduced while providing the user with high video quality. Furthermore, the present invention relates to a download agent including such a video generation device, a corresponding method, and a computer program.

Description

Technical Field

[0001] The invention relates to a video generation device and a download agent comprising such a video generation device. In addition, the invention also relates to a corresponding method and a computer program. Background Art

[0002] 360° (or 360-degree) video, also known as omnidirectional video, is spherical video with ultrahigh definition (UHD) resolution that includes an omnidirectional view of the scene, allowing users to freely locate their position within the video. Due to the high resolution and frame rate of 360° video, video streaming is a challenging task. These videos are encoded in 4k, 8k, 12k, or up to 24k resolutions, requiring high frame rates to match the HMD refresh rate to prevent motion sickness. However, due to the user's head movement and the average field of view (FoV) of 120° for head-mounted displays (HMDs), users viewing 360° videos through HMDs can only see a portion of the entire 360° video at any given moment (on average, one-third). This results in a waste of approximately 66% of bandwidth. Furthermore, to display the entire video with a 4k (3840x2160) FoV, the video stream must be at least 12k (11520x6480) video. For example, an 8K video at 60 frames per second (fps) encoded with the High Efficiency Video Coding (HEVC) codec has a bitrate of 100Mbps, but a 12K 360° video at 60fps requires a bandwidth of 418Mbps or more to enable advanced virtual reality (VR) experiences and deliver the full, high-quality 360° video, most of which the user will never see.

[0003] To reduce bandwidth requirements and associated bandwidth waste when delivering 360° videos, viewport adaptive streaming technology has been introduced to enable traditional rate adaptive streaming (DASH) technology to adapt to the user's head orientation, thereby providing the highest quality HMD heterogeneous video in the viewport area and lower quality HMD heterogeneous video in other areas, such as areas that are less likely to be seen. The disadvantage of existing heterogeneous video representations is that they are only suitable for video on demand. Therefore, existing heterogeneous video cannot support 360° video live broadcasting, for example, it cannot meet the needs of video conferencing over 5G networks, such as providing users with high-quality video under low bandwidth, nor can it achieve low-latency real-time video production and transmission when generating and recording 360° live video. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a solution to alleviate or solve the shortcomings and problems of traditional solutions.

[0005] Another object of an embodiment of the present invention is to provide a solution to achieve higher bandwidth utilization than traditional solutions.

[0006] The aforementioned and other objects are achieved by the subject-matter of the independent claims. Further advantageous embodiments of the invention can be found in the dependent claims.

[0007] According to a first aspect of the present invention, the aforementioned and other objects are achieved by a video generation device, the video generation device being configured to:

[0008] Get the Laplacian pyramid levels of the 360° video frame;

[0009] Upgrading each of the obtained Laplacian pyramid levels to recover a corresponding set of Gaussian pyramid levels for each 360° video frame;

[0010] For each pixel in each 360° video frame, determine a restored Gaussian pyramid level from the restored set of Gaussian pyramid levels by performing a pyramid level allocation process according to a live broadcast bitrate budget and a viewport area;

[0011] By aggregating the restored Gaussian pyramid levels determined for each pixel in each 360° video frame, a 360° video with heterogeneous spatial quality is synthesized.

[0012] A video can be understood as a sequence of video frames.

[0013] It is understood that the video generation device may include a processing circuit for executing the steps according to the embodiments of the present invention. The processing circuit may be integrated into a single device or distributed across two or more separate devices.

[0014] In addition, the viewport in the present disclosure can be understood as the current viewport of the client user, which is used to receive the 360° video with heterogeneous spatial quality generated according to an embodiment of the present invention.

[0015] The video generation device described in the first aspect has the advantage of reducing the bitrate required for 360° video (or equivalently improving bandwidth utilization) while providing users with high video quality when viewing 360° videos. Furthermore, this can reduce bandwidth waste caused by providing high-quality video for the entire 360° video when only an average of 1 / 3 of the video is viewed at any given moment. Furthermore, the video generation device described in the first aspect can reduce the transmission latency of high-quality 360° video. When producing heterogeneous 360° videos, the viewport area uses a high bitrate allowed by the available bandwidth, while other areas use lower bitrates through the Laplacian pyramid levels, thereby reducing bandwidth waste, increasing encoding speed, and reducing processing latency. By using a high bitrate allowed by the available bandwidth for the viewport when producing heterogeneous 360° videos, it is also beneficial to achieve higher viewport image quality for heterogeneous 360° videos than existing state-of-the-art solutions within the same bitrate budget. The quality of the videos generated by the video generation device described in the first aspect even improves as the bitrate budget increases.

[0016] In an implementation form of the video generation device according to the first aspect, the video generation device is further configured to:

[0017] The 360° composite video with heterogeneous spatial quality is produced for live broadcast according to the bit rate budget and the viewport area.

[0018] The advantage of the described implementation is that the synthesis of heterogeneous 360° videos can be adjusted according to the available bitrate and the current viewport area. For example, if the network bandwidth is low, the bitrate of the 360° video can be reduced by generating very low-quality 360° video content in areas outside the viewport area. Alternatively, if the bandwidth is large, the 360° video can be synthesized by allocating higher quality to areas outside the viewport area, that is, making the total video bitrate fit the bitrate budget. Network bandwidth utilization is optimized by prioritizing viewport quality. In addition, the heterogeneous video is reconstructed every time the viewport changes. The calculated Gaussian pyramid levels can be reused to synthesize heterogeneous videos with different bitrate budgets and viewport areas for a group of clients.

[0019] In an implementation form of the video generation device according to the first aspect, the lifting of each acquired Laplacian pyramid level includes:

[0020] from Starting from the level, each Laplace pyramid level obtained Move to the next level resolution, and after the improvement Add on the level levels, so as to restore the corresponding Gaussian pyramid level for each 360° video frame, where i represents the current level and N represents the total number of Gaussian pyramid levels.

[0021] The advantage of this implementation is that it encodes Laplacian pyramid levels rather than Gaussian pyramid levels and recovers the Gaussian pyramid from the received Laplacian pyramid, thereby increasing encoding speed, reducing the bandwidth required to transmit the pyramid levels, and overall reducing end-to-end latency. The improved pyramid is computed only once for each video slice and sent to the download agent, allowing it to be reused to synthesize various 360° video representations based on the client's viewport position and bitrate budget.

[0022] In an implementation form of the video generation device according to the first aspect, the video generation device is further configured to:

[0023] The bit rate of each boosted Gaussian pyramid level is calculated according to the sum of the bit rates of the boosted Laplacian levels.

[0024] The calculated bit rate may be an approximation of the bit rate of each boosted Gaussian pyramid level.

[0025] The advantage of this implementation is that encoding the Laplacian pyramid levels can be avoided, further reducing video processing time. The pyramid level allocation algorithm needs to use the increased bit rate of the encoded Laplacian pyramid levels to map the pyramid levels to the regions of interest in the 360° video.

[0026] In an implementation form of the video generation device according to the first aspect, the pyramid level allocation process includes:

[0027] (a) calculating the surface bit rate of each promoted Gaussian pyramid level according to the bit rate of the promoted Gaussian pyramid level;

[0028] (b) measuring the spherical area of ​​the viewport region of the 360° video frame;

[0029] (c) multiplying the surface bit rate of the improved Gaussian pyramid level by the area of ​​the spherical surface to obtain a bit rate required to allocate the optimal pyramid level to the viewport area;

[0030] (d) Mapping the viewport area to the highest quality pyramid level that does not exceed the bitrate budget.

[0031] The advantage of this implementation is that this process can be performed every time the viewport position changes, so that the video representation can be reconstructed using the same upgraded pyramid level for the new viewport position and bitrate budget. This implementation prioritizes image quality in the viewport area.

[0032] In an implementation form of the video generation device according to the first aspect, the pyramid level allocation process further includes:

[0033] Repeat steps (a) to (d) for each region of interest in the 360° video frame according to the priority of the region of interest.

[0034] The advantage of this implementation is that it prioritizes the quality of the viewport area and allocates the highest possible pyramid level to the viewport area so that the bitrate of the entire synthesized video is lower than the bitrate budget. The quality of areas outside the viewport is prioritized because the client is unlikely to request video content outside the viewport, at least not in the short term. This implementation reduces the priority of areas outside the viewport by mapping these areas to lower levels of the restored Gaussian pyramid. The video information corresponding to the lower Gaussian pyramid levels is sufficient to cope with sudden head movements as described below. If the client user suddenly and quickly moves their head, the user will see the video content corresponding to the Gaussian pyramid level allocated to the requested area outside the viewport before the network responds to the client's viewport change request.

[0035] In an implementation form of the video generation device according to the first aspect, synthesizing a 360° video with heterogeneous spatial quality includes:

[0036] The optimal restored Gaussian pyramid level determined for each pixel in each 360° video frame is summarized according to the following formula:

[0037]

[0038] Among them, O G is the output video with heterogeneous spatial quality corresponding to the quality distribution q(·), is the improved Laplace pyramid level i, B is the bit rate budget, is the mapping function that maps the quality distribution q(·) to the optimal level of the Laplacian pyramid for a given bitrate budget B for each pixel in each 360° video frame, where (x, y) are the coordinates of the pixel in the 360° video frame.

[0039] The advantage of this implementation is that it allows for pixel-level control of 360° video synthesis quality. Specifically, given a viewport location and an overall bitrate budget, the optimal Laplacian pyramid level is determined for each pixel in each video frame. Consequently, compared to existing state-of-the-art solutions (at the same bitrate), video quality within the viewport is improved for a given bitrate budget, and the bandwidth required to transmit 360° videos between proxies is lower than that required to transmit the original video.

[0040] According to a second aspect of the present invention, the aforementioned and other objects are achieved by a download agent of a communication system, wherein the download agent includes the video generation device according to an embodiment of the present invention.

[0041] The download agent may act as a server providing services to one or more clients in a communication system.

[0042] According to a third aspect of the present invention, the foregoing and other objects are achieved by a method for a video generating device, the method comprising:

[0043] Get the Laplacian pyramid levels of the 360° video frame;

[0044] Upgrading each of the obtained Laplacian pyramid levels to recover a corresponding set of Gaussian pyramid levels for each 360° video frame;

[0045] For each pixel in each 360° video frame, a restored Gaussian pyramid level is determined from a corresponding set of Gaussian pyramid levels by executing a pyramid level allocation process based on the live broadcast bitrate budget and viewport area.

[0046] By summarizing the restored Gaussian pyramid levels for each pixel in each 360° video frame, a 360° video with heterogeneous spatial quality is synthesized.

[0047] The method described in the third aspect can be expanded into implementation forms corresponding to the implementation forms of the video generation device described in the first aspect. Therefore, the implementation forms of the method include the features of the corresponding implementation forms of the video generation device.

[0048] The advantages of the method according to the third aspect are the same as the advantages of the corresponding embodiment of the video generating device according to the first aspect.

[0049] The present invention also relates to a computer program, characterized in that it includes program code. When the program code is executed by at least one processor, the at least one processor is caused to perform any method described in the embodiments of the present invention. Here, the present invention also relates to a computer program product, comprising a computer-readable medium and the computer program, wherein the computer program is included in the computer-readable medium, and the computer-readable medium includes one or more of the following: read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), and hard disk drive.

[0050] Further applications and advantages of embodiments of the present invention will be apparent from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are intended to illustrate and explain different embodiments of the invention, in which:

[0052] Figure 1 A video generating device according to an embodiment of the present invention is shown.

[0053] Figure 2 A method according to an embodiment of the present invention is shown.

[0054] Figure 3 A download agent according to an embodiment of the present invention is shown.

[0055] Figure 4 An upload agent and a download agent according to an embodiment of the present invention are shown.

[0056] Figure 5 The generation of a Laplacian pyramid according to an embodiment of the present invention is shown.

[0057] Figure 6 A Laplace-based recursive video decomposition according to an embodiment of the present invention is shown.

[0058] Figure 7 A flowchart of a pyramid level allocation process according to an embodiment of the present invention is shown.

[0059] Figure 8 The performance results of the embodiments of the present invention are shown.

[0060] Figure 9 Further performance results of embodiments of the present invention are shown. DETAILED DESCRIPTION

[0061] Existing heterogeneous video quality generation schemes include tiling and offset projection. The idea behind tiling is to spatially divide a video frame into non-overlapping pixel blocks, called tiles, and encode them independently. The client can then request the quality of each encoded tile based on the tile's position and the current quality-region input specification. However, despite its widespread adoption and flexibility, tiling techniques can incur bitrate overhead and compression efficiency losses. In offset projection, heterogeneous video is generated by distorting the original spherical sampling, mapping more pixels near the focal point (improving 360° video quality) and fewer pixels away from the focal point (reducing quality). The amount of distortion is controlled by the offset magnitude, meaning that the farther from the focal point, the greater the quality distortion. Here, a larger offset magnitude results in greater quality distortion in the opposite direction. Offset projection also reduces video bandwidth by reducing video resolution, which can result in lower-quality video content outside the region of interest. However, when network bandwidth needs to be reduced, reducing video resolution through offset projection can result in very low-quality video content outside the region of interest. This means that in this case, offset projection may not be able to cope well with sudden head movements. The inventors have recognized that neither block segmentation nor offset projection can achieve both a high-quality user experience and reduced bandwidth for VR 360° video transmission. Therefore, this paper proposes a video generation device and corresponding method that can meet these requirements.

[0062] Figure 1 FIG. 1 shows a video generating device 100 according to an embodiment of the present invention. Figure 1In the illustrated embodiment, video generation device 100 includes at least one processor core or processing circuit 102, which may be coupled to internal or external memory 104 via a coupling / communication mechanism 106 known in the art. Video generation device 100 may also include multiple processor cores or processing circuits 102. Memory 104 may store program code that, when executed, causes the processor core or processing circuit 102 of video generation device 100 to perform the functions and actions described herein. Video generation device 100 also includes input device 108 and output device 110, both coupled to processor core or processing circuit 102 via coupling / communication mechanism 106 known in the art. In the present invention, "video generation device 100 is configured to perform certain functions or actions" may be understood to mean that video generation device 100 includes suitable means for performing the functions or actions, such as processor core or processing circuit 102. Processor core or processing circuit 102 may be dedicated to performing methods or processes according to the present invention. However, in some embodiments, processor core or processing circuit 102 of video generation device 100 may be shared with another device. The video generating device 100 may be an independent device or a part of another device, such as Figure 3 The download agent shown. Furthermore, the processing circuitry may include hardware and software. The hardware may include analog or digital circuitry, or both. In one embodiment, the processing circuitry includes one or more processors and non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code that, when executed by the one or more processors, causes the video generation device 100 to perform the operations or methods described herein.

[0063] Figure 1 The video generation device 100 is configured to obtain Laplacian pyramid levels for a 360° video frame. The video generation device 100 is further configured to enhance each obtained Laplacian pyramid level to restore a corresponding set of Gaussian pyramid levels for each 360° video frame. The video generation device 100 is further configured to determine, for each pixel in each 360° video frame, a restored Gaussian pyramid level from the restored set of Gaussian pyramid levels based on a live broadcast bitrate budget and a viewport area by executing a pyramid level allocation process. The video generation device 100 is further configured to synthesize a 360° video with heterogeneous spatial quality by aggregating the restored Gaussian pyramid levels determined for each pixel in each 360° video frame.

[0064] Figure 2 Shows that the Figure 1Flowchart of a corresponding method 200 executed in the video generation device 100 shown. The method 200 includes obtaining (202) Laplacian pyramid levels of a 360° video frame. The method 200 also includes promoting each obtained Laplacian pyramid level (204) so ​​as to restore a corresponding set of Gaussian pyramid levels for each 360° video frame. The method 200 also includes: for each pixel in each 360° video frame, according to the bit rate budget and viewport area of ​​the live broadcast, determining (206) a restored Gaussian pyramid level from the corresponding set of Gaussian pyramid levels by performing a pyramid level allocation process. The method 200 also includes synthesizing (208) a 360° video with heterogeneous spatial quality by aggregating the restored Gaussian pyramid levels for each pixel in each 360° video frame.

[0065] The Gaussian pyramid is an image sequence, i.e., a multi-scale representation of the original image, where the image is blurred and reduced in resolution by a factor of 2 to obtain the input image of the next layer (i.e., a higher Gaussian pyramid level). The number of layers in the Gaussian pyramid represents the degradation of video quality at N+1 multi-scale levels. On the other hand, the Laplacian pyramid is calculated by using the difference between each two consecutive Gaussian pyramid levels, first improving the lower pyramid level g. N-i+1 , and then from a higher level g N-i Lower levels are subtracted from the original. Laplacian pyramid levels only contain edge and fine texture information, so each Laplacian level is better compressed than the corresponding Gaussian level. At the same time, each level of the Gaussian and Laplacian pyramids is better compressed than the previous one. Regarding the decomposition of the Gaussian and Laplacian pyramids, it takes longer to generate the Laplacian pyramid than to generate a Gaussian pyramid with the same number of levels because it includes an additional operation, namely the subtraction between consecutive Gaussian pyramid levels. However, compared to the Gaussian pyramid, the Laplacian pyramid can better compress and save more bandwidth, which is very important for the transmission of 360° video.

[0066] According to an embodiment of the present invention, the video generating device 100 is configured to:

[0067] Based on the bitrate budget and viewport area, 360° composite videos with heterogeneous spatial quality are produced for live streaming. The bitrate budget in this paper can be defined as the maximum bitrate of the 360° composite video that fits within the available network bandwidth, meaning that the 360° composite video can be transmitted to the client over the network without interruption. The viewport area can be defined as the small portion of the entire 360° video that is actually displayed on the HMD device and corresponds to the client's current Field of View (FoV).

[0068] In an embodiment, the video generating device 100 may be included in the download agent 300, for example, Figure 3 The download agent 300 shown. Therefore, the video generation device 100 can be a functional module, that is, a video processing device of the download agent 300. As further shown, the download agent 300 can include an input device 302 and an output device 304. The download agent 300 also includes other communication capabilities ( Figure 3 (not shown) so that the download agent 300 can communicate in the communication system. In an embodiment, the download agent 300 can act as a server to process the 360° video generated according to an embodiment of the present invention for one or more clients.

[0069] Figure 4 The communication between the upload agent 400 and the download agent 300 in the communication system 500 is shown. The communication system can be a third generation partnership project (3GPP) system, such as a long term evolution (LTE) system or a 5G new radio (NR) system or any other suitable communication system. Figure 4 The video processing performed in the download agent 300 and the transmission of the encoded composite video to the client 600 are also shown. In an embodiment, the proposed solution may include the following steps: generating a pyramid in the upload agent 400; performing pyramid enhancement in the download agent 300; executing a pyramid level allocation process in the download agent 300; and synthesizing a video representation with heterogeneous spatial quality in the download agent 300.

[0070] Therefore, combined Figure 4 , a 360° video is captured at a user equipment (UE) and, after encoding, is sent to an upload agent 400 via a 5G mobile network. The upload agent 400 can be located in a mobile edge computing (MEC) server close to the UE. In the upload agent 400, the encoded video is decoded, decomposed into multi-scale videos using a Laplacian pyramid decomposition method, and re-encoded. It is worth noting that the generated multi-scale videos are smaller than the original videos and can be sent in parallel via the 5G mobile network, thereby reducing transmission latency. Therefore, in step I, the encoded 360° video is input to the upload agent 400, that is, it is first decoded and then used as input for pyramid generation. In the pyramid generation step of the upload agent 400, a Gaussian pyramid is first created, and then Laplacian pyramid levels are created based on the differences between subsequent levels of the Gaussian pyramid. After the pyramid generation step, the Laplacian pyramid levels are transmitted to the download agent 300, which can be located in an MEC server close to the client 600. Therefore, the output of the upload agent 400 and the input of the download agent 300 are encoded Laplacian pyramid levels.

[0071] Figure 4 The download agent 300 in FIG. 1 includes three functional blocks: a pyramid lifting block 320, a pyramid level allocation process block 322, and a video synthesis block 324. In the download agent 300, the received encoded Laplacian pyramid levels are first decoded and then used as input for pyramid lifting in the pyramid lifting block 320. The output of the pyramid lifting block 320 and the input of the pyramid level allocation process block 322 is the lifted Gaussian pyramid, which includes all recovered Gaussian pyramid levels. The output of the pyramid level allocation process block 322 and the input of the video synthesis block 324 are the optimal Laplacian pyramid levels for each pixel in the image extracted from the 360° video, given the available bitrate budget and the current viewport. The output of the video synthesis block 324 is a heterogeneous 360° video synthesized from the pixels determined by the pyramid level allocation process. Then, in step III, the synthesized heterogeneous 360° video is encoded and sent from the download agent 300 to the client 600. In the download agent 300, a video with heterogeneity is generated / created based on the available bitrate budget and the user's head position information sent by the client 600. Therefore, in step IV, the updated current viewport is periodically fed back from the client 600 to the download agent via feedback signaling. When the user moves their head, the client 600, for example, sends the updated viewport position to the download agent 300 in the opposite direction of video transmission via feedback signaling. This feedback signaling can be implemented using conventional solutions.

[0072] Once the heterogeneous video is synthesized in the download agent 300, the video is encoded and sent over the network to the client 600 and its HMD, which extracts the viewport and displays the video to the user of the client 600. It should be noted that the client 600 periodically sends its updated viewport position to the download agent 300. The client 600 in this context can be, for example, an end user or a mobile device of the end user.

[0073] In the following, the steps of pyramid lifting, pyramid level allocation process, and 360° video synthesis will be described in more detail to provide a deeper understanding of the embodiments of the present invention.

[0074] Pyramid Generation

[0075] Each level of the Laplace pyramid is based on Figure 5 The difference between the subsequent levels of the Gaussian pyramid is created. In order to construct the Laplacian level l i , first improve the lower Gaussian level g N-i+1 , and then from the higher Gaussian level g N-iIn practice, the absolute difference between subsequent Gaussian levels is calculated, i.e., l i =|G (N-i+1) -g (N-i) |, where i = N:l (0) =G (N) .

[0076] Laplacian levels contain fine textures of non-zero values ​​and edges, so the compression efficiency of the Laplacian pyramid is higher than that of the Gaussian pyramid. This also means that each Laplacian level is better encoded than the corresponding Gaussian level, and each Laplacian level is better compressed than the previous level.

[0077] Pyramid promotion

[0078] Once the Laplace pyramid is calculated, it is obtained from l i The level starts with each Laplace level l i The resolution is increased to the next Laplace level. i+1 resolution once and add l on the increased level i+1 Level, such as Figure 6 This process is repeated until the desired optimal level is reached, which is determined in the next step of the pyramid level allocation process. In this way, the image with small detail loss can be reconstructed by Laplace decomposition, i.e., the Gaussian pyramid level. In other words, lifting consists of Starting from the level, each Laplace pyramid level obtained Move to the next level resolution, and after the improvement Add on the level levels, so as to restore the corresponding Gaussian pyramid level for each 360° video frame, where i represents the current level and N represents the total number of Gaussian pyramid levels.

[0079] The lifted pyramid is computed only once and can then be reused to synthesize various video representations depending on the viewport position and bitrate budget.

[0080] Pyramid-level allocation process

[0081] In order to create variable-quality videos through Laplace decomposition, we first need to determine the mapping between the Laplace levels and the quality (video) described by the quality distribution function q(·). This mapping is given by the mapping function The mapping function maps the quality distribution q(·) to the optimal level of the Laplacian pyramid for a given bitrate budget B for each pixel in each 360° video frame. The pyramid level allocation process includes several steps in the embodiment of the present invention, such as Figure 7 shown.

[0082] Step 212: Extract the surface bitrate of each level. In step 212, the restored Gaussian pyramid levels are obtained. The surface bitrate of each encoded raised Gaussian pyramid level is then calculated based on the raised Gaussian pyramid level and the spherical surface area of ​​each video region of interest (ROI), i.e., the bitrate of each sphere. The calculated surface bitrate can be used to estimate the required bitrate to assign a given pyramid level to a video region with a spherical surface area s.

[0083] To avoid wasting time and processing power by encoding each lifted Gaussian pyramid level, the size of the Gaussian pyramid level can be estimated using the encoded lifted Laplacian pyramid level. In addition, if encoding of the lifted Laplacian pyramid level is to be avoided, its size can also be estimated using the encoded Laplacian pyramid received from the upload agent 400.

[0084] Step 214: Perform linear optimization. Measure the spherical area of ​​the viewport region and determine which pyramid level to map to this region through linear optimization to allocate the highest quality without exceeding the bitrate budget. v The viewport area is assigned a specific pyramid level by multiplying the surface bitrate of a raised Gaussian pyramid level to obtain the bitrate required for that pyramid level. However, the final allocation depends on the available bitrate budget. This process prioritizes the viewport area, assigning the highest possible pyramid level to the viewport area so that the bitrate of the entire composite video is below the bitrate budget. Quality outside the viewport area is deprioritized because the client is unlikely to request video content outside the viewport area, at least not for a short period of time.

[0085] Step 216: Map the viewport area to the highest quality pyramid level that does not exceed the bitrate budget, that is, the optimal Gaussian level. Then, repeat steps 214 and 216 for the next priority area until all priority areas are processed.

[0086] Step 218: Through the pyramid level allocation process, according to the bit rate budget and the current viewport area, each region of interest in the 360° video frame is finally mapped to the determined optimal Gaussian level after restoration.

[0087] Video synthesis

[0088] According to an embodiment of the present invention, each variable-quality video version can be calculated by adding the boosted Laplacian level to the optimal level determined by the pyramid level allocation process based on the available bitrate budget B. More specifically, synthesizing a 360° video with heterogeneous spatial quality includes summing the optimal restored Gaussian pyramid level determined for each pixel in each 360° video frame according to the following formula:

[0089]

[0090] Among them, O G is the output video with heterogeneous spatial quality corresponding to the quality distribution q(·), is the improved Laplace pyramid level i, B is the bit rate budget, is the mapping function that maps the quality distribution q(·) to the optimal level of the Laplacian pyramid for a given bitrate budget B for each pixel in each 360° video frame, where (x, y) are the coordinates of the pixel in the 360° video frame.

[0091] When the viewport change request changes, the video representation is reconstructed for the new viewport area. A Laplacian pyramid can be calculated for each video segment and sent to the download agent 300. Based on the client's viewport position and bitrate budget, the download agent 300 can reuse the Laplacian pyramid to synthesize various video representations.

[0092] The effect of the embodiment of the present invention is evaluated by comparing with the traditional solution. Under the condition of a given bit rate, the embodiment of the present invention provides a higher video quality for the client compared with the existing most advanced solution using the same bit rate. Figure 8 As shown in Figure 2, the quality of 360° videos encoded using the proposed scheme even improves with increasing bitrate budget. In addition, the bandwidth required to transmit the Laplacian pyramid (between the upload agent and the download agent) in the proposed scheme is 10-15% lower than the bandwidth required to transmit the original 360° video, as shown in Figure 2. Figure 9 As shown in Figure 2, the bandwidth of the Laplacian pyramid is 15-30% lower than that of the H.264 Gaussian pyramid and up to 40% lower than that of the High Efficiency Video Coding (HEVC) Gaussian pyramid. It is worth noting that the performance evaluation was based on 4K (30fps), 6K (60fps), and 8K (30fps) video encoded using H.264 and HEVC. The Laplacian pyramid is approximately 5% faster than the Gaussian pyramid.

[0093] In addition, any method according to an embodiment of the present invention can be implemented in a computer program with an encoding method, which, when run through the processing measures, can cause the processing measures to perform method steps. The computer program is included in the computer-readable medium of the computer program product. The computer-readable medium can basically include any memory, such as ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), flash memory, EEPROM (electrically erasable programmable read-only memory) and hard disk drive.

[0094] Furthermore, the skilled person will recognize that embodiments of the video generation device 100 and the download agent 300 include the necessary communication capabilities in the form of functions, means, units, elements, etc. for performing the scheme. Examples of other such means, units, elements, and functions include: processors, memories, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, de-interleavers, modulators, demodulators, input devices, output devices, antennas, amplifiers, receiver units, transmitter units, DSPs, MSDs, TCM encoders, TCM decoders, power supply units, power feeds, communication interfaces, communication protocols, etc., which are appropriately arranged together to perform the scheme.

[0095] In particular, the processors of the video generation device 100 and the download agent 300 may include, for example, one or more of a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application specific integrated circuit (ASIC), a microprocessor, or other processing logic capable of interpreting and executing instructions. The term "processor" may therefore refer to a processing circuit comprising a plurality of processing circuits, examples of which are any, some, or all of the items listed above. The processing circuit may further perform data processing functions, inputting, outputting, and processing data, including data buffering and device control functions, such as call processing control, user interface control, etc.

[0096] Finally, it should be understood that the present invention is not limited to the embodiments described above, but relates simultaneously to and incorporates all embodiments within the scope of the appended independent claims.

Claims

1. A video generation device (100) for generating a 360° video, characterized in that The video generating device (100) is used for: Get the Laplacian pyramid levels of the 360° video frame; Lifting each obtained Laplacian pyramid level to recover a corresponding set of Gaussian pyramid levels for each 360° video frame, wherein lifting each obtained Laplacian pyramid level comprises: Starting from the level, each Laplace pyramid level obtained Move to the next level resolution, and after the improvement Add on the level level, so as to recover the corresponding Gaussian pyramid level for each 360° video frame, where i represents the current level and N represents the total number of Gaussian pyramid levels; For each pixel in each 360° video frame, determine a restored Gaussian pyramid level from the restored set of Gaussian pyramid levels by performing a pyramid level allocation process according to a live broadcast bitrate budget and a viewport area; Synthesizing a 360° video with heterogeneous spatial quality by aggregating the restored Gaussian pyramid levels determined for each pixel in each 360° video frame, including aggregating the optimal restored Gaussian pyramid levels determined for each pixel in each 360° video frame according to the following formula: Among them, O G is the output video with heterogeneous spatial quality corresponding to the quality distribution q(·), is the improved Laplace pyramid level i, B is the bit rate budget, is the mapping function that maps the quality distribution q(·) to the optimal level of the Laplacian pyramid for a given bitrate budget B for each pixel in each 360° video frame, where (x, y) are the coordinates of the pixel in the 360° video frame.

2. The video generating device according to claim 1, wherein The method for synthesizing a 360° video with heterogeneous spatial quality includes: The 360° synthetic video with heterogeneous spatial quality is produced for live broadcast.

3. The video generating device according to claim 1, wherein The video generating device is further configured to: The bit rate of each boosted Gaussian pyramid level is calculated according to the sum of the bit rates of the boosted Laplacian pyramid levels.

4. The video generating device according to claim 3, characterized in that The pyramid level allocation process includes: (a) calculating the surface bit rate of each promoted Gaussian pyramid level according to the bit rate of each promoted Gaussian pyramid level; (b) measuring the spherical area of ​​the viewport region of the 360° video frame; (c) multiplying the surface bit rate of each of the promoted Gaussian pyramid levels by the area of ​​the spherical surface to obtain a bit rate required for allocating each of the promoted pyramid levels to the viewport area; (d) mapping the viewport area to a highest quality pyramid level in each of the promoted pyramid levels that does not exceed the bitrate budget.

5. The video generating device according to claim 4, characterized in that The pyramid level allocation process also includes: Repeat steps (a) to (d) for each region of interest in the 360° video frame according to the priority of the region of interest.

6. A download agent (300) for a communication system (500), characterized in that The download agent (300) includes the video generating device (100) according to any one of claims 1 to 5.

7. A video generation method (200), characterized in that: The method (200) comprises: Obtain (202) the Laplacian pyramid levels of the 360° video frame; each Laplacian pyramid level obtained (204) is lifted to recover a corresponding set of Gaussian pyramid levels for each 360° video frame, wherein lifting each Laplacian pyramid level obtained comprises: Starting from the level, each Laplace pyramid level obtained Move to the next level resolution, and after the improvement Add on the level level, so as to recover the corresponding Gaussian pyramid level for each 360° video frame, where i represents the current level and N represents the total number of Gaussian pyramid levels; For each pixel in each 360° video frame, determine (206) a restored Gaussian pyramid level from a corresponding set of Gaussian pyramid levels according to a live broadcast bitrate budget and a viewport area, wherein the live broadcast bitrate budget is a maximum bitrate of the synthesized 360° video that fits within the available network bandwidth; Synthesizing (208) a 360° video with heterogeneous spatiality by aggregating the restored Gaussian pyramid levels for each pixel in each 360° video frame, including aggregating the optimal restored Gaussian pyramid levels determined for each pixel in each 360° video frame according to the following formula: Among them, O G is the output video with heterogeneous spatial quality corresponding to the quality distribution q(·), is the improved Laplace pyramid level i, B is the bit rate budget, is the mapping function that maps the quality distribution q(·) to the optimal level of the Laplacian pyramid for a given bitrate budget B for each pixel in each 360° video frame, where (x, y) are the coordinates of the pixel in the 360° video frame.

8. A non-transitory computer-readable storage medium, characterized in that The method comprises a computer program which, when run on a computer, causes the computer to perform the method of claim 7.

Images