Encoding a video sequence into a hierarchically encoded video stream with a target resolution

Abstract

The present disclosure relates to a method (500), software, and apparatus for encoding a video sequence into an encoded video stream having a target resolution, comprising: a step (s502) of encoding a first image frame of the video sequence by using a base layer to encode an image frame to a first resolution lower than the target resolution, and using an enhancement layer to encode at least a portion of the image data of the first image frame to reach the target resolution; a step (s504) of detecting an encoding scenario from a set of one or more predefined encoding scenarios; and a step (s506) of encoding a second image frame of the video sequence by using a base layer to encode a second image frame to the target resolution, and indicating to the encoded video stream that the second image frame can be decoded to the target resolution using only the base layer.

Description

Technology Field

[0001] The present disclosure relates to hierarchical video coding, and in particular to a method, apparatus, and non-transient computer-readable storage medium for providing an encoded video having a target resolution. Background Technology

[0002] Hierarchical video coding has enhanced the efficiency and flexibility of video streaming technologies. Technologies such as Low Complexity Enhancement Video Coding (LCEVC) or similar codecs enable the delivery of content of various resolutions from a single encoded source by encoding video data across multiple layers. This approach begins with a base layer containing a lower-resolution version of the video, upon which one or more enhancement layers are added to provide the details necessary to restore the video at higher resolutions. This scalable structure is particularly advantageous for adaptive streaming technologies because it ensures optimized video transmission by allowing seamless adjustment based on different network conditions and device performance.

[0003] The challenge within hierarchical coding is that in LCEVC, the enhancement layer is encoded without using motion vectors and relies solely on transform-based coding to represent residual differences between the base layer and the original content. While this design reduces computational complexity, it can lead to inefficiencies in certain scenarios where significant temporal redundancy exists between frames. Without motion compensation, large residuals can be generated, particularly in high-motion scenes or rapidly changing content, resulting in increased bandwidth requirements for the enhancement layer.

[0004] Therefore, improvement is needed in this context. The problem to be solved

[0005] Considering the above, it would be beneficial to address or at least reduce one or more of the disadvantages discussed above, as described in the appended independent claims. means of solving the problem

[0006] According to a first aspect of the present disclosure, a method for encoding a video sequence into an encoded video stream having a target resolution is provided, wherein the encoded video stream comprises an enhancement layer and a base layer, and the method comprises: a step of encoding a first image frame of the video sequence, wherein the image frame is encoded to a first resolution lower than the target resolution using the base layer and the target resolution is reached by encoding at least a portion of the image data of the first image frame using the enhancement layer; a step of detecting an encoding scenario from one or more predefined encoding scenario sets; and a step of encoding a second image frame of the video sequence, wherein the second image frame is a frame following the first image frame in the video sequence, wherein the second image frame is encoded to the target resolution using the base layer and the second image frame is indicated to be decoded to the target resolution using only the base layer within the encoded video stream.

[0007] Hierarchical video coding generates an encoded video stream of multiple resolutions using a base layer and one or more enhancement layers. The base layer encodes image frames at a resolution lower than the target resolution, while the enhancement layer encodes additional data that refines the decoded image, enabling the decoded video stream to reach the target resolution. Thus, the enhancement layer encodes the residual difference between the image data of the original image frame and the low-resolution image data encoded by the base layer, and said residual is encoded at the target resolution.

[0008] This approach offers several advantages, such as generally providing a lower average bitrate by efficiently separating base data and enhancement data. However, in certain encoding scenarios, such as high-motion scenes, LCEVC may face inefficiencies, resulting in a higher overall bitrate compared to encoding the video using only a base encoder.

[0009] In the context of this specification, the term "encoding scenario" refers to a specific combination of video content characteristics and encoding conditions that may result in inefficiency in LCEVC. Such scenarios may occur, for example, in video sequences with fast or irregular motion, abrupt scene changes, complex textures, or high spatial detail, in which case the residuals of the enhancement layer increase, causing the overall bitrate to increase beyond the bitrate required to directly encode the video sequence at the target resolution using only the base encoder.

[0010] The inventors recognized that such predefined encoding scenarios can be detected within a video stream. In such cases, the encoding method can dynamically adjust how subsequent image frames (i.e., second image frames) are processed to improve overall efficiency. Specifically, if an encoding scenario is detected in which the enhancement layer can increase the overall bitrate, the system can switch to encoding subsequent frames directly at the target resolution using only the base layer. By doing so, the enhancement layer is bypassed for the second image frames, and the overall bitrate is reduced while maintaining the quality of the encoded video stream.

[0011] Advantageously, this approach enables a more adaptive encoding process in which the enhancement layer is used selectively only when it is deemed advantageous, thereby mitigating the bandwidth overhead that would otherwise occur in these predefined encoding scenarios. Additionally, by including an indication within the encoded video stream that the second frame can be fully decoded using only the base layer, the decoder can smoothly handle this optimization without additional complexity.

[0012] In some examples, among one or more predefined encoding scenarios, the first encoding scenario involves continuous camera shake while capturing the video sequence. Continuous camera shake occurs when the position or orientation of the camera constantly changes due to, for example, handheld shooting, unstable mounting, or environmental factors such as wind or vibration. This movement causes frequent and irregular changes in the captured video frames. As a result, large residuals between consecutive frames can frequently occur compared to a stable scene. Consequently, the encoder may struggle to efficiently compress these residuals in the enhancement layer, which can lead to an increase in bitrate.

[0013] In some examples, the step of detecting an encoding scenario includes the step of detecting a first encoding scenario by: receiving sensor data from a camera's gyro sensor; and determining that the camera is shaking based on the analysis of said sensor data. Advantageously, camera shake detection using gyro sensor data can enable accurate and early detection. Since the gyro sensor can provide real-time motion feedback from the camera hardware, the encoder can rapidly identify the first encoding scenario and, accordingly, immediately adapt the encoding process to prevent inefficient enhancement layer encoding. Furthermore, since detection is based on physical sensor readings rather than image analysis, the possibility of false detection from other causes of visual motion, such as moving objects within the scene, can be reduced, thereby improving the reliability of the entire detection process.

[0014] In some examples, the step of detecting an encoding scenario includes the step of detecting a first encoding scenario by: analyzing motion vectors of a base layer of an encoded first image frame; and determining that the camera is shaking by comparing the motion vectors with image data of a reference encoded image frame used when encoding the first image frame and determining that the motion vectors correspond to uniform movement of the image data of the first image frame.

[0015] The term "uniform motion" as used here refers to a situation where the majority of motion vectors within an image frame (e.g., more than 50%, e.g., 75%, 85%, 90%, etc.) point in the same direction and have similar magnitudes. This indicates that the entire frame is moving consistently and is generally caused by external factors such as camera shake or panning. However, while local object movement within the captured scene may cause some motion vectors to deviate from the general pattern, this can be classified as uniform motion as long as the dominant motion vectors maintain consistency across a wide area of ​​the frame.

[0016] The advantage of detecting camera shake by analyzing motion vectors in the base layer is that it eliminates the need for additional sensors or hardware by utilizing already available encoding data. This makes the detection method efficient and allows for easy integration into existing encoding workflows.

[0017] In some examples, among one or more predefined encoding scenarios, a second encoding scenario includes the ratio of the bitrate of the enhancement layer to the bitrate of the base layer exceeding a threshold ratio, or the bitrate of the enhancement layer exceeding a threshold, and the step of detecting the second encoding scenario includes the step of monitoring the bitrate of the enhancement layer of the encoded first image frame.

[0018] In this example, the step of detecting an encoding scenario includes the step of detecting a second encoding scenario by monitoring the bitrate of the enhancement layer and, if applicable, comparing it with the bitrate of the base layer. This enables the system to detect inefficiencies caused by the bitrate of the enhancement layer increasing relatively compared to the base layer or increasing absolutely beyond a threshold, thereby reducing the computational complexity required to monitor the encoding efficiency of the enhancement layer.

[0019] In some examples, the portion of the image data of the first image frame represents a region of interest within a scene depicted by a video sequence, and among one or more predefined encoding scenarios, a third encoding scenario includes the existence of continuous movement within the region of interest of the scene, and the step of detecting the encoding scenario includes the step of detecting the third encoding scenario by: analyzing motion vectors of a base layer of a plurality of encoding image frames preceding a second encoding image frame in an encoded video stream; and determining that continuous movement exists within the region of interest of the scene based on the analysis.

[0020] In the context of this specification, the term “region of interest (RoI)” refers to a specific part of an image frame that is defined by the user or automatically identified based on a high level of visual detail or importance within a scene. Accordingly, an RoI may correspond to an area containing fine textures, important objects, or content designated by the application or user to be given higher encoding fidelity.

[0021] In the context of this specification, the term “persistent motion” refers to continuous movement of video content over a series of frames, as indicated by motion vectors consistently exhibiting movement within a region of interest over time. Such persistent motion is not necessarily unidirectional and may include oscillating motion, irregular but sustained motion, or repetitive scene activity. Such motion may be caused by factors such as moving objects, camera panning, or dynamic regions within the scene.

[0022] When continuous motion occurs in the high-resolution details of an image frame, the base layer is encoded at a lower resolution and may fail to capture most or all of this motion. This results in larger residuals in the enhancement layer, making it difficult to compress efficiently, particularly because the enhancement layer does not use motion vectors (as previously discussed). By switching to a higher-resolution base layer and bypassing the enhancement layer when a third encoding scenario is detected, bandwidth overhead can be reduced.

[0023] In some examples, the step of encoding a second image frame of a video sequence includes the step of encoding the base layer of the second image frame into a key frame having a target resolution. By inserting a new key frame having a target resolution, this example resets the prediction chain and provides a completely independent frame that does not rely on previous frames for decoding. This allows the base layer of the target resolution to be decoded using standard decoder functions without additional complexity, while allowing the enhancement layer to be bypassed.

[0024] In some examples, the step of encoding a second image frame of a video sequence includes encoding the base layer of the second image frame into a switch frame having a target resolution. The switch frame is a type of interframe in which the reference image frame does not need to have the same resolution as the image frame being encoded. This enables a smooth transition between a low-resolution base layer and a high-resolution base layer while maintaining efficient compression while referencing a previously encoded image frame (e.g., an encoded first image frame).

[0025] In some examples, the step of indicating in the encoded video stream that the second image frame can be decoded to a target resolution using only the base layer comprises at least one of the following: encoding the enhancement layer of the second image frame without image data; indicating in the metadata of the encoded image stream that the second image frame can be decoded to a target resolution using only the base layer; and encoding the base layer of the second image frame into a switch frame. For example, the metadata of the encoded video stream may include information regarding available resolutions (a first resolution and a target resolution), so that the decoder can easily determine whether decoding of the enhancement layer is required for the second image frame, or whether the base layer alone is sufficient to reach the target resolution for a given image frame. In other examples, the decoder may typically operate to continue decoding the enhancement layer, but since the enhancement layer does not contain image data, the process may be completed quickly without adding additional data to the decoded second image frame. Alternatively, the decoder may be configured to recognize that an enhancement layer is not required for decoding the second image frame when the base layer of the second image frame is encoded into a switch frame having an increased resolution compared to the base layer of the first image frame.

[0026] In some examples, the method further includes the step of encoding a plurality of image frames after a second image frame consecutively within a video stream, using a base layer to encode each of the plurality of image frames to a target resolution. This implies that the enhancement layer is bypassed for an extended period, thereby reducing the need for frequent switching between encoding modes. Advantageously, by prioritizing long-term stability over frequent mode switching, this example can effectively achieve a balance of power efficiency, bitrate optimization, and video quality across various encoding scenarios.

[0027] In some examples, the method further comprises: determining that the number of the plurality of image frames exceeds a threshold number; detecting that no encoding scenario among a set of one or more predefined encoding scenarios is occurring; and encoding a current image frame of a video sequence, wherein the current image frame is encoded to the first resolution using a base layer and at least a portion of the image data of the current image frame is encoded using an enhancement layer to reach a target resolution. Verification for encoding scenarios, such as camera shake or continuous movement, may be performed continuously during a time interval (e.g., 5 to 10 seconds) corresponding to the threshold, or may be performed after the threshold number of frames has been exceeded. Continuous verification may include, for example, real-time analysis of gyroscope data or motion vectors frame by frame, and, by providing more data samples, can make the judgment at the threshold number of image frames more stable and provide a more reliable decision by filtering out short-term fluctuations. Alternatively, if a threshold number of image frames are encoded to a target bitrate using only the base layer and then verified, computational power can be reduced by making a one-time judgment based on data collected at the threshold point.

[0028] In some examples, the first image frame is encoded using the Low Complexity Enhancement Video Coding (LCEVC) coding standard. Accordingly, the first image frame can be decoded using a standard-compliant LCEVC decoder.

[0029] In some examples, the base layer of each of the first and second image frames is encoded using one of the following coding standards: AV1, AV2, and VP9. These coding standards are examples of coding standards that implement variable resolution in switch frames and / or I-frames within a stream.

[0030] According to a second aspect of the present disclosure, the objective is achieved by a non-transient computer-readable storage medium storing instructions for implementing a method according to a first aspect when executed on a device having processing capability.

[0031] According to a third aspect of the present disclosure, the objective is achieved by an apparatus for encoding a video sequence into an encoded video stream having a target resolution, wherein the encoded video stream comprises an enhancement layer and a base layer, and the apparatus is configured to perform the following: encoding a first image frame of a video sequence, wherein the image frame is encoded to a first resolution lower than the target resolution using a base layer, and the target resolution is reached by encoding at least a portion of the image data of the first image frame using an enhancement layer; detecting a first encoding scenario from a set of one or more predefined encoding scenarios; and encoding a second image frame of a video sequence, wherein the second image frame is a frame following the first image frame in the video sequence, wherein the second image frame is encoded to the target resolution using a base layer, and indicating that the second image frame is decoded to the target resolution using only a base layer within the encoded video stream.

[0032] The second and third aspects may generally have the same features and advantages as the first aspect. Additionally, it should be noted that the present disclosure relates to all possible combinations of features unless expressly otherwise stated. Brief explanation of the drawing

[0033] In addition to the foregoing, further objects, features, and advantages of the present disclosure will be better understood through the following exemplary and non-limiting detailed description of embodiments of the present disclosure, with reference to the accompanying drawings in which the same reference numerals are used for similar components. Herein: FIG. 1 illustrates a system for encoding a video sequence into an encoded video stream having a target resolution according to embodiments. FIG. 2 illustrates the function of a device for encoding a video sequence into an encoded video stream having a target resolution according to embodiments. FIG. 3 illustrates an encoded video stream according to a first embodiment. FIG. 4 illustrates an encoded video stream according to a second embodiment. FIG. 5 illustrates a flowchart of a method for encoding a video sequence into an encoded video stream having a target resolution according to embodiments. Specific details for implementing the invention

[0034] Efficient video encoding is essential for delivering high-quality content while minimizing bitrate and computational overhead. Hierarchical video coding techniques, such as Low Complexity Enhancement Video Coding (LCEVC), enable scalability and adaptability to various network conditions by encoding video at multiple resolutions using base and enhancement layers. However, in certain encoding scenarios, inefficiencies can arise where the enhancement layer consumes excessive bitrate.

[0035] The present disclosure addresses these problems by introducing a method for dynamically adjusting the encoding process based on detected encoding scenarios. Specifically, using the techniques described herein, predefined encoding conditions, such as camera shake, excessive enhancement layer bitrate, or continuous movement within a region of interest, are monitored, and the encoding is adaptively adjusted accordingly. When such encoding scenarios are detected, a system / device implementing the techniques switches subsequent frames to be encoded at a target resolution using only the base layer, thereby bypassing the need to use an enhancement layer to reach the target resolution. This approach can optimize bitrate allocation, reduce computational complexity, and improve overall encoding efficiency.

[0036] FIG. 1 illustrates a system (100) that includes, by way of example, a video camera (102). The video camera (102) captures image frames (106a-c) representing a scene. The system (100) includes an encoding device (120) configured to encode a received video sequence (106) into an encoded video stream (150) having a target resolution. The encoding device (120) implements a hierarchical encoding method that encodes the video sequence using a base layer of low resolution and an enhancement layer that provides additional data to restore the video at the target resolution. Thus, the encoded video stream (150) includes an enhancement layer and a base layer, enabling scalable video encoding that can dynamically adapt based on predefined encoding scenarios. The enhancement layer and the base layer can be multiplexed into a single encoded video stream (150) using a multiplexer (130).

[0037] The encoding device (120) processes the first image frame (106a) by first encoding it into a base layer (e.g., using a base layer encoder (126)) at a first resolution (a resolution lower than the target resolution). This includes standard compression techniques such as motion estimation and compensation, transform coding, quantization, and entropy coding to efficiently represent the image frame (106a) at a lower resolution. The base layer can be encoded by the base layer encoder (126) using AV1, AV2, VP9, ​​or other suitable video codecs, ensuring compatibility with widely used video compression standards.

[0038] When the base layer is encoded, the encoding device (120) determines residual information representing the difference between the original image frame (106a) and the upscaled version of the base layer. Unlike the base layer, which can use motion vectors for inter-frame prediction, the encoding of the enhancement layer (using the enhancement layer encoder (128)) does not use motion vectors. Instead, the enhancement layer encodes only spatial residuals, which means it does not explicitly track or compensate for movement between frames. Thus, enhancement layer data can be derived from the transform coding of the residual signal, which captures additional high-frequency details to refine the frame that is subsequently decoded at the target resolution.

[0039] However, the entire residual signal is not necessarily encoded in the enhancement layer. Instead, the enhancement layer encodes at least a portion (108) of the image frame to ensure that the frame can still be decoded at the target resolution. This enables a flexible encoding strategy in which residual data can be selectively encoded based on perceptual importance, bitrate constraints, or user-defined regions of interest (RoI). Thus, in some embodiments, the portion of the image frame encoded in the enhancement layer may correspond to a region of interest (RoI), which can be defined by scene analysis (e.g., by the video analysis component (134) of the encoding device (120)) or manually by the user. Such regions of interest (RoI) allow specific areas of the image, such as faces, text, or other high-detail areas, to be processed first, ensuring that these areas receive higher-quality enhancement while maintaining overall bitrate efficiency.

[0040] In some examples, the encoding device (120) encodes the first image frame (106a) using the Low Complexity Enhancement Video Coding (LCEVC) standard. LCEVC supports multiple enhancement layers, denoted as L1, L2, etc., which progressively refine the restored image. In some examples, two enhancement layers (L1, L2) are implemented. The first enhancement layer (L1) operates at an intermediate resolution and encodes additional details that enhance sharpness and texture at a resolution typically intermediate between the base layer and the target resolution. The second enhancement layer (L2) operates at the target resolution and refines the decoded image by adding fine details and high-frequency components that enhance visual fidelity. In some embodiments, the first resolution is 1 / 4 of the target resolution. Downscaling of the first image frame (106a) can be implemented in the image downscaler component (124).

[0041] For clarity and brevity, this disclosure describes hierarchical encoding using only the L2 enhancement layer. However, the described techniques may be equally applicable to multiple enhancement layers, such as when both L1 and L2 are used, or when additional enhancement layers are used as needed. The techniques described herein are applicable regardless of the number of enhancement layers used, thereby ensuring flexibility in various encoding configurations.

[0042] The encoding device (120) also includes an encoding scenario detector (122). The encoding scenario detector (122) is configured to detect an encoding scenario from a set of one or more predefined encoding scenarios. In FIG. 1, dotted arrows indicate data sets and their sources, and the encoding scenario detector (122) can use them to determine whether the current encoding scenario matches one of the predefined encoding scenario(s).

[0043] These predefined encoding scenarios correspond to situations where encoding of enhancement layer residuals becomes inefficient, which means that the enhancement layer requires a disproportionately high bitrate. These scenarios generally occur when the residuals between the base layer and the image frames of the video sequence (106) are very unpredictable, large in size, or change rapidly, making it difficult to compress efficiently.

[0044] One such encoding scenario involves continuous shaking of a camera (102) capturing a video sequence (106). When the camera (102) is continuously shaking, residuals move unpredictably between frames, increasing the encoding cost. In one example, an encoding scenario detector (122) receives sensor data from a gyroscope sensor (104) of the camera (102) and detects this encoding scenario by determining that the camera (102) is shaking based on an analysis of the sensor data. Since the gyroscope data provides real-time measurements of the camera's (102) movement, the system can detect continuous or irregular movements that may result in inefficient residual encoding.

[0045] In other examples, the encoding scenario detector (122) detects a camera shake scenario by analyzing motion vectors of the base layer of the encoded first image frame. These motion vectors may be provided by a base layer encoder (126) and represent displacement of image data between frames.

[0046] By analyzing motion vectors, the encoding scenario detector (122) can detect uniform motion across the entire image to determine whether the camera (102) is shaking. Specifically, if the motion vectors indicate that all or most of the area of ​​the image frame (106a) is moving in the same direction and with a similar magnitude, this suggests global motion, such as camera shake or panning. This distinguishes the motion vectors from local object movements that generally differ in direction and magnitude across the frame.

[0047] In some examples, among one or more predefined encoding scenarios, a second encoding scenario includes the ratio of the bitrate of the enhancement layer to the bitrate of the base layer exceeding a threshold ratio, or the bitrate of the enhancement layer exceeding a threshold. The encoding scenario detector (122) can perform this analysis using bitrate data provided by a multiplexer (130) that monitors the encoding bitrate of the enhancement layer for each frame. By comparing the bitrate of the enhancement layer to the bitrate of the base layer or to an absolute threshold, the encoding scenario detector (122) can determine whether excessive use of the enhancement layer is occurring. The absolute threshold may vary depending on the total bitrate of the encoded video stream (150) or may be a fixed value.

[0048] In some examples, the portion of image data encoded by the enhancement layer represents a region of interest (RoI) (108) within a scene depicted by a video sequence (106). In these scenarios, among one or more predefined encoding scenarios, a third encoding scenario includes the presence of continuous movement within the region of interest of the scene. In these examples, detection of the encoding scenario by the encoding scenario detector (122) includes analyzing the motion vectors of the base layer of a plurality of encoding image frames preceding the second encoding image frame in the encoded video stream, and detecting the third encoding scenario by determining that there is continuous movement within the region of interest of the scene based on the analysis. This may occur, for example, when a pendulum is swinging within a high-detail area of ​​a frame, or when a moving vehicle or athlete remains continuously within the region of interest (user-defined or auto-defined).

[0049] It should be noted that the set of predefined encoding scenarios is not limited to those described above and can be customized according to application requirements or use cases. Which encoding scenarios are relevant, such as dynamic lighting changes or abrupt scene transitions, may be defined by the implementer or the specific application environment.

[0050] Additionally, it should be noted that the division of functions and components described in relation to FIG. 1, including the arrangement of the encoding device (120), video camera (102), multiplexer (130), and other processing components, is merely illustrative. The described system architecture represents one possible form of implementation, and the technologies described herein are not limited to this specific structure.

[0051] In some embodiments, various components may be integrated into a single device, such as a dedicated encoder within the camera (102). In other examples, the functions may be distributed across multiple devices or servers. For example, base layer encoding and enhancement layer encoding may be performed on different hardware units or across cloud-based processing resources. Similarly, the encoding scenario detector (122) may operate locally within the encoding device (120) or run on a remote server that analyzes streaming data before making an adaptive encoding decision.

[0052] When a scenario is detected, the encoding device (120) is configured to encode a subsequent second image frame (106b) of the video sequence to a target resolution using a base layer and to indicate within the encoded video stream (150) that the second image frame (106b) can be decoded to a target resolution using only a base layer (e.g., using a metadata handler (132) or other methods further described below). This is now further explained in relation to FIG. 2.

[0053] FIG. 2 illustrates the function of an encoding device (120) by way of example. As previously described, the encoding device (120) receives an image frame (106). When or before receiving the image frame (106), an encoding scenario detector (122) determines whether an encoding scenario is detected from a set of one or more predefined encoding scenarios. If such a scenario is not detected (202), the encoding device (120) performs conventional hierarchical encoding techniques.

[0054] As previously described, under conventional hierarchical encoding, an image frame (106) is downscaled to a first resolution (and one or more intermediate resolutions as described above) and encoded into a base layer by a base layer encoder (126). To restore the image at a target resolution, the encoding device (120) determines one or more residuals (or differences) between the upscaled version of the downscaled image frame and the original image frame (106) as described above. This process is performed by a difference determination component (204) that calculates the difference to be encoded to restore lost details.

[0055] The calculated residuals are then encoded into one or more enhancement layers by an enhancement layer encoder (128), thereby ensuring that the finally restored image matches the target resolution.

[0056] When an encoding scenario is detected (202), the image frame is not downscaled. Instead, the base layer encoder (126) directly encodes the image frame (106) at the target resolution, ensuring that the frame at full resolution can be decoded using only the base layer. In some examples, where an enhancement layer is not required, the enhancement layer of the image frame (106) may be encoded using an empty data structure (206), such as an enhancement layer image frame containing no image data or minimal image data. The empty enhancement layer functions as a signal to the decoder that no additional enhancement information is required for the frame. This can be achieved by inserting a placeholder frame (206) containing minimal metadata to maintain stream continuity. In other examples, the encoding device (120) omits encoding the enhancement layer for the image frame (106). In this case, the decoding device is configured to recognize from the metadata within the received encoding video stream that the enhancement layer is not present or is not required for decoding. This can be achieved by including an explicit flag, marker, or metadata field within the encoding stream that indicates to the decoder that the base layer alone is sufficient to restore the image at the target resolution (e.g., using a metadata handler (132)).

[0057] FIGS. 3 and FIGS. 4 illustrate exemplary encoded video streams (300, 400), respectively, which correspond to a base layer of multiple image frames from the encoded video stream (150) illustrated in FIG. 1. As previously mentioned, the base layers of multiple image frames may be encoded using temporal prediction techniques that utilize inter-frame dependencies to improve compression efficiency. In these examples, the Group of Pictures (GOP) structure of the encoded video streams (300, 400) is set to 3, which means that two P-frames (predictive coding frames) follow each I-frame (intra-coding frame) before the next I-frame appears. This GOP structure is provided merely as an example, and the present method is not limited to a specific GOP length; any GOP structure may be applied depending on the encoding configuration, bitrate constraints, or application requirements. Additionally, B-frames (bidirectional prediction frames) may also be used within the GOP structure.

[0058] In the example of FIG. 3, an encoding scenario is detected at the point when the next image frame (312) in the encoding video stream (300) (containing encoded image frames (302-314)) is typically encoded as a P-frame. However, in some examples, as illustrated in FIG. 3, when an encoding scenario is detected, the base layer of the next encoded image frame (312) is instead encoded as a key frame (I-frame) having a target resolution. This approach ensures that the prediction chain is reset (i.e., the next encoded image frame is encoded as a P-frame referencing the image frame (312) encoded as a key frame), and allows subsequent frames to be encoded without relying on previous frames (e.g., image frames (308), (310)) encoded at a first lower resolution.

[0059] FIG. 4 illustrates an exemplary implementation of another image frame encoding when a scenario is detected. In this example, when a scenario is detected, the next encoded image frame (412) is encoded into a switch frame (SF) having a target resolution. The switch frame is a type of prediction (P) frame that enables a smooth transition between different encoding configurations, such as transitioning from image data of a lower resolution (first resolution) used in the preceding image frames (402) to (410) to image data of a higher resolution (target resolution) used in the encoded image frame (412). Unlike the key frame (312) illustrated in FIG. 3, the key frame is completely independent and does not refer to previous frames, but the switch frame (412) can still utilize motion prediction from reference frames (e.g., image frames (410), (408)), thereby ensuring that the transition to a new resolution (target resolution) is efficiently performed while reducing the bitrate.

[0060] Such a switch frame (412) (i.e., when the image data resolution of the SF (412) is increased compared to the resolution of the image data (410) of the reference frame (410)) can indicate to the decoder that the image frame (412) can be decoded at the target resolution using only the base layer.

[0061] FIG. 5 illustrates a flowchart of a method (500) for encoding a video sequence into an encoded video stream having a target resolution, as an example. The method (500) includes the following steps:

[0062] ㆍ A step of encoding a first image frame (S502), wherein the image frame is encoded to a first resolution lower than the target resolution using a base layer, and at least a portion of the image data of the first image frame is encoded using an enhancement layer to reach the target resolution;

[0063] ㆍ A step of detecting an encoding scenario from a set of one or more predefined encoding scenarios (S504); and

[0064] ㆍ A step of encoding a second image frame (S506), wherein the second image frame is a frame following the first image frame in a video sequence, and the second image frame is encoded to a target resolution using a base layer, and the second image frame is decoded to a target resolution using only the base layer within the encoded video stream.

[0065] The method (500) of FIG. 5 includes additional steps (S508 to S514) that can be implemented in some examples.

[0066] In some examples, the method (500) includes the step (S508) of continuously encoding a plurality of image frames after a second image frame within a video stream, and encoding each of the plurality of image frames to a target resolution using a base layer. This means that after an encoding scenario is detected and the encoding mode is switched, the base layer continues to encode a plurality of subsequent frames to the target resolution. The method (500) may also further include the step (S510) of determining whether the number of the plurality of image frames encoded to the target resolution using only the base layer exceeds a threshold number. The threshold number may correspond to a video stream duration of, for example, at least 3 seconds, 5 seconds, 9 seconds, etc.

[0067] If this condition is satisfied, the method (500) can analyze the video stream again to determine whether the encoding scenario that initially triggered the transition is still occurring, or whether another encoding scenario among the set of predefined encoding scenarios is currently present. If no encoding scenario is detected (S512), the method may transition to the step (S514) of returning to a normal hierarchical encoding including an enhancement layer. However, if one of the predefined encoding scenarios is still detected, the base layer-only encoding mode may be maintained for additional frames. Accordingly, the method (500) may include the step (S512) of detecting that no encoding scenario among the set of one or more predefined encoding scenarios is occurring; and the step (S514) of encoding the current image frame of the video sequence, wherein the current image frame is encoded to a first resolution using a base layer and at least a portion of the image data of the current image frame is encoded using an enhancement layer to reach a target resolution.

[0068] In addition to the method described in FIG. 5, any other techniques described in this specification may be implemented using a non-transient computer-readable storage medium that stores instructions that cause the device to perform the described encoding processes when executed in a device having one or more processors.

[0069] The device may utilize one or more processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any combination thereof. These processors may be part of a dedicated video encoding system, a general-purpose computing device, a cloud-based processing platform, or an embedded system within a camera or other video processing hardware.

[0070] The above non-transient computer-readable storage medium may include, but is not limited to, flash memory, solid-state drive (SSD), read-only memory (ROM), random access memory (RAM), magnetic storage, or other suitable storage media capable of storing executable instructions.

[0071] The above embodiments should be understood as exemplary examples of the present disclosure. It should be understood that any feature described in connection with any one embodiment may be used alone, in combination with other described features, or in combination with one or more features of other embodiments or any combination of other embodiments. Furthermore, equivalents and modifications not described above may also be adopted without departing from the scope of the present disclosure as defined by the appended claims.

Claims

Claim 1 A method for hierarchically video encoding a video sequence into an encoded video stream having a target resolution, wherein the encoded video stream comprises an enhancement layer and a single base layer, and the method comprises: a step of encoding a first image frame of the video sequence, wherein the base layer is used to encode the image frame to a first resolution lower than the target resolution, and wherein the enhancement layer is used to encode at least a portion of the image data of the first image frame to reach the target resolution, wherein the enhancement layer encodes the residual between the image data of the first image frame and the image data encoded in the single base layer, and wherein the residual is encoded at the target resolution; a step of detecting an encoding scenario from a set of one or more predefined encoding scenarios; and a step of encoding a second image frame of the video sequence, wherein the second image frame is a frame following the first image frame in the video sequence, wherein the single base layer is used to encode the second image frame to the target resolution, and wherein the second image frame within the encoded video stream is decoded to the target resolution using only the single base layer. Claim 2 A method according to claim 1, wherein the first encoding scenario among the one or more predefined encoding scenarios includes continuous shaking of the camera capturing the video sequence. Claim 3 In paragraph 2, the step of detecting the encoding scenario comprises the step of receiving sensor data from the gyro of the camera and detecting the first encoding scenario by determining that the camera is shaking based on the analysis of the sensor data. Claim 4 A method according to claim 2, wherein the step of detecting the encoding scenario comprises the step of detecting the first encoding scenario by analyzing motion vectors of a single base layer of the encoded first image frame and determining that the camera is shaking by comparing the motion vectors with image data of a reference encoding image frame used when encoding the first image frame and determining that the motion vectors correspond to uniform movement of the image data of the first image frame. Claim 5 A method according to claim 1, wherein among the one or more predefined encoding scenarios, a second encoding scenario includes a situation in which the ratio of the bit rate of the enhancement layer to the bit rate of the single base layer exceeds a threshold ratio, or the bit rate of the enhancement layer exceeds a threshold, and the step of detecting the encoding scenario includes the step of detecting the second encoding scenario by monitoring the bit rate of the enhancement layer of the encoded first image frame. Claim 6 A method according to claim 1, wherein the portion of the image data of the first image frame represents a region of interest within a scene depicted by the video sequence, and among the one or more predefined encoding scenarios, a third encoding scenario includes the existence of continuous movement within the region of interest of the scene, and the step of detecting the encoding scenario comprises: analyzing motion vectors of a single base layer of a plurality of encoding image frames preceding the second encoding image frame in an encoded video stream, and detecting the third encoding scenario by determining that the continuous movement exists within the region of interest of the scene based on the analysis. Claim 7 A method according to claim 1, wherein the step of encoding a second image frame of the video sequence comprises the step of encoding a single base layer of the second image frame into a key frame having the target resolution. Claim 8 A method according to claim 1, wherein the step of encoding a second image frame of the video sequence comprises the step of encoding a single base layer of the second image frame into a switch frame having the target resolution. Claim 9 In claim 1, the step of indicating that the second image frame within the encoded video stream is decoding to the target resolution using only the single base layer comprises at least one of the following: encoding the enhancement layer of the second image frame without image data; indicating in the metadata of the encoded image stream that the second image frame is decoding to the target resolution using only the single base layer; and encoding the single base layer of the second image frame into a switch frame. Claim 10 The method of claim 1 further comprises the step of continuously encoding a plurality of image frames after the second image frame in the video stream, wherein each of the plurality of image frames is encoded to the target resolution using the single base layer. Claim 11 In claim 10, the method further comprises: a step of determining that the number of the plurality of image frames exceeds a threshold number; a step of detecting that no encoding scenario among the set of one or more predefined encoding scenarios is occurring; and a step of encoding a current image frame of a video sequence, wherein the current image frame is encoded to the first resolution using the single base layer and at least a portion of the image data of the current image frame is encoded using the enhancement layer to reach the target resolution. Claim 12 A method according to claim 1, wherein the first image frame is encoded using the Low Complexity Enhancement Video Coding (LCEVC) coding standard. Claim 13 In claim 1, a single base layer of each of the first and second image frames is encoded using one of the following coding standards: AV1, AV2, and VP9. Claim 14 A non-transient computer-readable storage medium storing instructions for implementing the method according to paragraph 1 when executing it on a device having processing capability. Claim 15 An apparatus for hierarchically video encoding a video sequence into an encoded video stream having a target resolution, wherein the encoded video stream comprises an enhancement layer and a single base layer, and the apparatus comprises: encoding a first image frame of the video sequence, wherein the single base layer is used to encode the image frame to a first resolution lower than the target resolution, and wherein the enhancement layer is used to encode at least a portion of the image data of the first image frame to reach the target resolution, wherein the enhancement layer is configured to encode a residual between the image data of the first image frame and the image data encoded in the single base layer, and wherein the residual is configured to be encoded at the target resolution; wherein the apparatus comprises detecting an encoding scenario from a set of one or more predefined encoding scenarios; and encoding a second image frame of the video sequence, wherein the second image frame is a frame following the first image frame in the video sequence, wherein the single base layer is used to encode the second image frame to the target resolution, and wherein the apparatus comprises indicating that the second image frame within the encoded video stream is decodingable to the target resolution using only the single base layer.

Images