Human vision system adaptive video coding

Abstract

The invention discloses human vision system adaptive video coding. An example apparatus for encoding a video frame includes a mask selector to select a subset of visual masks according to an actual target compression rate and a GOP configuration; and a complexity estimator for estimating a picture-level spatial / temporal complexity of the current frame. The example apparatus also includes a GOP adaptive visual mask selector to specify a visual mask corresponding to the estimated spatial and temporal complexity values from a subset of visual masks; and a good enough picture QP deriver for deriving a good enough picture QP value using the visual mask. The example apparatus also includes an adjuster to adjust a sufficient picture QP value based on the block-level human vision system sensitivity and the statistical information of the encoded frame to obtain a final human vision system QP map.

Description

[0001] This application is a divisional application of the invention patent application with the application date of November 27, 2020, application number 202011354226.3, and name "Human Visual System Adaptive Video Coding". Technical Field

[0002] The present invention relates to the field of video, and in particular to human visual system adaptive video coding. Background Art

[0003] Video codecs are used to efficiently encode video for transmission between devices or for playback on devices with specific playback support. For example, one such codec is the H.264 / AVC codec standard. Summary of the invention

[0004] An aspect of the present invention provides an apparatus for encoding a video sequence, the apparatus comprising: an interface circuit; machine-readable instructions; and at least one processor circuit, which is programmed by the machine-readable instructions to: access the video sequence; determine the complexity of blocks in the video sequence; select a quantization parameter value based on the complexity and human visual perception; encode blocks of the video sequence using the quantization parameter value; and enable transmission of the encoded blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram illustrating an example system for human visual system adaptive video encoding;

[0006] Figure 2 is a process flow chart of a method for target rate adaptive visual mask calculation;

[0007] Figure 3 is a process flow diagram of a method for spatial / temporal complexity analysis of a current frame;

[0008] Figure 4 is a process flow chart of the method used for HVS QP diagram derivation;

[0009] Figure 5 is a process flow chart of a method for adjusting historical complexity visual masking;

[0010] Figure 6 is a process flow diagram of a method for human visual system adaptive video encoding;

[0011] Figure 7 is a block diagram illustrating an example computing device that can adaptively encode video frames based on the human visual system; and

[0012] Figure 8 is a block diagram illustrating a computer-readable medium storing code for human visual system adaptive encoding.

[0013] The same numbers are used throughout the disclosure and drawings to refer to similar components and features. Numbers in the 100 series refer to those originally Figure 1 ; numbers in the 200 series indicate features originally found in Figure 2 Features found in etc. DETAILED DESCRIPTION

[0014] Bitrate control is a technique that distinguishes one video coding solution from other video coding solutions. The technique of achieving good enough video quality at a minimum bit rate is desired by many video applications, especially for high-quality content storage. In many cases, controlling the frame size to a predictable value can be used by applications (especially network-related applications). Through optimal bit allocation, subjective and objective improvements can be obtained. In order to achieve these improvements, given a target bit rate, most rate control algorithms adjust the quantization parameter (QP) value of each picture to control the bits generated from these pictures so that the target bit rate can be achieved. As used herein, a picture can refer to a frame in a video or other picture series.

[0015] Visual masking occurs when the visibility of a part or object in the picture is reduced due to the presence of another object and the background. Due to visual masking, the human eye is more sensitive to quality changes in areas with flat and simple textures in space. Humans are also more sensitive to small changes in time. In addition, the human visual experience is also affected by the expectations of the specific viewer for quality. The lower the expectation, the easier it is to get a good visual experience. In an embodiment, the quality expectation is highly correlated with the compression rate and the viewer's impression of the video clip just played.

[0016] For example, the HEVC test model (HM) reference rate control algorithm allocates a size for each frame of the video based on the position of the frame within the group of pictures (GOP) and the target compression rate. As used herein, a GOP is a collection of continuous pictures within a coded video stream. Each coded video stream consists of continuous GOPs. As an example, if GOP 8 is used, each frame within the GOP of eight frames always uses a fixed percentage of bits allocated to all eight frames, and the same number of bits are always allocated to each grouping of eight frames. This approach does not take into account the characteristics of the human visual system (HVS). In particular, this type of encoding cannot efficiently adapt to the content of the video, such as spatial complexity and temporal correlation. As a result, too many bits are wasted on some scenes and clips, and the quality of some complex clips or video scenes is low. In order to achieve sufficiently good quality content for high-quality content storage, a larger target bit rate is usually selected, and the associated storage requirements are significant.

[0017] The present technology generally relates to human visual system adaptive coding. In an embodiment, a target rate adaptive visual mask calculation is performed to derive a set of visual masks corresponding to a target compression rate and spatial / temporal complexity through offline training. During the encoding process, a spatial / temporal complexity analysis is applied to the current frame, and a visual mask is selected based on an estimate of the spatial / temporal complexity. A picture-level good enough QP is derived from the selected visual mask. Based on the human visual system (HVS) analysis, the picture-level good enough QP is adjusted for each block of the frame. In an embodiment, the HVS analysis includes block-level human visual system sensitivity and statistics of the encoded frame. Based on the HVS analysis, the picture-level good enough QP is adjusted for each, so that the HVS sensitive area of ​​the picture is encoded using a smaller QP value, while the non-HVS sensitive area of ​​the picture is encoded using a larger QP value. The derived block-level QP value can be further adjusted according to the historical complexity analysis. These adjustments are used to generate a good enough HVS QP map for the current frame. The HVS QP map is used to constrain the typical QP derived from the variable bit rate (VBR) rate control and generate the final QP for each block of the current frame.

[0018] As used herein, a QP value indicates the amount of quantization to be applied to a particular macroblock when encoding. A macroblock or block refers to a processing unit of a frame specified by a coding standard. The QP value regulates how much spatial detail is preserved. A picture-level QP value is a QP value applied to the entire frame, where all blocks of the frame use the picture-level QP. A block-level QP value is a QP value applied to each block of a frame. For a given frame, the block-level QP value can be different for each block. When the QP value is very small, almost all details are preserved. When the QP value increases, some details are aggregated, causing the bit rate to drop at the expense of some increase in distortion and some loss in quality. For example, in the H.264 coding standard, QP values ​​range from 0 to 51.

[0019] Figure 1 is a block diagram illustrating an example system for human visual system adaptive video encoding. The example system 100 may be used respectively Figure 2 , 3 , 4, 5 and 6 of methods 200, 300, 400, 500 and 600, wherein Figure 7 1 is implemented in a computing device 700 in FIG. As shown, the example system 100 includes a picture input 102 , a target bit rate input 104 , and a GOP structure and picture type input 106 .

[0020] For each video codec or video coding standard, a target rate adaptive visual mask calculation 108 is performed to derive a visual mask set for all target compression rates supported by the video codec or video coding standard. The visual mask set for each standard can be based on space / time complexity and GOP configuration, and the visual mask set is derived by offline training. During encoding, a subset of visual masks is selected at GOP visual mask selection 110. In an embodiment, a subset of visual masks is selected based on the actual target compression rate and GOP configuration associated with the current frame. A visual mask generally refers to a predefined visibility of one or more parts of a frame. For example, a high visibility mask is used because the encoding artifacts of the less complex parts of the image are more visible than the more complex parts of the image, taking into account the general space / time complexity.

[0021] During the encoding process of each frame, a spatial / temporal complexity analysis 112 is applied. The spatial / temporal complexity analysis 112 can determine the complexity of the current frame because the number of bits generated for each frame is used as a complexity measure for the input picture 102. In addition, the complexity can be determined as a prediction distortion value associated with the current picture. The spatial / temporal complexity analysis 112 estimates the picture-level spatial and temporal complexity of the current frame that is input to the good enough picture QP derivation 114. At the good enough picture QP derivation 114, a specified visual mask is selected from a subset of visual masks that correspond to the estimated spatial and temporal complexity values ​​of the current frame obtained by the spatial / temporal complexity analysis 112. The visual mask is used to estimate the good enough picture QP value.

[0022] A good enough picture level QP value is a QP value for all blocks of the current frame that are expected to generate a coded picture with the following visual quality: based on the complexity of the current frame and the target bit rate, the visual quality is acceptable. When a set of visual masks is generated, a good enough picture level QP value can be derived during offline training. Each mask corresponds to a set of specified QP values. Masks corresponding to high visibility are mapped to smaller QPs, while masks corresponding to low visibility are mapped to larger QPs.

[0023] The estimated good enough picture QP value is sent to the HVS based block adjustment module 116, where the good enough picture QP value is adjusted based on the block level HVS sensitivity of each macroblock / block of the current frame. In doing so, HVS sensitive blocks can use smaller QP values, while non-HVS sensitive blocks can use larger QP values. The adjusted good enough picture QP value for the block of the current frame is sent to the historical complexity visual masking adjustment module 118. At the historical complexity visual masking adjustment module 118, the adjusted QP is further adjusted according to the actual encoding statistics of the previous encoded frames. The final HVS QP map is obtained for each block in the current frame and is used in the final QP decision block 120. The picture level QP estimate 124 with variable bit rate control VBR is determined and provided as input to the final QP decision block 120. The picture level QP estimate 124 with variable bit rate control (VBR) is derived to derive the initial QP to encode the current frame. In an embodiment, given a target bit rate, an initial QP is determined for the entire frame, and a rate control algorithm adjusts the QP value for each picture in a GOP to control the bits generated from the picture so that the target bit rate can be achieved.

[0024] At the final QP decision block 120, for each block, the initial QP is compared with the HVS QP map obtained from the historical complexity visual masking adjustment module 118. For each block, if the initial QP value is less than the QP value of the corresponding block from the HVS QP map, the QP value from the HVS QP map is selected for the current block and used in the subsequent encoding process. The encoder 126 continues encoding the current frame at box 126 and provides the encoding statistics of the previously encoded frame to the historical complexity visual masking adjustment 118 and the picture QP estimation 124 with VBR rate control.

[0025] Figure 1 The diagram is not intended to indicate that the example system 100 will include Figure 1 Instead, the example system 100 may use fewer or additional components. Figure 1 The components not shown in FIG. 8 are implemented (e.g., additional statistics, QP, encoder, etc.).

[0026] In this way, the present technology provides a target rate adaptive visual mask estimation, which defines a good enough picture quality derived from a good enough picture QP value. In addition, the historical complexity analysis exploits the visual memory effect of the human visual system to dynamically adjust the target rate adaptive visual mask. In addition, a system according to the present technology is developed to integrate the extracted visual mask into traditional rate control. As a result, the rate control automatically allocates fewer bits on simple scenes and uses the saved bits on complex scenes. The overall quality can be improved. In addition, the present technology uses a single pass encoding instead of requiring multiple passes for encoding.

[0027] Figure 2 is a process flow diagram of a method 200 for target rate adaptive visual mask computation. The example method 200 may be Figure 1 System 100, Figure 7 The computing device 700 or Figure 8 The invention is implemented in a computer readable medium 800 of the present invention. As described above, human visual quality impression is highly correlated with expectation. The expectation is related to the target bit rate (compression rate) and historical quality information. Therefore, the target bit rate adaptation can be combined with the traditional space / time complexity analysis to derive a new target adaptive visual mask.

[0028] At block 202, a complexity-based visual mask is calculated for one or more video segments. To calculate the set of visual masks, a target bitrate is received and one or more video segments are encoded with different QP values. The video segments may be obtained from a test dataset that includes video segments for offline training with different complexities. A maximum QP value that can generate visually lossless quality for each video segment is derived. To determine the maximum QP, an expert group evaluates the encoded bitstream displayed in real time and finds the maximum QP that can produce visually lossless quality. The evaluation may be performed for each video coding standard. In an embodiment, the expert group may set the maximum QP value for the segment based on their own visual observations of the impact of the QP value on each segment. In an embodiment, a spatial / temporal analysis is applied to segments that share the same visually lossless QP value. Segments that share the same visually lossless QP are grouped together. A spatial / temporal complexity analysis is applied to each group and an average complexity value for the group is obtained. Then, a complexity-based visual mask is calculated for each segment group. Therefore, a complexity-based visual mask is established, in which complexity is taken as input and group QP is taken as output.

[0029] At block 204, each of the plurality of segments is encoded using a different compression rate. The different compression rates are derived from a target bit rate. The target bit rate may be identified based on the particular video coding standard being used. During the encoding, for each range of compression rates, an average QP value is calculated. The average QP value may be referred to as a good enough QP value.

[0030] At block 206, for each compression rate or target bit rate used, the visual mask from block 202 is adjusted to generate a specified visual mask for the corresponding compression rate. In an embodiment, if the average QP value obtained at block 204 is greater than the median of the group QP obtained at block 202, the visual mask is adjusted by gradually increasing the QP values ​​produced by each complexity. The increase step size is different for each complexity range. In an embodiment, the larger the complexity, the larger the increase step size in QP. A good enough two-dimensional QP lookup table is obtained, which takes complexity and compression rate as input for lookup. In an embodiment, Figure 2 The target rate visual mask calculation described in can be applied to different GOP configurations to generate a good enough QP lookup table accordingly. During the encoding process, a QP lookup table corresponding to the actual GOP and compression rate is selected.

[0031] This process flow diagram is not intended to indicate that the blocks of the example method 200 will be executed in any particular order, or that all blocks will be included in every case. In addition, any number of additional blocks not shown may be included within the example method 200 depending on the details of the particular implementation.

[0032] As an example, consider a frame level complexity divided into 20 levels. An example QP lookup table for high compression ratios may be shown below. Each QP value corresponds to a frame level complexity level. {18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37}

[0033] Figure 3 is a process flow diagram of a method 300 for space / time complexity analysis of a current frame. The example method 300 may be Figure 1 System 100, Figure 7 The computing device 700 or Figure 8 The invention is implemented in a computer readable medium 800.

[0034] Method 300 provides a complexity analysis with a simplified coding prediction process for downsampled video. At block 302, an input picture is downsampled. At block 304, the downsampled input picture is encoded with a constant QP. At block 306, a prediction distortion for each picture is obtained to be used as a spatial / temporal complexity estimate. The prediction distortion for each picture (from the simplified coding prediction process for downsampled video) is used as a picture complexity estimate. In an embodiment, the prediction distortion represents the prediction distortion between the original frame and the corresponding reference frame.

[0035] This process flow diagram is not intended to indicate that the blocks of the example method 300 will be executed in any particular order, or that all blocks will be included in every case. In addition, any number of additional blocks not shown may be included within the example method 300 depending on the details of the particular implementation.

[0036] Figure 4 is a process flow diagram of a method 400 for HVS QP map derivation. The example method 400 may be Figure 1 System 100, Figure 7 The computing device 700 or Figure 8 The invention is implemented in a computer readable medium 800.

[0037] At block 402, each block is classified according to its prediction distortion. In an embodiment, a threshold may be applied to classify each block into two classes. The first class of blocks has a prediction distortion above the threshold. In an embodiment, the threshold may be determined by empirical investigation. The second class of blocks has a prediction distortion less than or equal to the threshold.

[0038] At block 404, the block variance is used to determine a positive delta QP for a first class of blocks having prediction distortion above a threshold. The positive delta QP for each block is applied as an increase in QP above the picture level QP. At block 406, the block variance is used to determine a negative delta QP for a second class of blocks having prediction distortion less than or equal to the threshold. The negative delta QP for each block is applied as a decrease in block QP relative to the picture level QP. Figure 4 The described HVS-based block adjustment of block QP can be further adjusted using historical complexity visual masking adjustment.

[0039] This process flow diagram is not intended to indicate that the blocks of the example method 400 will be executed in any particular order, or that all blocks will be included in all cases. In addition, any number of additional blocks not shown may be included within the example method 400 depending on the details of the specific implementation.

[0040] Figure 5 is a process flow diagram of a method 500 for adjusting historical complexity visual masking. The example method 500 may be Figure 1 System 100, Figure 7 The computing device 700 or Figure 8 The invention is implemented in a computer readable medium 800.

[0041] At block 502, the average historical QP value is compared to the derived picture-level good enough QP value. Figure 1 The derived picture-level good enough QP is obtained at block 114 of Figure 1 The encoding statistics of previously encoded frames of the encoder 126 of the encoder 126 are obtained. The average historical QP value of the previously encoded picture can be, for example, the average picture-level QP of the four most recently encoded frames. At box 504, the average historical QP value of the previously encoded picture is compared with the derived picture-level good enough QP value. If the average historical QP value is less than the derived picture-level good enough QP value, the processing flow continues to box 506. At box 506, the derived picture-level good enough QP is adjusted. In particular, if the picture-level good enough QP value is greater than the average historical QP plus a threshold, the final good enough QP value is set equal to the average historical QP value plus the threshold. If the derived picture-level good enough QP is less than or equal to the average historical QP, the processing flow continues to box 508. At box 508, the picture-level good enough QP is used for the HVS QP map. As described above, the picture-level good enough QP can be used in box 120 ( Figure 1 ) to make the final QP decision.

[0042] This process flow diagram is not intended to indicate that the blocks of the example method 500 will be executed in any particular order, or that all blocks will be included in all cases. In addition, depending on the details of the specific implementation, any number of additional blocks not shown may be included within the example method 500.

[0043] Figure 6 is a process flow diagram of a method 600 for human visual system adaptive video encoding. The example method 600 may be Figure 1 System 100, Figure 7 The computing device 700 or Figure 8 In addition, the example method 600 can be implemented in a computer readable medium 800 of the embodiment of the present invention. Figure 2-5 The methods 200, 300, 400 and 500 are implemented.

[0044] At box 602, a subset of visual masks is selected. At box 604, a picture-level spatial / temporal complexity of the current frame is estimated. At box 606, a visual mask is specified from the subset of visual masks based on the estimated spatial / temporal complexity of the current frame. At box 608, a picture-level good enough QP value is derived using the visual mask. At box 610, the picture-level good enough QP value is adjusted based on block-level human visual system sensitivity. At box 612, the picture-level good enough QP value is further adjusted based on encoding statistics from previously encoded frames to obtain a final human visual system QP map. In an embodiment, the block-level QP value from the HVS QP map may be compared to an initial QP. The initial QP may be derived using variable bitrate control. After the final QP decision, the current frame may be encoded using the final QP.

[0045] This process flow diagram is not intended to indicate that the blocks of the example method 600 will be executed in any particular order, or that all blocks will be included in all cases. In addition, depending on the details of the specific implementation, any number of additional blocks not shown may be included within the example method 600.

[0046] Reference now Figure 7, a block diagram of an example computing device that can adaptively encode video frames based on the human visual system is shown. The computing device 700 can be, for example, a laptop computer, a desktop computer, a tablet computer, a mobile device, or a wearable device. In some examples, the computing device 700 can be a streaming video capture device, such as a security camera or other camera. The computing device 700 may include a central processing unit (CPU) 702 configured to execute stored instructions, and a memory device 704 storing instructions executable by the CPU 702. The CPU 702 can be coupled to the memory device 704 via a bus 706. In addition, the CPU 702 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. In addition, the computing device 700 can include more than one CPU 702. In some examples, the CPU 702 can be a system on a chip (SoC) with a multi-core processor architecture. In some examples, the CPU 702 can be a dedicated digital signal processor (DSP) for image processing. The memory device 704 can include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. For example, the memory device 704 may include dynamic random access memory (DRAM).

[0047] The memory device 704 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. For example, the memory device 704 may include dynamic random access memory (DRAM).

[0048] The computing device 700 may also include a graphics processing unit (GPU) 708. As shown, the CPU 702 may be coupled to the GPU 708 via a bus 706. The GPU 708 may be configured to perform any number of graphics operations within the computing device 700. For example, the GPU 708 may be configured to render or manipulate graphical images, graphical frames, videos, etc., to be displayed to a user of the computing device 700.

[0049] The memory device 704 may include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. For example, the memory device 704 may include a dynamic random access memory (DRAM). The memory device 704 may include a device driver 710 configured to execute instructions for training a plurality of convolutional neural networks to perform sequence-independent processing. The device driver 710 may be software, an application, application code, etc.

[0050] The CPU 702 may also be connected via the bus 706 to an input / output (I / O) device interface 712 configured to connect the computing device 700 to one or more I / O devices 714. The I / O devices 714 may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices 714 may be built-in components of the computing device 700, or may be devices connected externally to the computing device 700. In some examples, the memory 704 may be communicatively coupled to the I / O devices 714 via direct memory access (DMA).

[0051] CPU 702 may also be linked via bus 706 to a display interface 716 configured to connect computing device 700 to a display device 718. Display device 718 may include a display screen that is a built-in component of computing device 700. Display device 718 may also include a computer monitor, television, or projector that is internal to computing device 700 or externally connected to computing device 700.

[0052] Computing device 700 also includes a memory device 720. Memory device 720 is a physical memory such as a hard drive, an optical drive, a thumb drive, a drive array, a solid state drive, or any combination thereof. Memory device 720 may also include a remote storage drive.

[0053] The computing device 700 may also include a network interface controller (NIC) 722. The NIC 722 may be configured to connect the computing device 700 to a network 724 via the bus 706. The network 724 may be a wide area network (WAN), a local area network (LAN), or the Internet, among others. In some examples, the device may communicate with other devices via wireless technology. For example, the device may communicate with other devices via a wireless LAN connection. In some examples, the device may communicate with other devices via a wireless LAN connection. or similar technologies to connect and communicate with other devices.

[0054] Computing device 700 also includes camera 726. For example, camera 726 can include one or more imaging sensors. In some examples, camera 726 can include a processor for generating video frames.

[0055] The computing device 700 also includes an HVS adaptive encoder 728. For example, the HVS adaptive encoder 728 can be used to adaptively encode a video frame based on detected motion. The HVS adaptive encoder 728 may include a complexity estimator 730, a mask selector 732, a good enough QP deriver 734, an HVS / history adjuster 736, a final QP selector 738, and an encoder 740. In some examples, each component 730-740 of the HVS adaptive encoder 728 may be a microcontroller, an embedded processor, or a software module. The complexity estimator 730 may estimate the picture-level spatial / temporal complexity of the current frame. The mask selector 732 may specify a visual mask from a subset of visual masks based on the estimated spatial / temporal complexity. The good enough QP deriver 734 may use the visual mask to derive a picture-level good enough QP. The HVS / history adjuster 736 may adjust the picture-level good enough QP based on the block-level human visual system sensitivity. The HVS / history adjuster 736 can also adjust the picture-level good enough QP based on encoding statistics from previously encoded frames to obtain a final human visual system QP map. The final QP selector 738 can compare the block-level QP value from the HVS QP map with the initial QP to make a final QP decision. The initial QP can be derived using variable bitrate control. The encoder 740 can encode the current frame using the final QP.

[0056] Figure 7 The block diagram is not intended to indicate that computing device 700 will include Figure 7 Instead, computing device 700 may include fewer or additional components. Figure 7 Components not shown in FIG. 7 , such as additional buffers, additional processors, etc. Depending on the details of the specific implementation, the computing device 700 may include Figure 7 702. In addition, any functionality of the complexity estimator 730, the mask selector 732, the good enough QP deriver 734, the HVS / history adjuster 736, the final QP selector 738, and the encoder 740 may be implemented in part or in whole in hardware and / or in the processor 702. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processor 702, or in any other device. In addition, any functionality of the CPU 702 may be implemented in part or in whole in hardware and / or in the processor. For example, the functionality of the HVS adaptive encoder 728 may be implemented using an application specific integrated circuit, in logic implemented in a processor, in logic implemented in a dedicated graphics processing unit such as the GPU 708, or in any other device.

[0057] Figure 8800 is a block diagram illustrating a computer readable medium 800 storing code for human visual system adaptive encoding. The computer readable medium 800 can be accessed by a processor 802 via a computer bus 804. In addition, the computer readable medium 800 may include code configured to direct the processor 802 to perform the methods described herein. In some embodiments, the computer readable medium 800 may be a non-transitory computer readable medium. In some examples, the computer readable medium 800 may be a storage medium.

[0058] like Figure 8 As shown, various software components discussed herein may be stored on one or more computer-readable media 800. For example, the complexity estimator module 806 may estimate the picture-level spatial / temporal complexity of the current frame. The mask selector module 808 may specify a visual mask from a subset of visual masks based on the estimated spatial / temporal complexity. The good enough QP deriver module 810 may derive a picture-level good enough QP using the visual mask. The HVS / history adjuster module 812 may adjust the picture-level good enough QP based on the block-level human visual system sensitivity. The HVS / history adjuster module 812 may also adjust the picture-level good enough QP based on encoding statistics from previously encoded frames to obtain a final human visual system QP map. The final QP selector module 816 may compare the block-level QP value from the HVS QP map with the initial QP to make a final QP decision. The initial QP may be derived using variable bitrate control. The encoder module 818 may encode the current frame using the final QP.

[0059] Figure 8 The block diagram is not intended to indicate that the computer-readable medium 800 will include Figure 8 In addition, depending on the details of the specific implementation, the computer readable medium 800 may include Figure 8 Any number of additional components not shown.

[0060] Example 1 is an apparatus for encoding a video frame. The apparatus includes a mask selector for selecting a subset of visual masks based on an actual target compression rate and a group of pictures (GOP) configuration. The apparatus also includes a complexity estimator for estimating a picture-level spatial / temporal complexity of a current frame; and a GOP-adaptive visual mask selector for specifying a visual mask corresponding to the estimated spatial and temporal complexity values ​​from a subset of visual masks. In addition, the apparatus also includes a good enough picture QP deriver for deriving a good enough picture QP value using the visual mask; and an adjuster for adjusting the good enough picture QP value based on block-level human visual system sensitivity and statistical information of the encoded frame to obtain a final human visual system QP map.

[0061] Example 2 includes the apparatus of Example 1, including or excluding the optional features. In this example, a final human visual system QP map specifies a QP value for each block of the current frame.

[0062] Example 3 includes the apparatus of any one of Examples 1 to 2, including or excluding the optional features. In this example, the final QP value is obtained by comparing the initial QP value with the HVS QP map for each block, and in response to the initial QP value being less than the corresponding QP value in the HVS QP map, the corresponding QP value in the HVS QP map is selected for the current block and used in a subsequent encoding process.

[0063] Example 4 includes the apparatus of any one of Examples 1 to 3, including or excluding the optional features. In this example, target rate adaptive visual mask computation is performed to derive a set of visual masks for supported target compression rates, spatial / temporal complexities, and GOP configurations for a video coding standard through offline training.

[0064] Example 5 includes the apparatus of any one of Examples 1 to 4, including or excluding the optional features. In this example, the picture level spatial / temporal complexity is estimated based on the prediction distortion value obtained for each picture.

[0065] Example 6 includes the apparatus of any one of Examples 1 to 5, including or excluding the optional features. In this example, a GOP adaptive mask selector uses a GOP configuration as input and a group QP as output to specify a visual mask.

[0066] Example 7 includes the apparatus of any one of Examples 1 to 6, including or excluding the optional features. In this example, the visual mask is used to derive a two-dimensional good enough QP lookup table that takes complexity and compression rate as input for the lookup.

[0067] Example 8 includes the apparatus of any one of Examples 1 to 7, including or excluding the optional features. In this example, during an encoding process, a QP lookup table corresponding to a current GOP and a compression rate is selected.

[0068] Example 9 is a method for encoding a video frame. The method includes selecting a subset of visual masks based on an actual target compression rate and a group of pictures (GOP) configuration. The method also estimates a picture-level spatial / temporal complexity of a current frame and specifies a visual mask corresponding to the estimated spatial and temporal complexity values ​​from the subset of visual masks. The method also includes deriving a good enough picture QP value using the visual mask and adjusting the good enough picture QP value based on the sensitivity of the block-level human visual system and statistical information of the encoded frame to obtain a final human visual system QP map.

[0069] Example 10 includes the method of Example 9, including or excluding the optional features. In this example, the final human visual system QP map specifies a QP value for each block of the current frame.

[0070] Example 11 includes the method of any one of Examples 9 to 10, including or excluding the optional features. In this example, the final QP value is obtained by comparing the initial QP value with the HVS QP map for each block, and in response to the initial QP value being less than a corresponding QP value in the HVS QP map, the corresponding QP value in the HVS QP map is selected for the current block and used in a subsequent encoding process.

[0071] Example 12 includes the method of any one of Examples 9 to 11, including or excluding the optional features. In this example, target rate adaptive visual mask calculation is performed to derive a set of visual masks for target compression rates, spatial / temporal complexities, and GOP configurations supported by a video coding standard through offline training.

[0072] Example 13 includes the method of any one of Examples 9 to 12, including or excluding the optional features. In this example, the picture level spatial / temporal complexity is estimated based on the prediction distortion value obtained for each picture.

[0073] Example 14 includes the method of any one of Examples 9 to 13, including or excluding the optional features. In this example, the method includes specifying a visual mask using complexity as the input and using the group QP as an output.

[0074] Example 15 includes the method of any one of Examples 9 to 14, including or excluding the optional features. In this example, the visual mask is used to derive a two-dimensional good enough QP lookup table that takes complexity and compression rate as inputs for the lookup.

[0075] Example 16 includes the method of any one of Examples 9 to 15, including or excluding the optional features. In this example, during the encoding process, a QP lookup table corresponding to a current GOP and compression rate is selected.

[0076] Example 17 is at least one computer-readable medium for encoding a video frame, having instructions stored therein. The computer-readable medium includes instructions to direct a processor to select a subset of visual masks based on an actual target compression rate and a group of pictures (GOP) configuration. The instructions also direct the processor to estimate the image-level spatial / temporal complexity of the current frame and specify a visual mask corresponding to the estimated spatial and temporal complexity values ​​from the subset of visual masks. In addition, the instructions direct the processor to use the visual mask to derive a good enough picture QP value, and adjust the good enough picture QP value based on block-level human visual system sensitivity and statistics of the encoded frame to obtain a final human visual system QP map.

[0077] Example 18 includes the computer readable medium of Example 17, including or excluding the optional features. In this example, a final human visual system QP map specifies a QP value for each block of the current frame.

[0078] Example 19 includes the computer-readable medium of any one of Examples 17 to 18, including or excluding the optional features. In this example, the final QP value is obtained by comparing the initial QP value with the HVS QP map for each block, and in response to the initial QP value being less than a corresponding QP value in the HVS QP map, the corresponding QP value in the HVS QP map is selected for the current block and used in a subsequent encoding process.

[0079] Example 20 includes the computer readable medium of any one of Examples 17 to 19, including or excluding the optional features. In this example, target rate adaptive visual mask computation is performed to derive a set of visual masks for supported target compression rates, spatial / temporal complexities, and GOP configurations for a video coding standard through offline training.

[0080] Example 21 includes the computer readable medium of any one of Examples 17 to 20, including or excluding the optional features. In this example, the picture level spatial / temporal complexity is estimated based on the prediction distortion value obtained for each picture.

[0081] Example 22 includes the computer readable medium of any one of Examples 17 to 21, including or excluding the optional features. In this example, the computer readable medium includes specifying a visual mask using complexity as input and using a group QP as output.

[0082] Example 23 includes the computer readable medium of any one of Examples 17 to 22, including or excluding the optional features. In this example, the visual mask is used to derive a two-dimensional good enough QP lookup table that takes complexity and compression rate as input for the lookup.

[0083] Example 24 includes the computer readable medium of any one of Examples 17 to 23, including or excluding the optional features. In this example, during an encoding process, a QP lookup table corresponding to a current GOP and a compression rate is selected.

[0084] Not all components, features, structures, characteristics, etc. described and illustrated herein need to be included in one or more specific aspects. If the specification indicates that a component, feature, structure, or characteristic "may", "could", "can", or "can" be included, then the specific component, feature, structure, or characteristic does not have to be included. If the specification or claims refer to "an" element, this does not mean that there is only one element. If the specification or claims refer to "additional" elements, it does not exclude the presence of more than one additional element.

[0085] It should be noted that although some aspects have been described with reference to specific embodiments, according to some aspects, other embodiments are possible. In addition, the arrangement and / or order of circuit elements or other features shown in the drawings and / or described herein need not be arranged in the specific manner shown and described. According to some aspects, many other arrangements are possible.

[0086] In each system shown in the accompanying drawings, in some cases, elements can each have the same reference numeral or different reference numerals to indicate that the elements represented can be different and / or similar. However, elements can be flexible enough to have different embodiments and work with some or all of the systems shown or described herein. The various elements shown in the accompanying drawings can be the same or different. Which is called the first element and which is called the second element is arbitrary.

[0087] It should be understood that the details in the foregoing examples may be used anywhere in one or more aspects. For example, all optional features of the computing device described above may also be implemented relative to the methods or computer-readable media described herein. In addition, although flow charts and / or state diagrams may be used herein to describe various aspects, the technology is not limited to these figures or the corresponding descriptions herein. For example, the process does not need to move through each of the boxes or states shown, or move in exactly the same order as shown and described herein.

[0088] The present technology is not limited to the specific details listed herein. In fact, those skilled in the art who benefit from this disclosure will recognize that many other variations can be made to the foregoing description and drawings within the scope of the present technology. Therefore, the appended claims including any modifications thereto define the scope of the present technology.

Claims

1. A device for encoding a video sequence, the device comprising: Interface circuit; machine-readable instructions; as well as at least one processor circuit to be programmed by the machine-readable instructions to: accessing the video sequence; determining a complexity of a block in the video sequence; selecting a quantization parameter value based on the complexity and human visual perception; encoding a block of the video sequence using the quantization parameter value; and The encoded blocks are transmitted.

2. The device according to claim 1, wherein: The processor circuit is further configured to select the quantization parameter value based on activity within the block.

3. The apparatus of claim 1, further comprising a decoder for decoding the block.

4. The device according to claim 3, wherein: The decoder is for assembling the video sequence based on the decoded blocks.

Images