Controllable Video Coding Technology for Region of Interest

Abstract

Video encoding techniques include encoding at different bit rates or different qualities for one or more regions of interest or one or more non - regions of interest based on information including coordinates of one or more regions of interest, target complexity, residual encoder bit data, requested quality, difference between the current video data frame and the reconstructed video data frame, target quality, requested bit rate, frame target bit allocation, and encoded bit rate.

Description

BACKGROUND OF THE INVENTION

[0001] Many techniques are used to reduce the amount of data consumed for video transmission or storage. A common technique is to employ variable bitrate coding of video frame data. For example, a first bitrate can be used to encode one or more regions of interest (ROIs), and a second bitrate can be used to encode one or more non - regions of interest. Refer to Figure 1A , Figure 1A which shows an exemplary video frame image. The portion of the frame image containing jewelry is more important than the rest of the frame image which typically contains the background. The region of interest (ROI) 110 regarding the jewelry is specified by the bounding box 120, and the remaining portion is the non - region of interest 130. The detected region of interest (ROI) 110 is encoded using a higher bitrate, so the image of the jewelry will have better image quality than the portion of the non - region of interest 130 in the image that is encoded using a lower bitrate.

[0002] The detection of regions of interest and the variable bitrate coding of regions of interest and non - regions of interest are computationally intensive. Moreover, the variable bitrate coding is difficult to adjust. Therefore, it is necessary to continue to improve the variable bitrate coding of video images. SUMMARY OF THE INVENTION

[0003] The present technology can be best understood by reference to the following description and drawings, which are used to illustrate embodiments of the present technology directed to a controllable video processing system and method.

[0004] In one embodiment, a video processing unit includes a region of interest detector, a video encoder, and a rate controller. The region of interest detector is configured to receive an input video stream and determine one or more regions of interest of the image data frames of the input video stream. The video encoder is configured to perform differential encoding on the determined one or more regions of interest using a first bitrate and on one or more non - regions of interest using a second bitrate to generate a compressed bitstream of the image data frames. The rate controller is configured to control one or more parameters of the region of interest detection and control one or more parameters of the bitrate encoding based on one or more of a requested quality, an encoding quality estimate, and a complexity estimate.

[0005] In another embodiment, the video processing unit includes a region of interest detector, a rate controller, and a video encoder communicatively coupled together. The region of interest detector is configured to determine one or more regions of interest in an image data frame. The rate controller includes a region of interest / non-region of interest bit allocation unit, a region of interest quantization model unit, and a non-region of interest quantization model unit, etc. The region of interest / non-region of interest bit allocation unit is configured to generate a target bit allocation for the region of interest and a target bit allocation for the non-region of interest based on frame-level bit allocation, the coordinates of the one or more determined regions of interest, complexity estimates of the one or more regions of interest and the one or more non-regions of interest, and quality estimates of the one or more regions of interest and non-regions of interest.

[0006] The region of interest quantization model unit is configured to generate region of interest quantization parameters based on the target bit allocation for the region of interest. The non-region of interest quantization model unit is configured to generate non-region of interest quantization parameters based on the target bit allocation for the non-region of interest. The video encoder is configured to generate a compressed bitstream of the image data frame based on the region of interest quantization parameters and the non-region of interest quantization parameters.

[0007] In another embodiment, a video processing method includes: determining one or more regions of interest in a current video data frame; determining one or more quantization parameters or one or more distortion rate parameters for the one or more determined regions of interest and the one or more non-regions of interest in the current video data frame; and encoding the current video data frame into a compressed bitstream based on the one or more determined regions of interest, the one or more determined quantization parameters or distortion rate parameters, and a requested quality.

[0008] The above summary is provided to introduce some concepts in a simplified form that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present disclosure are illustrated in the drawings by way of example and not limitation, in which like reference numerals represent like elements, and in which:

[0010] Figure 1A and 1B shows an exemplary video frame image;

[0011] Figure 2 shows a block diagram of a video processing unit according to an embodiment of the present disclosure;

[0012] Figure 3 shows a flowchart of video processing according to an embodiment of the present disclosure.

[0013] Figure 4 Shows a block diagram of a video rate controller according to an embodiment of the present disclosure.

[0014] Figures 5A - 5D Shows the peak signal-to-noise ratio (PSNR) of an exemplary encoded frame according to an embodiment of the present disclosure.

[0015] Figure 6 Shows a block diagram of an exemplary processing unit including a video processing unit according to an embodiment of the present disclosure.

[0016] Figure 7 Shows a block diagram of an exemplary processing core according to an embodiment of the present disclosure. Detailed implementation

[0017] Embodiments of the present disclosure will now be described in detail, and examples thereof are illustrated in the accompanying drawings. Although the present disclosure is described in conjunction with these embodiments, it should be understood that they are not intended to limit the present disclosure to these embodiments. On the contrary, the present disclosure is intended to cover alternatives, modifications, and equivalents that are included within the scope defined by the appended claims. In addition, in the following detailed description of the present disclosure, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it can be understood that the present disclosure can be practiced without these specific details. In other instances, some well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring some aspects of the present disclosure.

[0018] In the following description, some embodiments of the present disclosure are presented in the form of routines, modules, logic blocks, and other symbolic representations of data operations within one or more electronic devices. The descriptions and representations are the means used by those skilled in the art to most effectively convey the substance of their work to other skilled persons in the art. Here, a routine, module, logic block, and / or the like is generally considered to be a self-consistent sequence of processes or instructions that result in a desired outcome. These processes include physical operations on physical quantities. Generally, although not necessarily, these physical operations take the form of electrical or magnetic signals that can be stored, transmitted, compared, and otherwise manipulated in an electronic device. For convenience and with reference to common usage, according to embodiments of the present disclosure, these signals are referred to as data, bits, values, elements, symbols, characters, terms, numbers, strings, and / or the like.

[0019] However, it should be remembered that these terms will be interpreted as referring to physical operations and quantities and are merely convenient labels and will be further interpreted in accordance with terms commonly used in the art. Unless expressly stated otherwise from the following discussion, it should be understood that, in the discussion of the present technology, the discussion using terms such as "receiving" refers to the actions and processes of an electronic device (e.g., an electronic computing device that manipulates and transforms data).

[0020] Data is represented as a physical quantity (e.g., an electrical quantity) within the logic circuits, registers, memories, and / or the like of the electronic device and is transformed into other data that is similarly represented as a physical quantity within the electronic device.

[0021] In this application, the use of a disjunctive is intended to include a conjunctive. The use of a definite or indefinite article is not intended to indicate cardinality. Specifically, a reference to "the" or "a" object is also intended to refer to one of a plurality of possible such objects. The use of terms such as "comprising," "including," etc. indicates the presence of the recited element but does not preclude the presence or addition of one or more other elements and / or groups of the recited elements. It should also be understood that although terms such as first, second, etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are used herein to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the embodiments. It should also be understood that when an element is referred to as "coupled" to another element, it may be directly or indirectly connected to the other element, or there may be intervening elements. In contrast, when an element is referred to as "directly connected" to another element, there are no intervening elements. It should also be understood that the term "and / or" includes any and all combinations of one or more related elements. It should also be understood that the phrases and terms used herein are for the purpose of description and should not be regarded as limiting.

[0022] Reference Figure 2 , Figure 2A video processing unit according to an embodiment of the present disclosure is shown. The video processing unit 200 includes a region of interest (ROI) detector 210, a video encoder 220, a rate controller 230, and a memory 240. The ROI detector 210 is configured to determine one or more regions of interest 110 and one or more non - regions of interest 130 in a frame of an input video stream 250. The term "region of interest" as used herein generally means the identification of an object in a dataset, including the identification of relevant object types. Regions where one or more object types of objects are not detected in an image are generally non - regions of interest. The video encoder 220 is configured to encode one or more regions of interest using a first bitrate and one or more non - regions of interest using a second bitrate to generate a compressed bitstream 260. The rate controller 230 is configured to control the parameters of the region of interest detection performed by the ROI detector 210 and the variable bitrate encoding performed by the video encoder 220. The memory 240 is configured to store, cache, or buffer frame data, region of interest data, non - region of interest data, encoded data, rate control data, intermediate result data, etc. The video processing unit 200 may be implemented in hardware, firmware, software, or a combination thereof. Reference will be made to Figure 3 further illustrate the operation of the video processing unit, Figure 3 A video processing method according to an embodiment of the present disclosure is shown.

[0023] In step 310, the ROI detector 210 receives the input video stream 250. The input video stream 250 received by the ROI detector 210 includes a plurality of image data frames. In step 315, the ROI detector 210 is configured to determine one or more regions of interest 110 in one or more image data frames of the input video stream 250. The one or more regions of interest may be determined by the ROI detector 210 based on various information, which will be described in more detail below. The various information includes information received from the rate controller 230 and the video encoder 220. The portion of the image data frame that is not within the determined one or more regions of interest 110 is considered as one or more non - regions of interest 130. The ROI detector 210 is further configured to determine the region of interest priority of the determined one or more regions of interest based on various information (including information received from the rate controller 230 and the video encoder 220).

[0024] The rate controller 230 may receive the current video data frame and the determined one or more regions of interest from the region of interest detector 210. The rate controller 230 is used to determine the quantization parameters or rate–distortion optimization (RDO) parameters for the determined one or more regions of interest and one or more non–regions of interest in step 320. The rate controller 230 may determine the quantization parameters or rate–distortion optimization parameters for the determined one or more regions of interest and one or more non–regions of interest based on various information, including the information received from the region of interest detector 210 and the video encoder 220. In one embodiment, the information includes the requested bitrate and / or the requested quality received by the rate controller 230. The information may also include the reconstructed video and / or the bit data of the residual encoder received from the video encoder 220. The information may also include the coordinates of the regions of interest received from the region of interest detector 210. The information may also include the target complexity estimate generated by the rate controller 230. The rate controller 230 may use this information to determine the quantization parameter (QP) and / or the rate–distortion optimization parameter for each of the one or more regions of interest and one or more non–regions of interest. Optionally, the rate controller determines the quantization parameters and / or the rate–distortion parameters for each of the one or more regions of interest and one or more non–regions of interest, whose rate of change is constrained. Constraining the rate of change of the quantization parameters and / or the rate–distortion parameters can improve the encoding performance of the video encoder 220.

[0025] The video encoder 220 may receive the current video data frame, the one or more determined regions of interest from the region of interest detector 210, and the determined quantization parameters and rate–distortion parameters (constrained or unconstrained) from the rate controller 230. In step 325, the video encoder 220 is used to generate a compressed bitstream of the current video data frame based on the constrained or unconstrained quantization parameters and / or rate–distortion parameters.

[0026] The compressed bitstream can be generated by the video encoder 220 based on various information, including the information received from the region of interest detector 210 and the rate controller 230. The rate controller 230 can provide information to the video encoder 220 so that the video encoder 220 encodes videos with specified quality for one or more regions of interest and one or more non-regions of interest respectively. In one embodiment, when the compressed bitstream of the current image data frame has a predefined total bit budget, if more bits are allocated to one or more regions of interest, the quality of the regions of interest can be improved. In another embodiment, when maintaining the same quality for the determined one or more regions of interest, bits are saved by degrading the quality of the non-regions of interest. In another embodiment, when precise bits are allocated to one or more regions of interest and one or more non-regions of interest, the quality of one or more regions of interest or one or more non-regions of interest is controlled. At step 330, the compressed bitstream is output by the video encoder 220. In one embodiment, the compressed bitstream is transmitted across one or more communication links with limited bandwidth. In another embodiment, the compressed bitstream is stored on a readable storage medium of one or more computing devices.

[0027] The video encoder 220 is also used to feedback the encoded information to the rate controller 230 and the region of interest detector 210. The rate controller 230 is also used to feedback the rate information to the region of interest detector 210. In one embodiment, the video encoder 220 feeds back the remaining bit budget information as the encoding bit rates of the determined one or more regions of interest and one or more non-regions of interest, the quality of the determined one or more regions of interest and one or more non-regions of interest, and so on. The rate controller 230 can use the information fed back from the video encoder 220 to the rate controller 230 to adjust the specification of the quantization parameters and / or the distortion rate parameters of the determined one or more regions of interest and one or more non-regions of interest in the next video frame, and / or other information generated by the rate controller 230. The region of interest detector 210 can use the information fed back from the video encoder 220 to the region of interest detector 210 to adjust the determined one or more regions of interest, the coordinates of the regions of interest, the priority of the regions of interest, and / or other information generated by the region of interest detector 210. In one embodiment, the information fed back from the video encoder 220 to the region of interest detector 210 indicates that the remaining bit budget is too low, the quality of one or more regions of interest is too high, the quality of the non-regions of interest is too low, and so on. The region of interest detector 210 uses the indication that the remaining bit budget is too low, the quality of one or more regions of interest, the quality of one or more non-regions of interest is too low, etc. to reduce the size of the determined one or more regions of interest in the video data frame to save more bits when encoding the video data frame into a compressed data stream.

[0028] Now referring to Figure 4 , a video rate controller according to an embodiment of the present disclosure is shown in the figure. The rate controller 230 is communicatively coupled to the region of interest detector 210 and the video encoder 220. The rate controller 230 includes a group of pictures (GOP) bit allocation unit 405 for receiving a requested bitrate and the GOP bit allocation of the video source. The video source may include a plurality of video data frames. The GOP bit allocation unit 405 is configured to perform GOP-level bit allocation based on the video data frames and the requested bitrate. The frame bit allocation unit 410 of the rate controller 230 is configured to perform frame-level bit allocation based on the GOP bit allocation to generate frame target bit allocations.

[0029] The region of interest / non-region of interest bit allocation unit 415 of the rate controller 230 is configured to receive the coordinates of one or more regions of interest determined by the region of interest detector 210 and the frame target bit allocations. The region of interest / non-region of interest bit allocation unit 415 is further configured to receive the target complexity estimates of one or more regions of interest and non-regions of interest estimated by the region of interest / non-region of interest complexity estimation unit 420, as further described below. The region of interest / non-region of interest bit allocation unit 415 is further configured to receive the quality estimates of one or more regions of interest and one or more non-regions of interest estimated by the region of interest / non-region of interest quality estimation unit 425, as further described below. The region of interest / non-region of interest bit allocation unit 415 is configured to allocate bits for one or more determined regions of interest and one or more non-regions of interest respectively based on the frame target bit allocations, the coordinates of one or more determined regions of interest, the target complexity estimates of one or more regions of interest and non-regions of interest, and the target quality estimates of one or more regions of interest and non-regions of interest.

[0030] The rate-lambda-quantization model unit 430 for the regions of interest of the rate controller 230 may receive the target bit allocations for the regions of interest from the region of interest / non-region of interest bit allocation unit 415. The rate-lambda-quantization model unit 430 for the regions of interest is configured to generate quantization parameter (QP) and / or rate-distortion optimization (RDO) parameters for one or more determined regions of interest based on the target bit allocations for the regions of interest.

[0031] The rate-lambda-quantization model unit 435 of the non-interested region of the rate controller 230 may receive the target bit allocation of the non-interested region from the region-of-interest / non-interested region bit allocation unit 415. The rate-lambda-quantization model unit 435 of the non-interested region is used to generate quantization parameters (QP) and / or rate-distortion optimization (RDO) parameters for one or more non-interested regions based on the target bits of the non-interested region.

[0032] The non-interested region constraint unit 440 receives the quantization parameters (QP) and / or rate-distortion optimization (RDO) parameters of the determined one or more regions of interest and one or more non-interested regions. The non-interested region constraint unit 440 is used to constrain the changes of the quantization parameters (QP) and / or rate-distortion optimization (RDO) parameters of the determined one or more regions of interest and one or more non-interested regions within a predetermined change rate range to achieve quality stability.

[0033] The video encoder 220 receives the constrained quantization parameters (QP) and rate-distortion optimization (RDO) parameters. The video encoder 220 is used to generate a compressed bitstream for the received video frame data based on the constrained quantization parameters (QP) and / or rate-distortion optimization (RDO) parameters. Optionally, the video encoder 220 generates a compressed bitstream based on the unconstrained quantization parameters (QP) and / or rate-distortion optimization (RDO) parameters. The video encoder 220 also generates feedback after encoding the current frame and provides it to the region-of-interest / non-interested region complexity estimation unit 420, the region-of-interest / non-interested region quality estimation unit 425, and the region-of-interest detector 210. In one embodiment, the video encoder 220 provides the residual encoder bit information to the region-of-interest / non-interested region complexity estimation unit 420. The video encoder 220 may also provide the reconstructed video frame data to the region-of-interest / non-interested region quality estimation unit 425. The video encoder 220 may also provide the encoded bit rate information to the region-of-interest detector 210.

[0034] The region-of-interest / non-interested region complexity estimation unit 420 may receive the residual encoder bit information from the video encoder 220. The region-of-interest / non-interested region complexity estimation unit 420 is used to estimate the target complexity of the region of interest and the non-interested region based on the residual encoder bits of the previous frame or the current frame. In one embodiment, the residual encoder bits are the mean absolute difference (MAD), the mean square absolute error (MSE), etc.

[0035] In one embodiment, the lower bit limits of the determined one or more regions of interest and regions of non - interest can be calculated by the region - of - interest / region - of - non - interest bit allocation unit 415 based on the complexity values generated by the region - of - interest / region - of - non - interest complexity estimation unit 420. The frame target bits minus the lower bit limits of the determined one or more regions of interest and regions of non - interest are the remaining bits, and the remaining bits can be used to perform quality control on the one or more regions of interest and regions of non - interest to reduce the chance that the determined one or more regions of interest and regions of non - interest consume too many bits and cause bit shortages during the generation of the compressed bitstream for the next image data frame.

[0036] The region - of - interest / region - of - non - interest quality estimation unit 425 can receive a quality information request. The requested quality information indicates the quality of the requested determined one or more regions of interest and the quality of the requested one or more regions of non - interest. In one embodiment, the requested quality information is a difference factor between the quality of the determined one or more regions of interest and the quality of the one or more regions of non - interest. For example, the requested quality can be expressed as the difference between the quality of the determined one or more regions of interest and the quality of the one or more regions of non - interest (e.g., 0 dB, 1 dB, 2 dB, etc.). The region - of - interest / region - of - non - interest quality estimation unit 425 is used to estimate the target quality of the determined one or more regions of interest and the one or more regions of non - interest based on the requested quality information. The region - of - interest / region - of - non - interest quality estimation unit 425 can also receive the input video source and the reconstructed video from the video encoder 220. Further, the region - of - interest / region - of - non - interest quality estimation unit 425 can estimate the target quality of the determined one or more regions of interest and the one or more regions of non - interest based on the difference between the input video source and the reconstructed video. The target quality of the determined one or more regions of interest and the one or more regions of non - interest can be output to the region - of - interest / region - of - non - interest bit allocation unit 415 and the region - of - interest detector 210.

[0037] In one embodiment, the region of interest / region of non - interest quality estimation unit 425 is used to adjust the weights of the target bit allocation for the determined one or more regions of interest and regions of non - interest using the feedback information from the video encoder 220. In one embodiment, if the quality of the determined one or more regions of interest is too low for the current frame (t - frame), more bits are allocated to the determined one or more regions of interest in the next frame (t + 1 - frame) to improve the quality. In one embodiment, the quality of a video data frame is some metric from the original frame and the reconstructed frame, such as mean absolute value (MAD), peak signal - to - noise ratio (PSNR), structural similarity index matric (SSIM), video multimethod assessment fusion (VMAF), etc. The quality can also be the difference of MAD, PSNR, SSIM, VMAF, etc.

[0038] The region of interest detector 210 receives the frame target bit allocation, the target quality, and the coding bit rate. The region of interest detector 210 is used to adjust the determined one or more regions of interest and one or more regions of non - interest based on the frame target bit allocation, the target quality, and the coding bit rate. In one embodiment, the size of the determined one or more regions of interest is reduced or increased based on the frame target bit allocation, the target quality, and the coding bit rate. For example, as Figure 1B shown, if the frame target bit allocation and the coding bit rate indicate that the target quality estimation cannot be met, the size of one or more regions of interest 140 is reduced. In another embodiment, the number of determined regions of interest is reduced or increased based on the frame target bit allocation, the target quality, and the coding bit rate.

[0039] Now refer to Figures 5A - 5D , Figures 5A - 5D shows the peak signal - to - noise ratio (PSNR) of an exemplary encoded frame according to an embodiment of the present disclosure. Figure 5A shows the peak signal - to - noise ratio of multiple frames of a 720p video encoded at 1.5 megabits per second (Mbps), where the peak signal - to - noise ratio (PSNR) difference between the region of interest and the region of non - interest is 0 dB, 1 dB, and 2 dB. The lower curve 505 shows the 0 - dB peak signal - to - noise ratio difference between the region of interest and the region of non - interest. The middle curve 510 shows the 1 - dB peak signal - to - noise ratio difference between the region of interest and the region of non - interest. The upper curve 515 shows the 2 - dB peak signal - to - noise ratio difference between the region of interest and the region of non - interest. Figure 5BShows a 0 dB peak signal-to-noise ratio (PSNR) difference between the region of interest 520 and the non-region of interest 525 in the encoded frame. Figure 5C Shows a 1 dB peak signal-to-noise ratio (PSNR) difference between the region of interest 530 and the non-region of interest 535 in the encoded frame. Figure 5D Shows a 2 dB peak signal-to-noise ratio (PSNR) difference between the region of interest 530 and the non-region of interest 535 in the encoded frame.

[0040] Now refer to Figure 6 , Figure 6 Shows an exemplary processing unit including a video processing unit according to an embodiment of the present disclosure. The processing unit 605 includes one or more communication interfaces, such as a peripheral component interface (e.g., PCIe4 interface) 610 and an inter-integrated circuit (I2C) interface 615, an on-chip circuit tester, such as a joint test action group (JTAG) engine 620, a direct memory access engine 625, a command processor (CP) 630, and one or more cores 635 - 650. The one or more cores 635 - 650 may be coupled in a direction loop bus configuration. The one or more cores 635 - 650 may execute one or more sets of computing device-executable instructions to perform one or more functions, including but not limited to region of interest detection, rate control, and video encoding as described above. One or more functions may be executed on a single core 635 - 650, may be distributed across multiple cores 635 - 650, may be executed together with one or more other functions on one or more cores, and so on.

[0041] The processing unit 605 may be a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a vector processor, a memory processing unit, etc., or a combination thereof. In one embodiment, one or more processors 605 are implemented in a computing device, and the computing device includes but is not limited to a cloud computing platform, an edge computing device, a server, a workstation, a personal computer (PC), etc.

[0042] Now refer to Figure 7 , Figure 7A block diagram of an exemplary processing core according to an embodiment of the present disclosure is shown. The processing core 700 includes a tensor engine (TE) 710, a pooling engine (PE) 715, a memory copy engine (ME) 720, a sequencer (SEQ) 725, an instructions buffer (IB) 730, a local memory (LM) 735, and a constant buffer (CB) 740. The local memory 735 can pre-install the weights of the model and can dynamically store the activation data in use. The constant buffer 740 stores constants for batch normalization, quantization, etc. The tensor engine 710 is used to accelerate fused convolution and / or matrix multiplication. The pooling engine 715 supports operations such as pooling, interpolation, region of interest, etc. The memory copy engine 720 is used for data replication, matrix transpose, etc. between and / or within processing cores. The tensor engine 710, the pooling engine 715, and the memory copy engine 720 can run in parallel. The sequencer 725 can coordinate the operations of the tensor engine 710, the pooling engine 715, the memory copy engine 720, the local memory 735, and the constant buffer 740 according to the instructions from the instructions buffer 730. The processing core 700 can provide efficient computing for video coding for functions such as region of interest detection, bitrate control, variable bitrate video coding, etc. under the control of operation fused coarse-grained instructions. To understand the embodiments of the present disclosure, it is not necessary to describe the exemplary processing core 700 in detail, so it will not be further described below.

[0043] The embodiments of the present disclosure help to improve bit allocation for variable bitrate video coding and / or improve the method for determining quantization parameters. When a video frame has a target bit budget, the quality of the region of interest can be improved by allocating more bits to the coding of the region of interest. When maintaining a predetermined quality of the region of interest, bits can be saved by reducing the quality of the non-region of interest. When accurately allocating bits to the region of interest and the non-region of interest, the target quality of the region of interest and the non-region of interest can be achieved. The size of the region of interest can also be adjusted to achieve a given quality, bit budget, etc. The feedback from the rate controller to the region of interest detector and / or the feedback from the video encoder to the rate controller and / or the region of interest detector all help to optimize performance, reduce computational workload, reduce bandwidth usage, and / or reduce energy consumption. As a result, variable bitrate coding control is improved.

[0044] The above description of the specific embodiments of the present disclosure is for illustration and description purposes. They are not intended to be exhaustive or to limit the application to the precise forms disclosed. Obviously, many modifications and variations are possible in light of the above teachings. The embodiments selected and described above are intended to best explain the principles of the present disclosure and its practical applications, so that others skilled in the art can make the best use of the technology and various embodiments with various modifications suitable for the specific purposes contemplated. The scope of the present disclosure is defined by the appended claims and their equivalents herein.

Claims

1. A video processing unit, comprising: A region of interest detector, configured to receive an input video stream and determine one or more regions of interest in the image data frames of the input video stream, and adjust the determined one or more regions of interest based on video encoder feedback information and rate controller feedback information; A video encoder, which differentially encodes the adjusted determined one or more regions of interest using a first bit rate and one or more non - interested regions using a second bit rate based on adjusted rate controller parameters to generate a compressed bit stream of the image data frame, and generates video encoder feedback information; And A rate controller, configured to control one or more parameters of the region of interest detection performed by the region of interest detector, and based on requested quality control, control one or more parameters of the bit rate encoding performed by the video encoder. The rate controller is further configured to adjust one or more parameters of the bit rate encoding of the video encoder based on the feedback information received from the video encoder, and generate rate controller feedback information.

2. The video processing unit according to claim 1, wherein, The rate controller is further configured to control one or more parameters of the bit rate encoding of the video encoder based on the difference in the requested quality between the determined one or more regions of interest and one or more non - interested regions.

3. The video processing unit according to claim 1, wherein, The rate controller is further configured to control one or more parameters of the bit rate encoding of the video encoder based on the difference between the image data frame and the reconstructed data frame of the compressed bit stream.

4. The video processing unit according to claim 1, wherein, The feedback information received by the rate controller from the video encoder includes the bit information of the residual encoder received from the video encoder.

5. The video processing unit according to claim 1, wherein, The rate controller is further configured to generate one or more parameters of the bit rate encoding performed by the video encoder, and the one or more parameters include quantization parameters of the region of interest and quantization parameters of the non - interested region.

6. The video processing unit according to claim 5, wherein, The rate controller is further configured to constrain the change rate of the quantization parameters of the region of interest and the quantization parameters of the non - interested region.

7. The video processing unit according to claim 1, wherein The rate controller is further configured to generate one or more parameters of the bit rate encoding performed by the video encoder, and the one or more parameters include distortion rate optimization parameters of the region of interest and distortion rate optimization parameters of the non - interested region.

8. The video processing unit according to claim 7, wherein, The rate controller is further configured to constrain the change rate of the distortion rate optimization parameters of the region of interest and the distortion rate optimization parameters of the non - interested region.

9. A video processing unit, comprising: A region of interest detector, configured to determine one or more regions of interest in an image data frame, and adjust the determined one or more regions of interest based on video encoder feedback information and rate controller feedback information; A rate controller, communicatively coupled to the region of interest detector, and generates rate controller feedback information, and the rate controller includes: A region of interest / region of non - interest bit allocation unit, configured to generate a target bit allocation for the determined one or more regions of interest and the one or more regions of non - interest based on frame - level bit allocation, the coordinates of the determined one or more regions of interest, the target complexity of the one or more regions of interest and the one or more regions of non - interest, and the target quality of the one or more regions of interest and the one or more regions of non - interest; A quantization model unit for regions of interest, configured to generate quantization parameters for regions of interest based on the target bit allocation of the one or more regions of interest; and A quantization model unit for regions of non - interest, configured to generate quantization parameters for regions of non - interest based on the target bit allocation of the one or more regions of non - interest; and A video encoder communicatively coupled to the region of interest detector and the rate controller, the video encoder being configured to generate a compressed bitstream of the image data frame based on the quantization parameters of the regions of interest and the quantization parameters of the regions of non - interest, and to generate video encoder feedback information.

10. The video processing unit according to claim 9, wherein, The rate controller further includes: A region of interest / region of non - interest quality estimation unit, configured to determine a target quality based on the requested quality and the difference between the image data frame and the reconstructed data frame; and The region of interest / region of non - interest bit allocation unit is further configured to generate a target bit allocation for the determined one or more regions of interest and the one or more regions of non - interest based on the target quality.

11. The video processing unit according to claim 10, wherein, The requested quality includes the difference in quality between the one or more regions of interest and the one or more regions of non - interest.

12. The video processing unit according to claim 10, wherein, The region of interest detector is further configured to determine one or more regions of interest in the image data frame based on the target bit information and the target quality from the rate controller.

13. The video processing unit according to claim 12, wherein, The region of interest detector is further configured to determine one or more regions of interest in the image data frame based on the encoded bit rate information from the video encoder.

14. The video processing unit according to claim 9, wherein, The rate controller further includes: A region of interest / region of non - interest complexity estimation unit, configured to estimate the target complexity based on the bit information of the residual encoder from the video encoder; and The region of interest / region of non - interest bit allocation unit is further configured to allocate bits for the determined one or more regions of interest and the one or more regions of non - interest based on the bit information of the residual encoder.

15. The video processing unit according to claim 9, wherein, The rate controller further includes a region of interest / region of non - interest quantization constraint unit, configured to constrain the change rate of the quantization parameters of the regions of interest and the quantization parameters of the regions of non - interest.

16. The video processing unit according to claim 9, wherein, The quantization model unit for regions of interest further generates the quantization parameters for regions of interest based on the target bit allocation of the regions of interest and the rate λ quantization model; and The quantization model unit for regions of non - interest is configured to generate the quantization parameters for regions of non - interest based on the target bit allocation of the regions of non - interest and the rate λ quantization model.

17. A video processing method, comprising: The region of interest detector receives a data frame of the current video; The region of interest detector determines one or more regions of interest in the data frame of the current video; The rate controller determines one or more quantization parameters or one or more distortion rate parameters for the determined one or more regions of interest and one or more non - regions of interest in the data frame of the current video; The region of interest detector adjusts the determined one or more regions of interest in the data frame of the current video based on the video coding information of the data frame of the previous video and the rate control information of the data frame of the previous video; The rate controller adjusts the rate control information of the data frame of the current video based on the video coding information of the data frame of the previous video; The video encoder encodes the data frame of the current video into a compressed bitstream based on the adjusted determined one or more regions of interest and the one or more quantization parameters or the one or more distortion rate parameters and the requested quality; and The video encoder outputs the compressed bitstream of the current video.

18. The method according to claim 17, further comprising determining the one or more quantization parameters or the one or more distortion rate parameters based on information, the information including one or more coordinates of the one or more regions of interest, target complexity, bit data of the residual encoder, requested quality, the difference between the data frame of the current video and the reconstructed data frame of the video, target quality, and requested bit rate.

19. The method according to claim 17, further comprising constraining the rate of change of the determined one or more quantization parameters or the one or more distortion rate parameters.

Images