Code rate control method and device based on region of interest, equipment and medium

Abstract

The embodiment of the invention provides a code rate control method and device based on a region of interest, electronic equipment, a storage medium and a computer program product, and relates to the technical field of image processing. The method comprises the following steps: respectively determining a first code control reference value and a second code control reference value of a region of interest and a non-region of interest according to code rate estimation of a to-be-coded video frame; and determining a quantization parameter adjustment value of each coding block based on the first code control reference value and the second code control reference value, and performing code rate control on each coding block based on the quantization parameter adjustment value. According to the method, differentiated code rate control is carried out on the region of interest and the non-region of interest, so that the subjective impression of a video picture is better. The first code control reference value and the second code control reference value are respectively determined through code rate estimation of the to-be-coded video frame, so that the overall code rate fluctuation of the video is kept stable. And through block-level quantization parameter adjustment, the overall image quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a rate control method, device, electronic device, storage medium and computer program product based on a region of interest. Background Art

[0002] In real-time transmission scenarios, viewers pay more attention to the face area in the picture and pay relatively less attention to the background. However, traditional encoding technologies often fail to make full use of information such as face detection to optimize the encoding quality of key areas, resulting in unclear portraits and loss of facial details, which affects the user's subjective experience. To solve this problem, face detection technology can be combined with the video encoding process. After the client identifies the area of ​​interest, it provides this information to the encoder so that the encoder can prioritize the protection and optimization of important parts of the data such as the face. Through this targeted strategy, not only can the quality of the face area in the video be significantly improved, but the overall viewing experience can also be improved, allowing users to get a clearer and more natural picture experience.

[0003] Bitrate control aims to optimize bandwidth usage, ensuring video clarity while reducing bandwidth consumption. Its core is to adjust the quantization parameter (QP) to balance image quality and bitrate size: a smaller QP value brings higher quality but larger bitrate, and vice versa. Combined with the region of interest recognition technology, the quality of key areas such as faces can be improved preferentially while maintaining overall coding efficiency, achieving a better visual experience and bandwidth utilization. Summary of the invention

[0004] Embodiments of the present disclosure provide a rate control method, device, electronic device, storage medium, and computer program product based on a region of interest.

[0005] According to a first aspect of an embodiment of the present disclosure, a rate control method based on a region of interest is provided, comprising: determining a target of interest in a video frame to be encoded; based on the target of interest, determining at least one region of interest and a region of non-interest in the video frame to be encoded; performing rate estimation on the video frame to be encoded to obtain a first estimated bit number; based on the first estimated bit number, determining a first coding reference value and a second coding reference value corresponding to the region of interest and the region of non-interest, respectively; determining a quantization parameter adjustment value corresponding to each coding block in the video frame to be encoded based on the first coding reference value and the second coding reference value; and performing rate control on each coding block based on the quantization parameter adjustment value.

[0006] In some exemplary embodiments of the present disclosure, determining at least one region of interest and region of non-interest in the to-be-encoded video frame based on the target of interest includes: generating a binary image based on the target of interest; using the binary image to calibrate the first region of interest and the first region of non-interest; corroding the first region of interest to obtain a corroded first region of interest; and dilating the corroded first region of interest to determine the at least one region of interest and region of non-interest.

[0007] In some exemplary embodiments of the present disclosure, the magnitude of the expansion process is not less than the magnitude of the corrosion process.

[0008] In some exemplary embodiments of the present disclosure, the method further includes: counting the area of ​​the at least one region of interest; calculating the area ratio of the at least one region of interest in the video frame to be encoded; and in response to the area ratio being within a preset area ratio range, performing the bit rate control on the video frame to be encoded.

[0009] In some exemplary embodiments of the present disclosure, determining the first coding reference value and the second coding reference value corresponding to the region of interest and the region of non-interest respectively according to the first estimated number of bits includes: setting a first coding control setting value corresponding to the region of interest; based on the first coding control setting value, performing a code rate estimation on the at least one region of interest to obtain a second estimated number of bits corresponding to the region of interest; based on the first estimated number of bits and the second estimated number of bits, calculating a third estimated number of bits corresponding to the region of non-interest; calculating the second coding control setting value corresponding to the region of non-interest according to the third estimated number of bits; determining that the first coding control setting value is the first coding control reference value, and the second coding control setting value is the second coding control reference value.

[0010] In some exemplary embodiments of the present disclosure, determining that the first code control setting value is the first code control reference value and the second code control setting value is the second code control reference value includes: determining whether the second code control setting value is greater than a non-interest area adjustment threshold; in response to the second code control setting value being greater than the non-interest area adjustment threshold, adjusting the first code control setting value and recalculating the second code control setting value; in response to the second code control setting value being not greater than the non-interest area adjustment threshold, determining that the first code control setting value is the first code control reference value and the second code control setting value is the second code control reference value.

[0011] In some exemplary embodiments of the present disclosure, the method also includes: obtaining a first complexity and a second complexity corresponding to the at least one region of interest and the non-region of interest, respectively; calculating the second estimated number of bits based on the first complexity and the first code control setting value; and calculating the second code control setting value based on the second complexity and the third estimated number of bits.

[0012] In some exemplary embodiments of the present disclosure, determining the quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame according to the first coding reference value and the second coding reference value includes: determining the first coding block in the to-be-encoded video frame; the first coding block belongs to each of the coding blocks; determining the first quantization parameter adjustment value of the first coding block relative to each of the regions of interest according to the first coding reference value and the second coding reference value; and determining the minimum value among each of the first quantization parameter adjustment values ​​as the quantization parameter adjustment value corresponding to the first coding block.

[0013] In some exemplary embodiments of the present disclosure, determining the first quantization parameter adjustment value of the first coding block relative to each of the regions of interest based on the first coding reference value and the second coding reference value includes: in response to the first coding block being located in a target region of interest, determining a rectangular region corresponding to the target region of interest; determining a center point of the rectangular region and a first distance from a vertex to the center point; determining a second distance from the first coding block to the center point; and determining the first quantization parameter adjustment value of the first coding block relative to the target region of interest based on the first distance, the second distance and the first coding reference value.

[0014] In some exemplary embodiments of the present disclosure, determining the first quantization parameter adjustment value of the first coding block relative to each of the regions of interest based on the first coding reference value and the second coding reference value includes: in response to the first coding block not being located in the target region of interest, determining a rectangular area corresponding to the target region of interest; determining a center point of the rectangular area, and a first distance from a vertex to the center point; determining a second distance from the first coding block to the center point; determining that the first coding block is located in a quadrant based on the center point; determining a third distance from a vertex of the to-be-encoded video frame in the quadrant to the center point; and determining the first quantization parameter adjustment value of the first coding block relative to the target region of interest based on the first distance, the second distance, the third distance, and the second coding reference value.

[0015] In some exemplary embodiments of the present disclosure, the rate control of each coding block based on the quantization parameter adjustment value includes: based on a block-level code control algorithm, obtaining a second quantization parameter adjustment value corresponding to a second coding block; the second coding block belongs to each of the coding blocks; determining whether the rate control direction of the second quantization parameter adjustment value is consistent with the rate control direction of the quantization parameter adjustment value of the second coding block; in response to the consistency of the rate control direction, superimposing the second quantization parameter adjustment value with the quantization parameter adjustment value of the second coding block, and performing rate control on the second coding block; in response to the inconsistency of the rate control direction, adjusting the second quantization parameter adjustment value based on an adjustment coefficient; superimposing the adjusted second quantization parameter adjustment value with the quantization parameter adjustment value of the second coding block, and performing rate control on the second coding block.

[0016] In some exemplary embodiments of the present disclosure, determining the target of interest in the video frame to be encoded includes: obtaining region of interest identification information corresponding to the video frame to be encoded; the region of interest identification information is used to identify at least one region of interest; and determining the target of interest based on the region of interest identification information.

[0017] According to a second aspect of an embodiment of the present disclosure, a rate control device based on a region of interest is provided, comprising: a target determination module, configured to determine a target of interest in a video frame to be encoded; a region determination module, configured to determine at least one region of interest and a region of no interest in the video frame to be encoded based on the target of interest; a rate estimation module, configured to perform rate estimation on the video frame to be encoded to obtain a first estimated bit number; a coding control reference value determination module, configured to determine, according to the first estimated bit number, a first coding control reference value and a second coding control reference value corresponding to the region of interest and the region of no interest, respectively; a quantization parameter adjustment value determination module, configured to determine, according to the first coding control reference value and the second coding control reference value, a quantization parameter adjustment value corresponding to each coding block in the video frame to be encoded; and a rate control module, configured to perform rate control on each coding block based on the quantization parameter adjustment value.

[0018] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the executable instructions to implement any one of the rate control methods based on the region of interest.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the rate control methods based on a region of interest.

[0020] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, any one of the rate control methods based on a region of interest is implemented.

[0021] The rate control method based on the region of interest provided by the embodiment of the present disclosure determines the region of interest and the region of non-interest in the video frame to be encoded; according to the rate estimation of the video frame to be encoded, the first code control reference value and the second code control reference value of the region of interest and the region of non-interest are respectively determined; based on the first code control reference value and the second code control reference value, the quantization parameter adjustment value of each coding block is respectively determined, and the rate control of each coding block is performed based on the quantization parameter adjustment value. This method makes the subjective perception of the video picture better by performing differentiated rate control on the region of interest and the region of non-interest. By respectively determining the first code control reference value and the second code control reference value based on the rate estimation of the video frame to be encoded, it is ensured that the overall video rate fluctuation remains stable. By adjusting the quantization parameters at the block level, the overall image quality is improved.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0025] Figure 2 The figure is a flow chart showing a rate control method based on a region of interest according to an exemplary embodiment.

[0026] Figure 3 The figure is a flowchart of a method for determining a region of interest and a region of no interest according to an exemplary embodiment.

[0027] Figure 4A The following is a flow chart of a method for determining a code control reference value according to an exemplary embodiment. Figure 1 .

[0028] Figure 4B The following is a flow chart of a method for determining a code control reference value according to an exemplary embodiment. Figure 2 .

[0029] Figure 5A The figure is a flowchart of a method for determining a quantization parameter adjustment value according to an exemplary embodiment.

[0030] Figure 5B The figure is a flowchart of a method for determining a quantization parameter adjustment value within a region of interest according to an exemplary embodiment.

[0031] Figure 5C The figure is a schematic diagram showing a method for determining a quantization parameter adjustment value within a region of interest according to an exemplary embodiment.

[0032] Figure 5D The figure is a flowchart of a method for determining a quantization parameter adjustment value outside a region of interest according to an exemplary embodiment.

[0033] Figure 5E The figure is a schematic diagram of a method for determining a quantization parameter adjustment value outside a region of interest according to an exemplary embodiment.

[0034] Figure 6 The figure is a flow chart of a rate control method according to an exemplary embodiment.

[0035] Figure 7 The figure is a block diagram showing a rate control device based on a region of interest according to an exemplary embodiment.

[0036] Figure 8 It is a schematic diagram showing the structure of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0038] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0039] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0042] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0043] like Figure 1 As shown, the system architecture may include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used to provide a medium for a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0044] The server 101 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal device 103, the terminal device 104, or the terminal device 105. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device 103, the terminal device 104, or the terminal device 105.

[0045] Terminal device 103, terminal device 104 and terminal device 105 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, etc., but are not limited thereto.

[0046] It should be understood that Figure 1 The number of terminal devices 103, terminal devices 104, terminal devices 105, networks 102 and servers 101 are merely illustrative. Server 101 may be a physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, any number of terminal devices, networks and servers may be provided.

[0047] Below, each step of the method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0048] Figure 2 The figure is a flow chart showing a rate control method based on a region of interest according to an exemplary embodiment. Figure 2 The method provided in the embodiment can be executed by any electronic device, such as the above Figure 1 The terminal device in Figure 1 Servers in, or Figure 1 The terminal device and the server in the embodiment are jointly executed, but the present disclosure is not limited to this.

[0049] In step S210, an object of interest in a video frame to be encoded is determined.

[0050] In the embodiment of the present disclosure, in the real-time transmission scenario, the information of the region of interest / non-interest is combined with the encoding to guide the bit rate allocation in the encoding process. The bit rate allocation of the region of interest is improved to reduce the related encoding QP; the bit rate allocation of the region of non-interest is reduced to improve the related encoding QP.

[0051] In the embodiment of the present disclosure, a target of interest in a video frame to be encoded is determined. The video frame to be encoded is any video frame in the video transmission process. The target of interest is a target of interest in the video frame. For example, a face area. It should be noted that there are many ways to determine the target of interest in the video frame to be encoded. The present disclosure does not limit the specific method of use.

[0052] In an exemplary embodiment, a face object in a video frame to be encoded can be identified by a face recognition algorithm, and the face object can be used as a target of interest. Of course, other recognition algorithms can also be used to identify targets of interest in other scenarios. For example, in an e-commerce scenario, people pay more attention to commodities, and the identified commodities can be used as the target of interest.

[0053] In an exemplary embodiment, the target of interest may also be identified through other pre-processes, and the target of interest is determined directly based on the existing identification information. For example, the region of interest identification information corresponding to the video frame to be encoded is obtained; the region of interest identification information is used to identify the at least one region of interest; and the target of interest is determined based on the region of interest identification information.

[0054] This method can be applied to the video transcoding side. On the video transcoding side, the encoded video sent by the transmitter is received, and the encoded video is enhanced and transcoded, and then the transcoded video data is sent to the receiving end. Therefore, the video transmitter may have identified the target of interest in the video frame to be encoded, and written into SEI (Supplemental Enhancement Information) through the region of interest identification information. In this way, the transcoding side does not need to identify and determine the target of interest again, and can directly reuse the region of interest identification information.

[0055] In step S220, at least one region of interest and a region of no interest in the to-be-encoded video frame is determined based on the object of interest.

[0056] In the disclosed embodiment, based on the determined target of interest, the region of interest and the region of non-interest of the relevant target of interest are calibrated in the video frame to be encoded. The region of interest may be one or more. Moreover, the multiple regions of interest may be independent of each other or overlap with each other. The region other than the region of interest in the video frame to be encoded is the region of non-interest. By determining the region of interest and the region of non-interest in the video frame to be encoded, differentiated bit rate control can be performed.

[0057] In step S230, a bit rate estimation is performed on the to-be-encoded video frame to obtain a first estimated bit number.

[0058] In the embodiment of the present disclosure, the bit rate of the video frame to be encoded without bit rate control is estimated to obtain a first estimated number of bits. The first estimated number of bits is an estimated value of the number of bits consumed after encoding the video frame to be encoded. It should be noted that there can be a variety of bit rate estimation algorithms to estimate the number of bits consumed by the video frame to be encoded. The present disclosure does not limit the specific bit rate estimation algorithm used.

[0059] In an exemplary embodiment, the encoder may estimate the bit rate of the to-be-encoded video frame by using the following formula.

[0060] I=(coeff*satd+offset) / (qscale*count)

[0061] Among them, coeff is an adjustment factor, which is used to adjust the impact of the SATD value on the final estimated number of bits according to the characteristics of the video content. Satd (Sum of Absolute Transformed Differences) is used to evaluate the distortion value of the residual block after motion compensation in video encoding. Offset is an offset used to correct the error introduced due to model simplification. Qscale (Quantization Scale) is a quantization parameter in video encoding, which determines the degree of data compression. The smaller the Qscale value, the finer the quantization. Conversely, the larger the Qscale value, the coarser the quantization. The Qscale is related to the quantization parameter QP. Count is the number of macroblocks or the number of pixels. I is the first estimated bit number, which represents the estimated number of bits consumed by the video frame to be encoded.

[0062] In step S240, a first coding control reference value and a second coding control reference value corresponding to the region of interest and the region of no interest are determined respectively according to the first estimated number of bits.

[0063] In the embodiment of the present disclosure, in order to make the overall bit rate fluctuation of the video as stable as possible after the bit rate control is performed based on the region of interest. The present disclosure needs to balance the bit rate increased for the region of interest and the bit rate reduced for the region of no interest, so that the number of bits consumed by the video frame to be encoded after the bit rate control of the present disclosure is close to the first estimated number of bits. Based on this, according to the first estimated number of bits, the code control reference values ​​corresponding to the region of interest and the region of no interest are determined respectively. The region of interest corresponds to the first code control reference value, and the region of no interest corresponds to the second code control reference value. By adjusting and balancing the first code control reference value and the second code control reference value, so that after the rate control is performed on the region of interest and the region of no interest respectively based on the relevant code control reference value, the number of bits consumed by the video frame to be encoded is close to the first estimated number of bits.

[0064] It should be pointed out that there are many ways to select and determine the first coding control reference value and the second coding control reference value. As long as it can be ensured that after the bit rate control is performed based on the relevant coding control reference values, the overall bit rate fluctuation of the video remains stable, it should be regarded as within the protection scope of the present disclosure.

[0065] In step S250, a quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame is determined according to the first coding reference value and the second coding reference value.

[0066] In the disclosed embodiment, the first code control reference value and the second code control reference value are only reference values ​​for bit rate control, which are used to ensure that the overall bit rate fluctuation of the video after bit rate control remains stable. However, directly performing bit rate control on the entire region of interest or non-region of interest based on the reference value will make the bit rate control transition abrupt and fail to achieve a good video enhancement effect.

[0067] In the disclosed embodiment, according to the relevant code control reference value, the quantization parameter adjustment value corresponding to each coding block in the video frame to be encoded is determined respectively. By performing differentiated bit rate control on each coding block, the code control optimization result is more accurate and efficient, and the picture quality is improved.

[0068] In step S260, rate control is performed on each coding block based on the quantization parameter adjustment value.

[0069] The rate control method based on the region of interest provided by the embodiment of the present disclosure determines the region of interest and the region of non-interest in the video frame to be encoded; according to the rate estimation of the video frame to be encoded, the first code control reference value and the second code control reference value of the region of interest and the region of non-interest are respectively determined; based on the first code control reference value and the second code control reference value, the quantization parameter adjustment value of each coding block is respectively determined, and the rate control of each coding block is performed based on the quantization parameter adjustment value. This method makes the subjective perception of the video picture better by performing differentiated rate control on the region of interest and the region of non-interest. By respectively determining the first code control reference value and the second code control reference value based on the rate estimation of the video frame to be encoded, it is ensured that the overall video rate fluctuation remains stable. By adjusting the quantization parameters at the block level, the overall image quality is improved.

[0070] Figure 3 FIG. 1 is a flowchart of a method for determining an area of ​​interest and an area of ​​non-interest according to an exemplary embodiment. Figure 2 Based on the rate control method based on the region of interest, the aforementioned step S220 may include the following steps.

[0071] In step S310, a binary image is generated based on the object of interest; the binary image is used to calibrate the first region of interest and the first region of no interest.

[0072] In the disclosed embodiment, a binary image corresponding to the video frame to be encoded is generated according to the target of interest. The binary image can be a mask image marked by two numerical values ​​0 and 1. In this binary image, the pixels belonging to the region of interest are calibrated as 1 (or white); and the pixels of the non-interest region are calibrated as 0 (or black). Through this calibration, the corresponding first region of interest and the first non-interest region can be determined.

[0073] In step S320, the first region of interest is etched to obtain an etched first region of interest.

[0074] In the disclosed embodiment, since the detection of the region of interest sometimes detects a very small face of a passerby as the region of interest, this region is not actually the region of interest of the video. In order to remove those smaller and unconnected regions of interest, the aforementioned first region of interest is corroded to obtain the first region of interest after corrosion. The corrosion process is to shrink a preset number of pixels inward along the edge of the target area to form a corresponding reduced area. By corroding the first region of interest, those smaller and unconnected regions of interest can be removed, and the larger region of interest can be retained.

[0075] In step S330, the eroded first region of interest is expanded to determine the at least one region of interest and the region of no interest.

[0076] In the disclosed embodiment, the first region of interest after the above-mentioned corrosion process is further subjected to dilation processing. The dilation processing is the opposite process of the corrosion processing, which is to expand a preset number of pixels outward along the edge of the target area to form an enlarged corresponding area. It should be noted that if the region of interest is small and is removed by the above-mentioned corrosion processing, it will not be expanded and restored during the dilation processing, and only the region of interest retained after the corrosion processing will be dilated. Through corrosion processing and dilation processing, the smaller region of interest can be removed and the original region of interest can be restored. Based on the at least one region of interest determined, other regions can be determined as non-regions of interest.

[0077] In an exemplary embodiment, the amplitude of the dilation process is not less than the amplitude of the erosion process. When the amplitude of the dilation process is equal to the amplitude of the erosion process, after removing the smaller region of interest, the other regions of interest that have not been removed are restored. When the amplitude of the dilation process is greater than the amplitude of the erosion process, after removing the smaller region of interest, the other regions of interest that have not been removed are appropriately enlarged.

[0078] The bit rate control method based on the region of interest provided in the embodiment of the present disclosure can remove smaller and unconnected regions of interest by first corroding and then dilating the region of interest, and retain the regions of interest with larger areas, thereby optimizing the bit rate of more important regions of interest and improving the image quality.

[0079] In some embodiments of the present disclosure, whether to perform rate control on the video frame to be encoded may also be determined based on the area ratio of the region of interest in the entire video frame to be encoded. The following steps may be included.

[0080] The area of ​​the at least one region of interest is counted.

[0081] Calculate the area ratio of the at least one region of interest in the to-be-encoded video frame.

[0082] In response to the area ratio being within a preset area ratio range, the bit rate control is performed on the to-be-encoded video frame.

[0083] In the disclosed embodiment, the area of ​​each region of interest is first counted. The area statistics can be counted by the number of pixels in the region of interest. The area ratio of the region of interest in the video frame to be encoded is calculated based on the region area of ​​the region of interest. The area ratio can be calculated by the region area of ​​the non-interested region. The area ratio can also be calculated by the total number of pixels in the entire video frame to be encoded. When the area ratio is within a preset area ratio range, the rate control of the video frame to be encoded continues. When the area ratio is not within a preset area ratio range, the rate control process can be skipped, and the rate control of the video frame to be encoded is not performed. This is because if the proportion of the region of interest in the video picture is too large or too small, the visual effect benefit of the rate control is relatively small, and the related processing computing power overhead can be saved by skipping the rate control process.

[0084] In an exemplary embodiment, the area ratio range is set to A∈[0.05B, 2B], where A is the area of ​​the region of interest and B is the area of ​​the region of no interest.

[0085] The bit rate control method based on the region of interest provided in the embodiment of the present disclosure determines whether to perform bit rate control on the video frame according to the area ratio of the region of interest in the video frame. This ensures that the bit rate control is performed on the video frame with obvious image quality improvement effect, while skipping the video frame with no obvious image quality improvement effect, saving the related processing computing power overhead.

[0086] Figure 4A The following is a flow chart of a method for determining a code control reference value according to an exemplary embodiment. Figure 1 In the embodiment of the present disclosure, Figure 2 Based on the rate control method based on the region of interest, the aforementioned step S240 may include the following steps.

[0087] In step S410, a first code control setting value corresponding to the region of interest is set.

[0088] In the embodiment of the present disclosure, a first code control setting value is first set for the region of interest. The first code control setting value may be a manually set value, a value calculated by a relevant algorithm, or a value obtained by stepwise adjustment based on an initial value.

[0089] In an exemplary embodiment, in order to ensure that the overall bit rate of the video frame to be encoded is stable, a limit value G can be pre-set for the first code control setting value, where G is a negative number (eg, -6). In this way, the first code control setting value can be ensured to be within a certain range.

[0090] In step S420, based on the first code control setting value, a code rate estimation is performed on the at least one region of interest to obtain a second estimated number of bits corresponding to the region of interest.

[0091] In the embodiment of the present disclosure, according to the first code control setting value, the code rate of the at least one region of interest is estimated to obtain a second estimated number of bits corresponding to the region of interest. The second estimated number of bits is the number of bits that the region of interest needs to consume after the code rate control is performed on the region of interest based on the first code control setting value. The code rate estimation is similar to the code rate estimation algorithm in the aforementioned step S230, and will not be repeated here.

[0092] In an exemplary embodiment, the encoder may estimate the code rate of at least one region of interest by using the following formula.

[0093] K=(coeff*E+offset) / ((qscale+J)*count)

[0094] Wherein, J is a first code control setting value, E is a first complexity corresponding to the region of interest, and K is a second estimated number of bits.

[0095] In an exemplary embodiment, a first complexity E corresponding to the region of interest can be obtained by performing complexity statistics on at least one region of interest. The first complexity E can be an average value of the complexity of each region of interest, or an accumulated value of the complexity of each region of interest. It should be noted that there are many statistical algorithms for image complexity, and the present disclosure does not limit the specific algorithms used.

[0096] Based on the first complexity E obtained by statistics and the first code control setting value J set above, the second estimated number of bits K can be calculated based on the above formula.

[0097] In step S430, a third estimated number of bits corresponding to the non-interest region is calculated based on the first estimated number of bits and the second estimated number of bits.

[0098] As mentioned above, we hope to ensure that the overall bit rate fluctuation of the video after bit rate control remains stable. Therefore, it is hoped that the sum of the number of bits in the region of interest and the number of bits in the non-interest region in the video frame to be encoded after bit rate control can be close to the first estimated number of bits without bit rate control. Therefore, in the embodiment of the present disclosure, based on the first estimated number of bits I and the second estimated number of bits K, the third estimated number of bits IK corresponding to the non-interest region is calculated. That is, in addition to ensuring the number of bits after the bit rate enhancement in the region of interest, the number of bits left for the non-interest region can also be guaranteed.

[0099] In step S440, a second code control setting value corresponding to the non-interest region is calculated according to the third estimated bit number.

[0100] In the embodiment of the present disclosure, the second code control setting value corresponding to the non-interested region is calculated according to the third estimated bit number. The calculation algorithm is the inverse operation of the aforementioned code rate estimation, so the code rate estimation algorithm in the aforementioned step S230 can be referred to.

[0101] In an exemplary embodiment, the encoder may calculate the second code control setting value by the following formula.

[0102] L=((coeff*F+offset) / ((IK)*count))-qscale

[0103] Wherein, IK is the third estimated bit number, F is the second complexity corresponding to the non-interest region, and L is the second code control setting value.

[0104] In an exemplary embodiment, complexity statistics of the non-interest region may be performed to obtain a second complexity F corresponding to the non-interest region. The second complexity F may be an average value of the complexity of the non-interest region or an accumulated value of the complexity of the non-interest region.

[0105] Based on the statistically obtained second complexity F and the aforementioned determined third estimated bit number IK, the second code control setting value L can be calculated based on the above formula.

[0106] In the disclosed embodiment, the first code control setting value corresponding to the region of interest is first determined, and then the second estimated number of bits of the region of interest is calculated. In order to ensure that the sum of the number of bits of the region of interest and the number of bits of the non-interest region after rate control can be close to the first estimated number of bits. According to the first estimated number of bits and the second estimated number of bits, the third estimated number of bits corresponding to the non-interest region is calculated. Then, according to the third estimated number of bits, based on the inverse operation of the rate estimation, the second code control setting value corresponding to the non-interest region is calculated.

[0107] In step S450 , it is determined that the first coding control setting value is the first coding control reference value, and the second coding control setting value is the second coding control reference value.

[0108] In the embodiment of the present disclosure, the first code control setting value is determined to be the first code control reference value, and the second code control setting value is determined to be the second code control reference value through the above steps. In this way, the bit rate of the video frame to be encoded is controlled based on the first code control reference value and the second code control reference value, so that the overall bit rate fluctuation of the video can be kept stable.

[0109] Figure 4B The following is a flow chart of a method for determining a code control reference value according to an exemplary embodiment. Figure 2 .

[0110] In the disclosed embodiment, Figure 4B In the method for determining the code control reference value shown in FIG. 1 , steps S410, S420, S430, and S440 are respectively Figure 4A Steps S410, S420, S430, and S440 in the method for determining the code control reference value are corresponding to each other and will not be repeated here.

[0111] In the embodiment of the present disclosure, Figure 4A Based on the method for determining the code control reference value shown, the aforementioned step S450 may include the following steps.

[0112] In step S451, it is determined whether the second code control setting value is greater than a non-interest region adjustment threshold.

[0113] Figure 4A Although the embodiment shown can ensure that the sum of the number of bits in the region of interest and the number of bits in the region of non-interest after bit rate control is close to the first estimated number of bits, thereby ensuring that the overall bit rate fluctuation of the video remains stable, if the first code control setting value is set unreasonably, resulting in an excessive increase in the bit rate of the region of interest, which cannot be compensated by reducing the bit rate of the region of non-interest, the overall bit rate fluctuation will be too large, and therefore it is necessary to reduce the bit rate allocation of the region of interest.

[0114] In the embodiment of the present disclosure, in order to ensure that the bit rate of the non-interest region is not reduced too much, a non-interest region adjustment threshold H is set for the non-interest region. That is, the bit rate adjustment for the non-interest region shall not exceed the threshold H.

[0115] Based on this, the second code control setting value calculated in the aforementioned step S440 is compared with the non-interest region adjustment threshold H to determine whether the second code control setting value is greater than the non-interest region adjustment threshold H.

[0116] In step S452, in response to the second code control setting value being greater than the non-interest region adjustment threshold, the first code control setting value is adjusted, and the second code control setting value is recalculated.

[0117] In the embodiment of the present disclosure, when the second code control setting value L is greater than the non-interest region adjustment threshold H, it means that the code rate of the non-interest region is too much reduced, and the first code control setting value J set in the aforementioned step S410 needs to be adjusted. Figure 4A The calculation process shown recalculates the second code control setting value L so that the second code control setting value L is not greater than the non-interest region adjustment threshold H.

[0118] In an exemplary embodiment, the first code control setting value J can be adjusted in a step-by-step manner. That is, the first code control setting value J is first set to an initial value. The initial value may be the aforementioned limit value G. When the second code control setting value L calculated based on the initial value is greater than the non-interest region adjustment threshold value H, the first code control setting value J is adjusted in a step-by-step manner. The first code control setting value J is adjusted in a step-by-step manner in this way until the second code control setting value L is no greater than the non-interest region adjustment threshold value H.

[0119] In step S453, in response to the second coding setting value being not greater than the non-interest region adjustment threshold, the first coding setting value is determined to be the first coding reference value, and the second coding setting value is determined to be the second coding reference value.

[0120] In the embodiment of the present disclosure, when the second code control setting value L is not greater than the non-interest area adjustment threshold H, it means that the code rate reduction in the non-interest area is reasonable, and it can be determined that the current first code control setting value is the first code control reference value, and the current second code control setting value is the second code control reference value.

[0121] The rate control method based on the region of interest provided by the embodiment of the present disclosure obtains a first estimated number of bits by estimating the rate of the video frame to be encoded, and determines a first code control reference value and a second code control reference value according to the first estimated number of bits, so that the number of bits consumed by the video frame to be encoded after rate control is close to the first estimated number of bits, thereby ensuring that the overall video rate fluctuation remains stable.

[0122] Figure 5A is a flow chart of a method for determining a quantization parameter adjustment value according to an exemplary embodiment. Figure 2 Based on the rate control method based on the region of interest, the aforementioned step S250 may include the following steps.

[0123] In step S510, a first coding block in the to-be-coded video frame is determined; the first coding block belongs to each of the coding blocks.

[0124] As mentioned above, the present disclosure determines the quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame according to the relevant coding reference value. By performing differentiated bit rate control on each coding block, the coding optimization result is more accurate and efficient, and the picture quality is improved.

[0125] In the disclosed embodiment, the quantization parameter adjustment value corresponding to each coding block is determined one by one with each coding block as a unit. First, the first coding block in the to-be-coded video frame is determined. The first coding block is any coding block in the to-be-coded video frame.

[0126] In step S520, a first quantization parameter adjustment value of the first coding block relative to each of the regions of interest is determined according to the first coding reference value and the second coding reference value.

[0127] As mentioned above, there may be multiple regions of interest in the video frame to be encoded, and the first quantization parameter adjustment value of the first coding block relative to each region of interest is determined according to the relative position relationship between the first coding block and each region of interest. The first quantization parameter adjustment value is a rate control adjustment value determined based on the relative position relationship between the first coding block and the target region of interest.

[0128] As mentioned above, in order to ensure that the overall video bit rate fluctuation remains stable, it is necessary to refer to the first code control reference value and the second code control reference value determined above in the process of determining the first quantization parameter adjustment value, so that the determined first quantization parameter adjustment value is within a reasonable range and will not cause excessive fluctuations in the overall bit rate.

[0129] In step S530, a minimum value among the first quantization parameter adjustment values ​​is determined as the quantization parameter adjustment value corresponding to the first coding block.

[0130] As mentioned above, there may be multiple regions of interest in the video frame to be encoded. Therefore, according to step S520, several first quantization parameter adjustment values ​​of the first coding block relative to each region of interest can be obtained. Since a negative value of the quantization parameter adjustment value means increasing the bit rate of the coding block, and a positive value means reducing the bit rate of the coding block, that is, the smaller the quantization parameter adjustment value, the higher the bit rate of the coding block. Based on this, the minimum value is selected from the several first quantization parameter adjustment values ​​to determine as the quantization parameter adjustment value corresponding to the first coding block, so as to perform bit rate control on the first coding block based on the quantization parameter adjustment value.

[0131] Figure 5BThe figure is a flowchart of a method for determining a quantization parameter adjustment value within a region of interest according to an exemplary embodiment. Figure 5C FIG. 1 is a schematic diagram showing a method for determining a quantization parameter adjustment value within a region of interest according to an exemplary embodiment. Figure 5A Based on the method for determining the quantization parameter adjustment value shown, the aforementioned step S520 may include the following steps.

[0132] In step S521, in response to the first coding block being located in a target region of interest, a rectangular region corresponding to the target region of interest is determined.

[0133] As mentioned above, according to the relative positional relationship between the first coding block and each region of interest, the first quantization parameter adjustment value of the first coding block relative to each region of interest is determined. Since the first coding block may be located within the range of the region of interest, it may also be located outside the range of the region of interest. Therefore, it is necessary to perform differentiated processing on whether the first coding block is located within or outside the region of interest.

[0134] In the embodiment of the present disclosure, for the case where the first coding block is located in the target region of interest, a rectangular area corresponding to the target region of interest is determined. The target region of interest is any one of the multiple regions of interest of the video frame to be encoded. Since the region of interest is usually an irregular shape, the rectangular area corresponding to the target region of interest is first determined for calculation convenience.

[0135] In an exemplary embodiment, the rectangular region is a circumscribed rectangular region based on the target region of interest.

[0136] In step S522, the center point of the rectangular area and the first distance from the vertex to the center point are determined.

[0137] In the embodiment of the present disclosure, based on the rectangular area, the center point N of the rectangular area and the first distance R0 between the vertices of the rectangular area and the center point N are determined.

[0138] In step S523, a second distance from the first coding block to the center point is determined.

[0139] In the embodiment of the present disclosure, based on the rectangular area, a second distance R1 from the first coding block to the center point N of the rectangular area is determined. The second distance R1 may be a distance from the center point of the first coding block to the center point N of the rectangular area.

[0140] In step S524, the first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance, the second distance and a first code control reference value.

[0141] In the embodiment of the present disclosure, the first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance R0, the second distance R1 and the first code control reference value J determined above. The present disclosure only points out that the first quantization parameter adjustment value can be determined with reference to the first distance R0, the second distance R1 and the first code control reference value J. The specific calculation algorithm can be adjusted according to actual control needs, and the present disclosure does not limit the specific adjustment algorithm.

[0142] In an exemplary embodiment, the first quantization parameter adjustment value may be calculated by the following formula.

[0143] roi_qpoffset=(R0-R1) / R1*J

[0144] Wherein, roi_qpoffset is the first quantization parameter adjustment value in the region of interest, R0 is the first distance, R1 is the second distance, and J is the first code control reference value determined in the aforementioned step S240.

[0145] Figure 5D The figure is a flowchart of a method for determining a quantization parameter adjustment value outside a region of interest according to an exemplary embodiment. Figure 5E FIG. 1 is a schematic diagram of a method for determining a quantization parameter adjustment value outside a region of interest according to an exemplary embodiment. Figure 5A Based on the method for determining the quantization parameter adjustment value shown, the aforementioned step S520 may include the following steps.

[0146] In step S525, in response to the first coding block not being located in the target region of interest, a rectangular area corresponding to the target region of interest is determined.

[0147] In the embodiment of the present disclosure, for the case where the first coding block is outside the target region of interest, a rectangular area corresponding to the target region of interest is determined. The target region of interest is any one of the multiple regions of interest of the video frame to be encoded. Since the region of interest is usually an irregular shape, the rectangular area corresponding to the target region of interest is first determined for calculation convenience.

[0148] In an exemplary embodiment, the rectangular region is a circumscribed rectangular region based on the target region of interest.

[0149] In step S526, the center point of the rectangular area and the first distance from the vertex to the center point are determined.

[0150] In the embodiment of the present disclosure, based on the rectangular area, the center point N of the rectangular area and the first distance R0 between the vertices of the rectangular area and the center point N are determined.

[0151] In step S527, a second distance from the first coding block to the center point is determined.

[0152] In the embodiment of the present disclosure, based on the rectangular area, a second distance R1 from the first coding block to the center point N of the rectangular area is determined. The second distance R1 may be a distance from the center point of the first coding block to the center point N of the rectangular area.

[0153] In step S528, it is determined whether the first coding block is located in a quadrant based on the center point.

[0154] Since the non-interested areas are directly used as mentioned above Figure 5B The algorithm for the region of interest shown in the figure may result in some areas not being covered, or the area of ​​the area is too large, exceeding the area of ​​the entire image by too much, resulting in inaccurate estimation and other problems. Therefore, the four-quadrant method is used to adjust the non-interested region. That is, the first quantization parameter adjustment value is determined based on the quadrant direction of the target region of interest where the first coding block is located.

[0155] In the disclosed embodiment, based on the center point N of the target region of interest, the to-be-encoded video frame is divided into four quadrants, namely, the upper right quadrant, the upper left quadrant, the lower left quadrant, and the lower right quadrant. The corresponding quadrant where the first coding block is located is determined according to the four divided quadrants.

[0156] In step S529, a third distance from the vertex of the to-be-encoded video frame in the quadrant to the center point is determined.

[0157] In the embodiment of the present disclosure, based on the four quadrants divided above, it can be known that each quadrant corresponds to a vertex of the video frame to be encoded. According to the corresponding quadrant where the first coding block is located, the distance between the vertex of the video frame to be encoded in the corresponding quadrant and the center point N is determined as the third distance R2.

[0158] In step S5210, the first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance, the second distance, the third distance and a second coding reference value.

[0159] In the embodiment of the present disclosure, the first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance R0, the second distance R1, the third distance R2 and the second code control reference value L determined above. The present disclosure only points out that the first quantization parameter adjustment value can be determined with reference to the first distance R0, the second distance R1, the third distance R2 and the second code control reference value L. The specific calculation algorithm can be adjusted according to the actual control needs, and the present disclosure does not limit the specific adjustment algorithm.

[0160] In an exemplary embodiment, the first quantization parameter adjustment value may be calculated by the following formula.

[0161] noroi_qpoffset=MAX(0,R1-R0) / (R2-R0)*L

[0162] Wherein, noroi_qpoffset is the first quantization parameter adjustment value outside the region of interest, R0 is the first distance, R1 is the second distance, and J is the first code control reference value determined in the aforementioned step S240.

[0163] The method for rate control based on regions of interest provided by the embodiment of the present disclosure calculates each coding block in the encoded video frame and each region of interest respectively, determines the first quantization parameter adjustment value of the first coding block relative to each region of interest according to the relative position relationship between the first coding block and each region of interest, and then determines the quantization parameter adjustment value of the first coding block. Through block-level quantization parameter adjustment, different coding blocks can be subjected to differentiated rate control, thereby improving the overall image quality.

[0164] Figure 6 is a flow chart of a rate control method according to an exemplary embodiment. Figure 2 Based on the rate control method based on the region of interest, the aforementioned step S260 may include the following steps.

[0165] In step S610, based on a block-level code control algorithm, a second quantization parameter adjustment value corresponding to a second coding block is obtained; the second coding block belongs to each of the coding blocks.

[0166] In addition to the rate control method shown above, other rate control algorithms can be used to control the rate of the video frame to be encoded. In order to be compatible with different rate control algorithms, the quantization parameter adjustment value calculated in step S250 needs to be merged with the quantization parameter adjustment value obtained by other rate control algorithms.

[0167] In the disclosed embodiment, a second quantization parameter adjustment value corresponding to the second coding block is obtained based on a block-level code control algorithm. The second coding block may be any coding block in the video frame to be encoded. The block-level code control algorithm refers to a technique for performing differential processing on different regions of an image (usually macroblocks or smaller blocks) during video encoding to optimize overall coding efficiency and visual quality. This type of algorithm achieves differentiated bit rate control for different regions by adjusting the quantization parameters (QP) of different blocks. For example, the AQ (Adaptive Quantization) algorithm and the CUTREE (Coding UnitTree Rate Estimation and Encoding) algorithm. The second quantization parameter adjustment value is a quantization parameter adjustment value calculated based on the block-level code control algorithm.

[0168] In step S620, it is determined whether the rate control direction of the second quantization parameter adjustment value is consistent with the rate control direction of the quantization parameter adjustment value of the second coding block.

[0169] In the disclosed embodiment, it is determined whether the second quantization parameter adjustment value corresponding to the same coding block (such as the second coding block) is consistent with the rate control direction of the quantization parameter adjustment value calculated in the aforementioned step S250. The so-called rate control direction refers to increasing the bit rate of the coding block or reducing the bit rate of the coding block. This is reflected in the quantization parameter adjustment value, that is, the two quantization parameter adjustment values ​​are both positive or both negative.

[0170] In step S630, in response to the rate control directions being consistent, the second quantization parameter adjustment value is superimposed on the quantization parameter adjustment value of the second coding block, and rate control is performed on the second coding block.

[0171] In the disclosed embodiment, in response to the rate control directions being consistent, the two quantization parameter adjustment values ​​are superimposed, and rate control is performed on the second coding block based on the superimposed quantization parameter adjustment value.

[0172] In step S640, in response to the rate control directions being inconsistent, the second quantization parameter adjustment value is adjusted based on an adjustment coefficient.

[0173] In the disclosed embodiment, in response to the inconsistent bit rate control directions, that is, one quantization parameter adjustment value is a positive number and the other quantization parameter adjustment value is a negative number, the second quantization parameter adjustment value is adjusted based on a preset adjustment coefficient.

[0174] In an exemplary embodiment, the second quantization parameter adjustment value is multiplied by the adjustment coefficient to obtain the adjusted second quantization parameter adjustment value. For example, the adjustment coefficient is 0.5.

[0175] In step S650, the adjusted second quantization parameter adjustment value is superimposed on the quantization parameter adjustment value of the second coding block, and bit rate control is performed on the second coding block.

[0176] In the disclosed embodiment, the adjusted second quantization parameter adjustment value is superimposed on the corresponding quantization parameter adjustment value, and the bit rate control is performed on the second coding block based on the superimposed quantization parameter adjustment value.

[0177] The rate control method based on the region of interest provided by the embodiment of the present disclosure considers that in the rate control process, in addition to the rate control method shown above, other rate control algorithms can also be used to control the rate of the video frame to be encoded. In order to be compatible with different rate control algorithms, a fusion solution for obtaining the quantization parameter adjustment value with other rate control algorithms is designed to achieve compatibility with other rate control algorithms.

[0178] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0179] Figure 7 is a block diagram of a rate control device based on a region of interest according to an exemplary embodiment. Figure 7 The rate control device 700 based on the region of interest may include: a target determination module 710, a region determination module 720, a rate estimation module 730, a code control reference value determination module 740, a quantization parameter adjustment value determination module 750 and a rate control module 760.

[0180] The target determination module 710 is configured to determine the target of interest in the video frame to be encoded.

[0181] The region determination module 720 is configured to determine at least one region of interest and a region of no interest in the to-be-encoded video frame based on the target of interest.

[0182] The bit rate estimation module 730 is configured to perform bit rate estimation on the to-be-encoded video frame to obtain a first estimated number of bits.

[0183] The coding control reference value determination module 740 is configured to determine a first coding control reference value and a second coding control reference value corresponding to the region of interest and the region of no interest, respectively, according to the first estimated number of bits.

[0184] The quantization parameter adjustment value determination module 750 is configured to determine the quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame according to the first coding reference value and the second coding reference value.

[0185] The bit rate control module 760 is configured to perform bit rate control on each coding block based on the quantization parameter adjustment value.

[0186] In some exemplary embodiments of the present disclosure, the region determination module 720 is further configured to generate a binary image based on the target of interest; the binary image is used to calibrate the first region of interest and the first region of no interest; the first region of interest is corroded to obtain the corroded first region of interest; the corroded first region of interest is expanded to determine at least one region of interest and region of no interest.

[0187] In some exemplary embodiments of the present disclosure, the region determination module 720 is further configured such that the amplitude of the dilation process is not less than the amplitude of the erosion process.

[0188] In some exemplary embodiments of the present disclosure, the region determination module 720 is further configured to count the area of ​​the at least one region of interest; calculate the area ratio of the at least one region of interest in the video frame to be encoded; and perform the bit rate control on the video frame to be encoded in response to the area ratio being within a preset area ratio range.

[0189] In some exemplary embodiments of the present disclosure, the code control reference value determination module 740 is further configured to set a first code control setting value corresponding to the region of interest; based on the first code control setting value, perform a code rate estimate on the at least one region of interest to obtain a second estimated number of bits corresponding to the region of interest; based on the first estimated number of bits and the second estimated number of bits, calculate a third estimated number of bits corresponding to the region of no interest; based on the third estimated number of bits, calculate a second code control setting value corresponding to the region of no interest; determine that the first code control setting value is the first code control reference value, and the second code control setting value is the second code control reference value.

[0190] In some exemplary embodiments of the present disclosure, the code control reference value determination module 740 is also configured to determine whether the second code control setting value is greater than a non-interest area adjustment threshold; in response to the second code control setting value being greater than the non-interest area adjustment threshold, adjust the first code control setting value and recalculate the second code control setting value; in response to the second code control setting value being not greater than the non-interest area adjustment threshold, determine that the first code control setting value is the first code control reference value and the second code control setting value is the second code control reference value.

[0191] In some exemplary embodiments of the present disclosure, the code control reference value determination module 740 is also configured to obtain the first complexity and the second complexity corresponding to the at least one region of interest and the non-region of interest, respectively; calculate the second estimated number of bits based on the first complexity and the first code control setting value; calculate the second code control setting value based on the second complexity and the third estimated number of bits.

[0192] In some exemplary embodiments of the present disclosure, the quantization parameter adjustment value determination module 750 is also configured to determine a first coding block in the video frame to be encoded; the first coding block belongs to each of the coding blocks; based on the first coding reference value and the second coding reference value, determine a first quantization parameter adjustment value of the first coding block relative to each of the regions of interest; and determine that the minimum value of each of the first quantization parameter adjustment values ​​is the quantization parameter adjustment value corresponding to the first coding block.

[0193] In some exemplary embodiments of the present disclosure, the quantization parameter adjustment value determination module 750 is also configured to determine, in response to the first coding block being located in the target region of interest, a rectangular area corresponding to the target region of interest; determine a center point of the rectangular area and a first distance from a vertex to the center point; determine a second distance from the first coding block to the center point; and determine the first quantization parameter adjustment value of the first coding block relative to the target region of interest based on the first distance, the second distance and the first code control reference value.

[0194] In some exemplary embodiments of the present disclosure, the quantization parameter adjustment value determination module 750 is also configured to determine, in response to the first coding block not being located in the target region of interest, a rectangular area corresponding to the target region of interest; determine a center point of the rectangular area, and a first distance from a vertex to the center point; determine a second distance from the first coding block to the center point; determine that the first coding block is located in a quadrant based on the center point; determine a third distance from a vertex of the to-be-encoded video frame in the quadrant to the center point; and determine the first quantization parameter adjustment value of the first coding block relative to the target region of interest based on the first distance, the second distance, the third distance and the second code control reference value.

[0195] In some exemplary embodiments of the present disclosure, the rate control module 750 is further configured to obtain a second quantization parameter adjustment value corresponding to a second coding block based on a block-level code control algorithm; the second coding block belongs to each of the coding blocks; determine whether the rate control direction of the second quantization parameter adjustment value is consistent with the quantization parameter adjustment value of the second coding block; in response to the rate control direction being consistent, superimpose the second quantization parameter adjustment value with the quantization parameter adjustment value of the second coding block, and perform rate control on the second coding block; in response to the rate control direction being inconsistent, adjust the second quantization parameter adjustment value based on an adjustment coefficient; superimpose the adjusted second quantization parameter adjustment value with the quantization parameter adjustment value of the second coding block, and perform rate control on the second coding block.

[0196] In some exemplary embodiments of the present disclosure, the target determination module 710 is further configured to obtain region of interest identification information corresponding to the video frame to be encoded; the region of interest identification information is used to identify at least one region of interest; and based on the region of interest identification information, determine the target of interest.

[0197] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0198] Refer to the following Figure 8 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0199] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0200] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 The steps shown.

[0201] For example, electronic devices can achieve Figure 2 The steps shown.

[0202] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 821 and / or a cache memory unit 822 , and may further include a read-only memory unit (ROM) 823 .

[0203] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, such program modules 825 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0204] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0205] The electronic device 800 may also communicate with one or more external devices 870 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0206] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0207] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0208] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method in the above embodiment is implemented.

[0209] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0210] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A rate control method based on a region of interest, characterized in that: include: Determining an object of interest in a video frame to be encoded; Based on the object of interest, determining at least one region of interest and non-region of interest in the to-be-encoded video frame; Estimating a bit rate of the to-be-encoded video frame to obtain a first estimated number of bits; Determine, according to the first estimated number of bits, a first coding reference value and a second coding reference value corresponding to the region of interest and the region of non-interest, respectively; Determine, according to the first coding control reference value and the second coding control reference value, a quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame; The coding blocks are rate controlled based on the quantization parameter adjustment value.

2. The method according to claim 1, characterized in that The determining, based on the target of interest, at least one region of interest and a region of non-interest in the to-be-encoded video frame comprises: Based on the object of interest, a binary image is generated; the binary image is used to calibrate the first region of interest and the first region of no interest; Performing an etching process on the first region of interest to obtain an etched first region of interest; Performing expansion processing on the first region of interest after the erosion to determine the at least one region of interest and a region of no interest; Wherein, the amplitude of the expansion treatment is not less than the amplitude of the corrosion treatment.

3. The method according to claim 1, characterized in that The method further comprises: Counting the area of ​​the at least one region of interest; Calculating the area ratio of the at least one region of interest in the video frame to be encoded; In response to the area ratio being within a preset area ratio range, the bit rate control is performed on the to-be-encoded video frame.

4. The method according to claim 1, characterized in that The determining, according to the first estimated number of bits, a first coding reference value and a second coding reference value corresponding to the region of interest and the region of non-interest, respectively, comprises: Setting a first code control setting value corresponding to the region of interest; Based on the first code control setting value, perform code rate estimation on the at least one region of interest to obtain a second estimated number of bits corresponding to the region of interest; Calculating a third estimated number of bits corresponding to the non-interest region based on the first estimated number of bits and the second estimated number of bits; Calculating a second code control setting value corresponding to the non-interested region according to the third estimated bit number; The first code control setting value is determined to be the first code control reference value, and the second code control setting value is determined to be the second code control reference value.

5. The method according to claim 4, characterized in that The step of determining the first code control setting value as the first code control reference value and the second code control setting value as the second code control reference value comprises: Determining whether the second code control setting value is greater than a non-interest region adjustment threshold; In response to the second code control setting value being greater than the non-interest region adjustment threshold, adjusting the first code control setting value and recalculating the second code control setting value; In response to the second coding control setting value being not greater than the non-interest region adjustment threshold, the first coding control setting value is determined to be the first coding control reference value, and the second coding control setting value is determined to be the second coding control reference value.

6. The method according to claim 4, characterized in that The method further comprises: Acquire a first complexity and a second complexity respectively corresponding to the at least one region of interest and the region of non-interest; Calculating the second estimated number of bits based on the first complexity and a first code control setting value; The second code control setting value is calculated based on the second complexity and a third estimated number of bits.

7. The method according to claim 1, characterized in that The determining, according to the first coding reference value and the second coding reference value, a quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame comprises: Determine a first coding block in the to-be-coded video frame; the first coding block belongs to each of the coding blocks; Determine, according to the first coding reference value and the second coding reference value, a first quantization parameter adjustment value of the first coding block relative to each of the regions of interest; A minimum value among the first quantization parameter adjustment values ​​is determined as the quantization parameter adjustment value corresponding to the first coding block.

8. The method according to claim 7, characterized in that The determining, according to the first coding reference value and the second coding reference value, a first quantization parameter adjustment value of the first coding block relative to each of the regions of interest includes: In response to the first coding block being located in a target region of interest, determining a rectangular region corresponding to the target region of interest; Determine the center point of the rectangular area and a first distance from a vertex to the center point; Determining a second distance from the first coding block to the center point; The first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance, the second distance and a first code control reference value.

9. The method according to claim 7, characterized in that: The determining, according to the first coding reference value and the second coding reference value, a first quantization parameter adjustment value of the first coding block relative to each of the regions of interest includes: In response to the first coding block not being located in a target region of interest, determining a rectangular region corresponding to the target region of interest; Determine the center point of the rectangular area and a first distance from the vertex to the center point; Determining a second distance from the first coding block to the center point; Determining that the first coding block is located in a quadrant based on the center point; Determine a third distance from a vertex of the to-be-encoded video frame in the quadrant to the center point; The first quantization parameter adjustment value of the first coding block relative to the target region of interest is determined according to the first distance, the second distance, the third distance and a second code control reference value.

10. The method according to claim 1, characterized in that The performing bit rate control on each coding block based on the quantization parameter adjustment value includes: Based on a block-level code control algorithm, obtaining a second quantization parameter adjustment value corresponding to a second coding block; the second coding block belongs to each of the coding blocks; Determining whether the second quantization parameter adjustment value and the rate control direction of the quantization parameter adjustment value of the second coding block are consistent; In response to the rate control directions being consistent, superimposing the second quantization parameter adjustment value with the quantization parameter adjustment value of the second coding block, and performing rate control on the second coding block; In response to the rate control directions being inconsistent, adjusting the second quantization parameter adjustment value based on an adjustment coefficient; The adjusted second quantization parameter adjustment value is superimposed on the quantization parameter adjustment value of the second coding block, and bit rate control is performed on the second coding block.

11. The method according to claim 1, characterized in that: The determining of the target of interest in the to-be-encoded video frame comprises: Acquire region of interest identification information corresponding to the to-be-encoded video frame; the region of interest identification information is used to identify the at least one region of interest; The object of interest is determined based on the region of interest identification information.

12. A rate control device based on a region of interest, characterized in that: include: A target determination module, configured to determine a target of interest in a video frame to be encoded; A region determination module, configured to determine at least one region of interest and a region of non-interest in the to-be-encoded video frame based on the target of interest; A bit rate estimation module is configured to perform bit rate estimation on the to-be-encoded video frame to obtain a first estimated bit number; A coding control reference value determination module is configured to determine a first coding control reference value and a second coding control reference value corresponding to the region of interest and the region of non-interest, respectively, according to the first estimated number of bits; a quantization parameter adjustment value determination module, configured to determine a quantization parameter adjustment value corresponding to each coding block in the to-be-encoded video frame according to the first coding control reference value and the second coding control reference value; The bit rate control module is configured to perform bit rate control on each coding block based on the quantization parameter adjustment value.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the rate control method based on the region of interest as described in any one of claims 1 to 11.

14. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the rate control method based on a region of interest as claimed in any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the rate control method based on the region of interest as claimed in any one of claims 1 to 11 is implemented.

Images