Target detection enhanced video code rate control method

Abstract

The invention relates to a target detection enhanced video code rate control method, and belongs to the technical field of video compression. The method comprises the following steps: preprocessing each frame of image of an input video, dividing a plurality of coding tree units in each frame of image, and calculating gradient features of the image and the coding tree units; the video is coded through different quantization parameters, and the code rate of the coded video and the distortion degree of the video are calculated; fitting a cubic logarithmic model to establish a relationship between the distortion degree and the code rate of the video by combining the gradient characteristics of each frame and the distortion degree and the code rate of the video; marking the ROI region of each image frame of the input video through a target detection network, and dividing the compensation level of the coding tree unit in combination with the ROI region and the gradient feature of the coding tree unit; and in combination with the compensation level of each coding tree unit, through a code rate allocation mechanism, allocating a coding code rate to each coding tree unit in each image frame of the input video. According to the invention, the video compression process can be optimized, and the overall performance of the compressed video is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video compression, and relates to a video code rate control method enhanced by target detection. BACKGROUND

[0002] With the deep integration of digital video technology and artificial intelligence (AI), video has evolved from a traditional "human eye consumption medium" to a "core data carrier driven by human eye watching and machine analysis", which needs to meet the dual demands of visual quality and target detection reliability. This technical requirement is derived from the expansion of application scenarios, the evolution of technical systems, and the inherent limitations of traditional solutions.

[0003] In the evolution process of video coding technology, from H.264 / AVC, H.265 / HEVC to the new generation VVC (Versatile Video Coding), the core goal has always been to "improve human eye perception quality under limited bandwidth" - through intra prediction, inter motion compensation, entropy coding and other technologies, to minimize the code rate consumption while reducing block effects, blurriness and other distortions that affect human eye viewing experience. At the same time, the maturity of target detection technology (such as the iteration of YOLO, Faster R-CNN and other algorithms) has promoted the transition of video from "passive watching" to "active analysis", that is, machines locate and identify targets (such as vehicles, pedestrians, obstacles) in video frames to support decision-making in key scenarios such as autonomous driving, intelligent monitoring, and industrial quality inspection. Video gradually assumes the dual roles of "human eye watching carrier" and "machine analysis data source", and the technical requirements of the two form a symbiotic relationship: on the one hand, efficient coding is needed to ensure acceptable visual quality for the human eye (such as no obvious lag or blur in live streaming), and on the other hand, the coded video needs to provide enough feature information (such as the edges, textures, and colors of the target) for target detection algorithms.

[0004] In summary, with the large-scale application of scenarios such as autonomous driving, intelligent monitoring, and remote industrial quality inspection, video needs to meet both the visual quality of human eye watching and the reliability of machine target detection. However, the traditional coding scheme that "pays more attention to the human eye than to the machine" and the technical system that "decouples detection from coding" have become a key bottleneck restricting the development of such scenarios. Therefore, developing video coding and detection collaborative technology that takes into account the dual demands of humans and machines has become a core research direction in the current industry.

[0005] The existing code rate control technology does not consider the special requirements of machine vision on video features due to the "people-centered" technical positioning, the core mechanism mismatched with the requirements of machines, and the lack of machine dimension evaluation system. In the scene where video needs to serve both "human eye watching" and "machine analysis", this technical limitation has become a key bottleneck restricting the application landing, and a new code rate control technology that takes into account the dual requirements of man and machine is urgently needed. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a target detection enhanced video code rate control method, which optimizes the video compression process by combining code rate control, target detection and code rate compensation, and improves the overall performance of the compressed video.

[0007] To achieve the above purpose, the present application provides the following technical solutions: A target detection enhanced video code rate control method, the method comprising: Preprocessing each frame of image of the input video, dividing a plurality of coding tree units in each frame of image, and calculating the gradient features of each frame and each coding tree unit in the frame; Encoding the video by different quantization parameters, and calculating the code rate of the encoded video; then quantizing the distortion between each pixel of the input video and the encoded video to obtain the distortion degree; Y Combining the gradient features of each frame, the distortion degree and the code rate of the video, and establishing the relationship between the distortion degree and the code rate of the video by fitting a cubic logarithmic model; Analyzing the input video by a target detection network, marking the ROI region, and dividing the compensation levels of the coding tree units in combination with the ROI region and the gradient features of the coding tree units; Combining the compensation levels of each coding tree unit, and allocating the encoding code rate for each coding tree unit in each image frame of the input video by a code rate allocation mechanism.

[0008] Further, each frame of image of the input video is converted into a gray image, and the gradients of the gray image in horizontal and vertical directions are calculated, and then the gradient amplitude of each pixel is calculated; Each frame of image is divided into a plurality of coding tree units, the average value of the pixel gradient amplitude in each coding tree unit is calculated, and the average pixel gradient of the coding tree unit is obtained as the gradient feature of the coding tree unit ; the average pixel gradient of the image frame is obtained by dividing the total gradient amplitude of the image frame by the total number of pixels, as the gradient feature of the image frame .

[0009] ​Further, the input video is encoded by different quantization parameters, and after encoding, the code rate of the encoded video corresponding to each quantization parameter is calculated; meanwhile, the distortion between each pixel of the input video and the encoded video is quantified in the form of mean square error to obtain the distortion degree of the video. Y

[0010] Further, a cubic logarithmic model is fitted by combining the image frame gradient feature, the distortion degree of the video and the code rate:

[0011] In the formula, D represents the distortion degree; R represents the input video code rate; G represents the gradient data of a single frame of the sequence; and P represents the model parameter.

[0012] Further, the input video is input into a target detection network, and each coding tree unit related to target detection in each frame of image is marked as an ROI region, and the compensation levels are divided by combining the ROI region and the coding tree unit gradient feature: If a coding tree unit belongs to the ROI region and the gradient feature value is higher than the image frame gradient feature value, the coding tree unit is divided into a first-level coding tree unit; If a coding tree unit only satisfies one of belonging to the ROI region or the gradient feature value being higher than the image frame gradient feature value, the coding tree unit is divided into a second-level coding tree unit; If a coding tree unit does not belong to the ROI region and the gradient feature value is lower than the image frame gradient feature value, the coding tree unit is divided into a third-level coding tree unit.

[0013] Further, a target code rate is set for each frame of image of the video, and the target code rate is divided into a total code rate for basic allocation and a total code rate for compensation; According to the total code rate for basic allocation, a basic code rate is allocated to each coding tree unit in each frame of image, and based on the compensation level of the coding tree unit, a compensation code rate is allocated to the coding tree unit by a code rate compensation mechanism on the basis of the basic code rate, and finally the code rate allocated to each coding tree unit in each frame of image is the sum of the basic code rate and the compensation code rate.

[0014] Further, the basic code rate allocated to each coding tree unit is represented as:

[0015] In the formula, Rtotal represents the total code rate for basic allocation of the current frame of image, Rbasic(i) represents the basic code rate allocated to the i-th coding tree unit, and W(i) represents the importance weight of the i-th coding tree unit. In the formula, Rtotal represents the total code rate for basic allocation of the current frame of image, Rbasic(i) represents the basic code rate allocated to the i-th coding tree unit, and W(i) represents the importance weight of the i-th coding tree unit.​​​​​​​

[0016] in, This is a balancing coefficient used to balance the ROI region and gradient features. The gradient features of the i-th coding tree unit, Represents the gradient features of the current frame image; As a binary variable, if the coding tree unit belongs to the ROI region, then Select 1 if the value is 1, otherwise select 0. At base bitrate Based on this, a rate compensation mechanism is used to allocate compensation rate to the first and second level coding tree units. The allocated compensation rate is expressed as follows:

[0017]

[0018] in, , Represents the first-level coding tree unit. This represents a second-level coding tree unit. For compensation coefficient, Let i be the distortion of the i-th coding tree unit. The distortion threshold; As the global base threshold, For the first k Priority coefficient of a hierarchical coding tree unit.

[0019] Furthermore, when allocating code rates to each coding tree unit, it is necessary to ensure that the total code rate does not exceed the limit: , The target bitrate for each frame is set; if the bitrate to be compensated by the first and second level coding tree units is greater than the total bitrate used for compensation in the current frame, then a portion is reduced from the base bitrate allocated to the third level coding tree unit to ensure the compensation bitrate of the first and second level coding tree units.

[0020] The beneficial effects of this invention are as follows: This invention proposes a video bitrate control method with enhanced target detection, which overcomes the technical limitations of traditional video coding schemes that "emphasize human vision and neglect machine learning". By integrating target detection and bitrate control, it achieves synergistic optimization of human visual quality and machine analysis reliability.

[0021] This invention establishes an intrinsic relationship between video distortion and bitrate by building a cubic logarithmic model based on gradient data. This model fully reflects the texture complexity and spatial features of video content, providing a more accurate mathematical basis for rate-distortion optimization. Simultaneously, this invention implements a differentiated bitrate allocation strategy, ensuring that more bitrate is allocated to textured or target-critical regions, preserving more edge and detail information and enhancing the feature extraction capabilities of machine vision algorithms. Bitrate is conserved in smooth or non-critical regions, avoiding resource waste and achieving a balance in overall coding performance. Furthermore, this invention introduces a compensation level division mechanism for coding tree units. Combining the region of interest output by the object detection network and the gradient features of the coding tree units, different compensation levels are assigned to the coding tree units. Based on the basic bitrate allocation, coding resources in critical regions are dynamically adjusted. According to the compensation level, more bitrate is further allocated to regions important for visual quality and object detection accuracy, thereby maintaining both high quality for human viewing and high reliability for machine analysis in the compressed video.

[0022] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic flowchart of a video bitrate control method for enhanced target detection provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the preprocessing process; Figure 3 Flowchart for classifying coding tree unit levels; Figure 4 This is a bitrate allocation mechanism. Detailed Implementation

[0024] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0026] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0027] like Figure 1 As shown, an embodiment of the present invention provides a video bitrate control method for enhanced target detection, the method comprising: 1. Preprocess each frame of the video to calculate the gradient data of each coding tree unit (CTU) in each frame.

[0028] like Figure 2 As shown, each frame of the test sequence is first converted to a grayscale image to reduce computational complexity while preserving the image's brightness information. Then, the Sobel operator is applied to the grayscale image to calculate the gradients in the horizontal and vertical directions, respectively, to obtain information about the grayscale changes of pixels in different directions.

[0029] Then, based on the calculated gradient, the gradient magnitude of each pixel is calculated to characterize the local grayscale changes of the pixel.

[0030] Based on the H.266 / VVC encoding standard, video frames are divided using a 128×128 pixel CTU.

[0031] For each coding tree unit (CTU), the average gradient magnitude per pixel is calculated to obtain the average gradient per pixel of that CTU, which serves as the CTU gradient feature and characterizes the intensity of local grayscale changes within the CTU. The average gradient per pixel of the entire frame is obtained by dividing the gradient magnitude of the entire frame by the total number of pixels, and this serves as the overall gradient feature of the entire frame, reflecting the overall grayscale change level at the frame level. The gradient feature can be used for subsequent bitrate allocation and rate-distortion optimization operations on the coding tree units.

[0032] 2. Regarding video bitrate and distortion A cubic logarithmic model based on gradient data was established.

[0033] First, HEVC test sequences are selected for encoding, and quantization parameters are selected. Full I-frame intra-coding is performed to cover the range of variations in coding quality, where, for each quantization parameter Each one is encoded once.

[0034] Then, after encoding is complete, for each The bitrate of the video sequence generated under the given value is calculated, and the bitrate of each pixel in the source video and the encoded video is compared. Y The distortion between components is quantified in the form of mean squared error (MSE).

[0035] Gradient features of each frame in the sequence G The bitrate and distortion data are analyzed and processed. A cubic logarithmic model is established through fitting to describe the relationship between video distortion and bitrate. This yields the video distortion level. D With bit rate R The relationship model is represented as follows:

[0036] In the formula, Indicates the degree of distortion, expressed as mean square error; Indicates the test sequence bit rate. The gradient features of a single frame in the sequence. Indicates model parameters.

[0037] 3. Combining the target detection network and gradient data, the compensation levels of the coding tree units are determined, such as... Figure 3 As shown.

[0038] (1) Combining the target detection network and gradient data, the coding tree unit is divided into compensation levels so that different regions can be treated differently in the subsequent coding process.

[0039] The specific operation is as follows: the test sequence is input into the object detection network, the video content of each frame is analyzed, and the coding tree units related to object detection are identified. These coding tree units are marked as regions of interest (ROI).

[0040] (2) If a coding tree unit satisfies both "belongs to the region of interest" and "its gradient feature value is higher than the gradient feature value of the image frame", it indicates that the region is both a key region for object detection and contains rich visual details, so it is classified as a first-level coding tree unit.

[0041] (3) For coding tree units that only meet one of the conditions of "belonging to the region of interest" or "gradient feature value is higher than the gradient feature value of the image frame", they are classified as second-level coding tree units to reflect their moderate importance in object detection or visual quality.

[0042] (4) For coding tree units that do not belong to the region of interest and whose gradient feature values ​​are lower than the gradient feature values ​​of the image frame, they are divided into third-level coding tree units to identify regions with smooth textures and low correlation with target detection.

[0043] 4. Construct a bitrate allocation mechanism within the framework of the rate-distortion optimization model.

[0044] By constructing a rate-distortion (RD) model based on gradient features and regional importance, we can achieve differentiated allocation of bitrate. The core is to minimize the weighted total distortion under the constraint of total bitrate.

[0045] (1) To minimize the CTU-weighted total distortion, an RD model integrating gradient features and ROI attributes is constructed, and rate-distortion optimization is performed in units of CTU. The optimization objective is:

[0046] in, The target bitrate for the current frame. The importance weight of the i-th CTU (integrating gradient and ROI features); Let i be the distortion of the i-th CTU. Let i be the bitrate of the i-th CTU. This is the set of CTU indices for the current frame.

[0047] The constrained optimization is transformed into unconstrained optimization using the Lagrange multiplier method, and the optimal bitrate for each CTU is solved. :

[0048] in, These are Lagrange multipliers (weights that balance distortion and bit rate). (2) CTU bit budget allocation based on multi-feature weights

[0049] in, ∈[0,1] is a balancing coefficient used to balance the ROI region and gradient features. Let i be the gradient feature of the i-th CTU. The gradient features of the current frame; if the CTU belongs to the ROI region, then Select 1 if the value is 1, otherwise select 0.

[0050] Based on this, the base code rate allocated to the i-th CTU is:

[0051] in, The total bitrate used for base allocation in the current frame. The base bitrate is allocated to the i-th CTU.

[0052] 5. A bitrate compensation mechanism that integrates detection accuracy and image quality is introduced to optimize the above bitrate allocation mechanism.

[0053] (1) Division of the bit rate pool:

[0054] in The size is , The total bitrate used for compensation, with a size of .

[0055] (2) Compensation rate allocation based on distortion feedback and detection accuracy: For Level 1 and Level 2 CTUs, a compensation bitrate is added to the base bitrate, resulting in a final bitrate of:

[0056]

[0057] in, , Representing the first-tier CTU, Representing the second level of CTU, , . This is the distortion threshold.

[0058] (3) Distortion threshold classification: like , ;like , ;like , ;

[0059] in, As the global base threshold, These are the priority coefficients for the three levels of CTU. , , , These are the priority coefficients for CTUs at the first, second, and third levels, respectively.

[0060] (4) Final bitrate constraint Ensure the total bitrate does not exceed the limit: If the budget is exceeded after compensation, the base bitrate of the third-level CTU will be appropriately reduced (up to 30%), with priority given to the first two levels.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A video bitrate control method for enhanced object detection, characterized in that, Each frame of the input video is preprocessed, and each frame is divided into multiple coding tree units. The gradient features of each frame and each coding tree unit in the frame are calculated. The video is encoded using different quantization parameters, and the bitrate of the encoded video is calculated. Then, the bitrate of each pixel in the input video and the encoded video is compared. Y The distortion between components is quantized to obtain the distortion degree. By combining the gradient features of each frame, the video distortion and bitrate, the relationship between video distortion and bitrate is established by fitting a cubic logarithmic model. The input video is analyzed by an object detection network, the ROI region is marked, and the compensation level of the coding tree unit is divided by combining the ROI region and the gradient features of the coding tree unit. By combining the compensation levels of each coding tree unit, a bitrate allocation mechanism is used to allocate coding bitrate to each coding tree unit in each image frame of the input video.

2. The method according to claim 1, characterized in that, Each frame of the input video is converted into a grayscale image, and the gradient of the grayscale image in the horizontal and vertical directions is calculated. Then the gradient magnitude of each pixel is calculated. Each frame of the image is divided into several coding tree units. The average pixel gradient magnitude within each coding tree unit is calculated to obtain the average gradient per pixel of the coding tree unit, which is then used as the gradient feature of the coding tree unit. The average gradient per pixel of an image frame is obtained by dividing the total gradient magnitude of the image frame by the total number of pixels, and this average gradient is used as the gradient feature of the image frame. .

3. The method according to claim 2, characterized in that, The input video is encoded using different quantization parameters. After encoding, the bitrate of the encoded video corresponding to each quantization parameter is calculated. Simultaneously, the bitrate of each pixel in the input video and the encoded video is compared. Y The distortion between components is quantified in the form of mean square error to obtain the video distortion level.

4. The method according to claim 3, characterized in that, By combining image frame gradient features, video distortion, and bitrate, a cubic logarithmic model is fitted: In the formula, Indicates the degree of distortion; Indicates the input video bitrate; Represents the gradient data of a single frame in the sequence; Indicates model parameters.

5. The method according to claim 4, characterized in that, The input video is fed into the object detection network. Each frame's encoding tree units related to object detection are labeled as Regions of Interest (ROIs). Compensation levels are then determined by combining the ROI regions and the gradient features of the encoding tree units. If a coding tree unit belongs to the ROI region and its gradient feature value is higher than the gradient feature value of the image frame, it is classified as a first-level coding tree unit. If a coding tree unit satisfies either belonging to a ROI region or having a gradient feature value higher than one of the gradient feature values ​​of the image frame, it is classified as a second-level coding tree unit. If a coding tree unit does not belong to the ROI region and its gradient feature value is lower than the gradient feature value of the image frame, it is classified as a third-level coding tree unit.

6. The method according to claim 5, characterized in that, Set a target bitrate for each frame of the video, and divide the target bitrate into a total bitrate for base allocation and a total bitrate for compensation; The base bitrate is allocated to each coding tree unit in each frame based on the total bitrate used for base allocation. At the same time, based on the compensation level of the coding tree unit, a compensation bitrate is allocated to the coding tree unit on the basis of the base bitrate through a bitrate compensation mechanism. Finally, the bitrate allocated to each coding tree unit in each frame is the sum of the base bitrate and the compensation bitrate.

7. The method according to claim 6, characterized in that, The base code rate allocated to each coding tree unit is represented as follows: in, The total bitrate used for the base allocation of the current frame image. The base code rate assigned to the i-th coding tree unit; The importance weight for the i-th coding tree unit: in, This is a balancing coefficient used to balance the ROI region and gradient features. The gradient features of the i-th coding tree unit, The gradient features of the current frame image; As a binary variable, if the coding tree unit belongs to the ROI region, then Select 1 if the value is 1, otherwise select 0. At base bitrate Based on this, a rate compensation mechanism is used to allocate compensation rate to the first and second level coding tree units. The allocated compensation rate is expressed as follows: in, , Represents the first-level coding tree unit. This represents a second-level coding tree unit. For compensation coefficient, Let i be the distortion of the i-th coding tree unit. The distortion threshold; As the global base threshold, For the first k Priority coefficient of a hierarchical coding tree unit.

8. The method according to claim 7, characterized in that, When allocating code rates to each coding tree unit, it is necessary to ensure that the total code rate does not exceed the limit: , The target bitrate for each frame is set; if the bitrate to be compensated by the first and second level coding tree units is greater than the total bitrate used for compensation in the current frame, then a portion is reduced from the base bitrate allocated to the third level coding tree unit to ensure the compensation bitrate of the first and second level coding tree units.

Images