A frame-level adaptive ROI video compression method based on deep learning

Abstract

The present invention relates to a frame-level adaptive ROI video compression method based on deep learning. The method uses a deep learning target detection model to dynamically analyze the content of a video scene, and dynamically calculates the ROI coding area according to key target position information in each frame. The method removes the redundancy of the external content of the ROI area by setting differentiated coding parameters, while ensuring the internal picture quality of the ROI area, thereby effectively reducing the video code stream, obtaining a higher compression rate, and greatly reducing the storage requirements of video data. The designed technical scheme is applicable to various monitoring scenarios, which can be general monitoring scenarios or monitoring scenarios focusing on specific objects. At the same time, the adaptive ROI calculation is performed on the target detection result, and the detection error of the target detection model has a certain correction capability. In addition, the entire technical scheme is compatible with a variety of video encoding and decoding standards, is easy to integrate, and has high reusability.

Description

Technical Field

[0001] The present invention relates to a frame-level adaptive ROI video compression method based on deep learning, belonging to the technical field of video compression processing. Background Art

[0002] In recent years, with the advancement of technology and increased awareness of safety, security systems have become increasingly important in people's daily lives. Typically, surveillance video needs to be stored for a certain period of time to accommodate subsequent video playback needs. However, with the increasing demand for security and the rapid development of big video data, the volume of video data is growing exponentially, and the cost of video storage and maintenance is also increasing. Traditional video encoding technologies such as H.264 / AVC and H.265 / HEVC are no longer able to meet these demands. Therefore, how to more effectively compress and encode video, achieve higher compression rates, and reduce storage costs while maintaining the quality of critical video content has become a pressing issue.

[0003] Surveillance scene video is unique. Unlike other video used in network and multimedia communications, most surveillance scenes contain non-critical static areas, such as the sky, road surface, walls, doors, and windows. Surveillance video contains a large amount of redundant information, and encoding and compressing the entire surveillance area would place significant pressure on video storage.

[0004] Currently, commonly used ROI extraction algorithms include background modeling and deep learning semantic segmentation. Background modeling algorithms build a mathematical model of the video sequence background and then subtract the image to be detected from the background model to obtain the moving target, thereby distinguishing the moving target from the background. Deep learning semantic segmentation algorithms use a neural network semantic segmentation model to distinguish the region of interest in the video from the background.

[0005] However, the aforementioned ROI extraction algorithm still has shortcomings. Background modeling algorithms are sensitive to moving objects and may over-extract irrelevant motion information such as animals and trembling leaves in the surveillance footage. However, in surveillance videos, we are more concerned with pedestrians and vehicles. Deep learning semantic segmentation algorithms are based on pixel-level segmentation, making dataset annotation extremely expensive. Model prediction errors can also cause deviations in compressed video regions. Furthermore, semantic segmentation models take longer to infer than detection and classification models, limiting their practical application. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a frame-level adaptive ROI video compression method based on deep learning. The deep learning target detection model is used to dynamically analyze the video scene content, and the ROI coding area is dynamically calculated according to the key target position information in each frame. By setting differentiated coding parameters, the redundancy of the content outside the ROI area is removed while ensuring the picture quality inside the ROI area, thereby effectively reducing the video bit rate and obtaining a higher compression rate.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: the present invention designs a frame-level adaptive ROI video compression method based on deep learning, which is used to compress the target video, including the following steps:

[0008] Step A. Obtain the image frame sequence corresponding to the target video and proceed to step B;

[0009] Step B. For each image frame in the image frame sequence, identify and obtain each target local image region belonging to each preset moving object type in the image frame, and then proceed to Step C;

[0010] Step C. For each image frame in the image frame sequence, obtain the entire circumscribed area of ​​all target local image regions in the image frame to form a region of interest in the image frame, that is, obtain the region of interest in each image frame, and then proceed to step D;

[0011] Step D: Based on the first CRF value CRF1 corresponding to the region of interest in the image frame and the preset adjustment parameter offset corresponding to the target video, the following formula is used:

[0012] CRF2=CRF1 / (1+offset)

[0013] offset = (CRF1 - CRF2) / CRF2

[0014] Obtain a second CRF value CRF2 corresponding to the remaining areas in the image frame, and then proceed to step E;

[0015] Step E. For each image frame in the image frame sequence, image encoding is performed on the region of interest in the image frame according to the first CRF value CRF1, and image encoding is performed on the remaining regions in the image frame according to the second CRF value CRF2, and the image frame is updated, and then each image frame is updated, and encapsulated to obtain the updated target video.

[0016] As an optimal technical solution of the present invention: in step A, if the target video is an encoded video, the target video is first parsed to obtain the image frame sequence corresponding to the target video, and then the image frame sequence is decoded, the image frame sequence corresponding to the target video is updated, and step B is entered.

[0017] As a preferred technical solution of the present invention: in the step B, for each image frame in the image frame sequence, a target recognition model is applied with the feature group corresponding to each pixel in the image frame as input and the classification label corresponding to each pixel in the image frame as output to identify each target local image area belonging to each preset moving object type in the image frame, and then enter step C.

[0018] As a preferred technical solution of the present invention: the target recognition model in step B is obtained according to the following steps i to iii;

[0019] Step i. Obtain a preset number of sample image frames containing preset types of moving objects, and determine the classification labels corresponding to the pixels in each sample image frame according to the preset classification labels of the moving object types and the image background types, and then proceed to step ii;

[0020] Step ii. For each sample image frame, obtain the feature value of each pixel in the sample image frame corresponding to each preset target type feature, constitute the feature group corresponding to each pixel, that is, obtain the feature group corresponding to each pixel in each sample image frame, and then proceed to step iii;

[0021] Step iii. Taking the feature groups corresponding to the pixels in the sample image frame as input and the classification labels corresponding to the pixels in the sample image frame as output, a preset classification network is trained to obtain a target recognition model.

[0022] As a preferred technical solution of the present invention: the preset target type features in step ii include pixel values ​​of RGB three channels.

[0023] As a preferred technical solution of the present invention: based on the identification of each target local image area in each image frame in step B, the minimum bounding rectangular area corresponding to each target local image area in each image frame is obtained. In step C, for each image frame in the image frame sequence, the minimum bounding rectangular area of ​​the entire minimum bounding rectangular area corresponding to each target local image area in the image frame is obtained to constitute the region of interest in the image frame, that is, the region of interest in each image frame is obtained.

[0024] As a preferred technical solution of the present invention: the value of the preset adjustment parameter offset is [-1, 0].

[0025] The frame-level adaptive ROI video compression method based on deep learning described in the present invention has the following technical effects compared with the existing technology:

[0026] The frame-level adaptive ROI video compression method designed by the present invention is based on deep learning. It uses a deep learning target detection model to dynamically analyze the content of the video scene, and dynamically calculates the ROI coding area according to the key target position information in each frame. By setting differentiated coding parameters, the redundancy of the external content of the ROI area is removed, while the internal picture quality of the ROI area is guaranteed, thereby effectively reducing the video bit rate, obtaining a higher compression rate, and greatly reducing the storage requirements of the video data. The designed technical solution is applicable to various monitoring scenarios, which can be general monitoring scenarios or monitoring scenarios focusing on specific objects. At the same time, adaptive ROI calculation is performed on the target detection results, and the detection error of the target detection model has a certain correction capability. In addition, the entire technical solution is adaptable to a variety of video coding and decoding standards, easy to integrate, and highly reusable. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the frame-level adaptive ROI video compression method based on deep learning designed by the present invention;

[0028] Figure 2 Schematic diagram of the implementation of step A of the present invention;

[0029] Figure 3 Schematic diagram of the implementation of step B of the present invention;

[0030] Figure 4 This is a schematic diagram of the implementation of the minimum circumscribed rectangular area corresponding to step C of the present invention;

[0031] Figure 5 Schematic diagram of the implementation of step E of the present invention; DETAILED DESCRIPTION

[0032] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] The present invention designs a frame-level adaptive ROI video compression method based on deep learning, which is used to compress the target video. In practical applications, such as Figure 1 As shown, specifically perform the following steps A to E.

[0034] Step A: Obtain the image frame sequence corresponding to the target video and proceed to step B.

[0035] like Figure 2 As shown, in the actual application of the above-mentioned step A, if the target video is an encoded video, the target video is first parsed to obtain the image frame sequence corresponding to the target video, and then the image frame sequence is decoded, the image frame sequence corresponding to the target video is updated, and step B is entered.

[0036] Step B: for each image frame in the image frame sequence, identify and obtain each target local image region belonging to each preset moving object type in the image frame, and then proceed to step C. The preset moving object types here include, for example, pedestrians and vehicles.

[0037] In practical application, the above step B is as follows: Figure 3 As shown, for each image frame in the image frame sequence, a target recognition model is specifically designed and applied with the feature group corresponding to each pixel in the image frame as input and the classification label corresponding to each pixel in the image frame as output to identify and obtain each target local image area belonging to each preset moving object type in the image frame; and regarding the target recognition model therein, in actual application, it is obtained according to the following steps i to iii.

[0038] Step i. Obtain a preset number of sample image frames containing preset moving object types, and determine the classification labels corresponding to each pixel in each sample image frame based on the preset classification labels of the preset moving object types and image background types, and then proceed to step ii.

[0039] The acquisition of each sample image frame here is designed to cover as many different scenes, different time periods, different lighting conditions, and different background information as possible.

[0040] Step ii. For each sample image frame, obtain the feature value corresponding to each preset target type feature for each pixel in the sample image frame, forming a feature group corresponding to each pixel, that is, obtain the feature group corresponding to each pixel in each sample image frame, and then proceed to step iii. In a specific implementation, the preset target type features are designed to use pixel values ​​of the RGB three channels.

[0041] Step iii. Train a pre-defined classification network using the feature set corresponding to each pixel in the sample image frame as input and the classification label corresponding to each pixel in the sample image frame as output to obtain an object recognition model. In practice, the pre-defined classification network used here can be a model such as Fast R-CNN or the YOLO series.

[0042] Regarding the identification and acquisition of each target local image region belonging to each preset moving object type in the image frame involved in step B, in the application, the minimum circumscribed rectangular region corresponding to each target local image region in each image frame is specifically obtained.

[0043] Step C. For each image frame in the image frame sequence, obtain the entire circumscribed area of ​​all target local image regions in the image frame to form the region of interest in the image frame, that is, obtain the region of interest in each image frame, and then proceed to step D.

[0044] Based on the design of the above step B, the identification of each target local image area in each image frame is achieved through the minimum bounding rectangular area. In step C here, for each image frame in the image frame sequence, the minimum bounding rectangular area of ​​the entire minimum bounding rectangular area corresponding to each target local image area in the image frame is obtained to constitute the region of interest in the image frame, that is, the region of interest in each image frame is obtained.

[0045] In actual implementation, the minimum bounding rectangle area corresponding to the region of interest in the image frame is obtained according to the following formula:

[0046]

[0047]

[0048]

[0049]

[0050] In formula (1) are the x-coordinates of the upper left corner of the first, second, and nth key target detection frames, respectively. Based on formula (1), the minimum value of the x-coordinates of all key target detection frames can be calculated. Corresponding to the x-coordinates of the lower right corners of the first, second, and nth key target detection frames, respectively, the maximum value of the x-coordinates of all key target detection frames can be calculated based on formula (2). are the y coordinates of the upper left corner of the first, second and nth key target detection frames respectively. Based on formula (3), the minimum value of the y coordinates of all key target detection frames can be calculated. These correspond to the y coordinates of the lower right corners of the first, second, and nth key target detection frames, respectively. Based on formula (4), the maximum value of the y coordinates of all key target detection frames can be calculated.

[0051] Then in practical application, the above step C is as follows: Figure 4 As shown, the region of interest in each image frame is obtained, where Figure 4 The dotted area in the left figure is the key target area. The adaptive ROI area calculated by formulas (1), (2), (3) and (4) corresponds to Figure 4 The solid-line box area in the right figure.

[0052] The adaptive ROI calculation strategy corrects errors that may occur during scene content analysis in step B. The object recognition model used in step B is subject to some degree of missed and false detections when acquiring key object detection frames. Using only individual object detection results as the ROI can lead to deviations in the compressed area and data loss in key object regions. Therefore, comprehensively considering all detection results in each frame and calculating the minimum bounding rectangle of multiple key object regions as the ROI can significantly correct errors in scene content analysis.

[0053] Traditional video coding technologies such as H.264 / AVC and H.265 / HEVC all utilize a hybrid coding framework, specifically comprising modules such as intra-frame prediction, inter-frame prediction, transform, quantization, and entropy coding. Intra-frame prediction primarily removes spatial correlation in images. This occurs because the grayscale variation between adjacent pixels in an image is minimal, resulting in spatial redundancy. Inter-frame prediction primarily removes temporal correlation in images. This occurs because video contains multiple frames per second, and pixels in corresponding or nearby locations between previous and subsequent frames are very similar, resulting in temporal redundancy. Transform coding transforms the image from a time-domain signal to the frequency domain, concentrating energy in low-frequency regions. Quantization reduces the dynamic range of image coding. Entropy coding encodes coding control data, quantized transform coefficients, intra-frame prediction data, and motion data into a binary stream for storage.

[0054] The only lossy process in the aforementioned hybrid video coding framework is quantization. Quantization is the process of mapping a signal's continuous values ​​(or a large number of possible discrete values) into a finite number of discrete amplitudes, achieving a many-to-one mapping of signal values. In video coding, the transform coefficients transformed to the frequency domain have a large dynamic range. Quantizing these transform coefficients effectively reduces the signal value space, achieving good compression. Therefore, a larger quantization parameter results in a higher video compression ratio, a lower bitrate, and lower quality. Conversely, a smaller quantization parameter results in a lower video compression ratio, a higher bitrate, and better quality.

[0055] In actual video coding, the Constant Rate Factor (CRF) is an important encoding parameter for video quality. To ensure consistent subjective quality across the entire video, CRF dynamically adjusts the compression size of each frame, using different quantization parameters for each frame.

[0056] The differentiated encoding parameter settings designed in step D next involve setting different video encoding parameters for the regions inside and outside the ROI within each frame. This differentiation is achieved by setting the CRF within the encoding parameters. During the encoding process, different CRF values ​​are set for different ROI regions to maintain the desired video quality while minimizing the video bitrate. Generally speaking, a higher CRF indicates lower video quality and a smaller storage footprint; conversely, a lower CRF indicates better video quality and a larger storage footprint.

[0057] Step D: Based on the first CRF value CRF1 corresponding to the region of interest in the image frame and the preset adjustment parameter offset corresponding to the target video, the following formula is used:

[0058] CRF2=CRF1 / (1+offset)

[0059] offset = (CRF1 - CRF2) / CRF2

[0060] Obtain the second CRF value CRF2 corresponding to the remaining areas in the image frame, and then proceed to step E. In practical applications, the offset value range here is [-1, 0]. For CRF1, refer to the default parameter settings of the H.264 / AVC and H.265 / HEVC open source encoders, or set it according to image quality requirements.

[0061] Step E. For each image frame in the image frame sequence, image encoding is performed on the region of interest in the image frame according to the first CRF value CRF1, and image encoding is performed on the remaining regions in the image frame according to the second CRF value CRF2, and the image frame is updated, and then each image frame is updated, and encapsulated to obtain the updated target video.

[0062] In practical applications, if offset = 0, the CRF values ​​outside and inside the ROI in the image are the same. When offset < 0, CRF2 is greater than CRF1, and the compression rate outside the image ROI is greater than that inside the ROI, that is, the image quality outside the ROI is worse than that inside the ROI; when offset > 0, CRF2 is less than CRF1, and the compression rate outside the image ROI is lower than that inside the ROI, and the image quality outside the ROI is better than that inside the ROI.

[0063] The above-mentioned technical solution designs a frame-level adaptive ROI video compression method based on deep learning. It uses a deep learning target detection model to dynamically analyze the content of the video scene, and dynamically calculates the ROI coding area according to the key target position information in each frame. By setting differentiated coding parameters, it removes the redundancy of content outside the ROI area while ensuring the picture quality inside the ROI area, thereby effectively reducing the video bit rate, obtaining a higher compression rate, and greatly reducing the storage requirements of video data. The designed technical solution is suitable for various monitoring scenarios, which can be general monitoring scenarios or monitoring scenarios focusing on specific objects. At the same time, it performs adaptive ROI calculation on the target detection results, and has a certain correction capability for the detection errors (missed detection, false detection, etc.) of the target detection model. In addition, the entire technical solution is compatible with multiple video coding standards (such as H.264 / AVC, H.265 / HEVC, etc.), is easy to integrate, and has high reusability.

[0064] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in this field without departing from the spirit of the present invention.

Claims

1. A frame-level adaptive ROI video compression method based on deep learning, used for compressing a target video, characterized in that: The steps include: Step A. Obtain the image frame sequence corresponding to the target video and proceed to step B; Step B. for each image frame in the image frame sequence, identify and obtain each target local image region belonging to each preset moving object type in the image frame, and then proceed to step C; Step C. for each image frame in the image frame sequence, obtain the circumscribed area of ​​all the target local image regions in the image frame as a whole to form the region of interest in the image frame, that is, obtain the region of interest in each image frame, and then proceed to step D; Step D: According to the first CRF value CRF1 corresponding to the region of interest in the image frame, combined with the preset adjustment parameter offset corresponding to the target video, the following formula is used: CRF2=CRF1 / (1+offset) offset=(CRF1-CRF2) / CRF2 Obtain a second CRF value CRF2 corresponding to the remaining areas in the image frame, and then proceed to step E; Step E. For each image frame in the image frame sequence, image encoding is performed on the region of interest in the image frame according to the first CRF value CRF1, and image encoding is performed on the remaining region in the image frame according to the second CRF value CRF2, and the image frame is updated, and then each image frame is updated, and encapsulated to obtain an updated target video.

2. The frame-level adaptive ROI video compression method based on deep learning according to claim 1, characterized in that: In step A, if the target video is an encoded video, the target video is first parsed to obtain the image frame sequence corresponding to the target video, and then the image frame sequence is decoded to update the image frame sequence corresponding to the target video, and then step B is entered.

3. The frame-level adaptive ROI video compression method based on deep learning according to claim 1, characterized in that: In step B, for each image frame in the image frame sequence, a target recognition model is applied with the feature group corresponding to each pixel in the image frame as input and the classification label corresponding to each pixel in the image frame as output to identify each target local image area belonging to each preset moving object type in the image frame, and then proceed to step C.

4. The frame-level adaptive ROI video compression method based on deep learning according to claim 3, characterized in that: The target recognition model in step B is obtained according to the following steps i to iii; Step i. Obtain a preset number of sample image frames containing preset moving object types, and determine the classification labels corresponding to the pixels in each sample image frame according to the preset moving object types and the preset classification labels of the image background type, and then proceed to step ii; Step ii. For each sample image frame, obtain the feature value of each pixel in the sample image frame corresponding to each preset target type feature, and form a feature group corresponding to each pixel, that is, obtain the feature group corresponding to each pixel in each sample image frame, and then enter step iii; Step iii. Taking the feature groups corresponding to the pixels in the sample image frame as input and the classification labels corresponding to the pixels in the sample image frame as output, training is performed on a preset classification network to obtain a target recognition model.

5. The frame-level adaptive ROI video compression method based on deep learning according to claim 4, characterized in that: The preset target type features in step ii include pixel values ​​of three channels of RGB.

6. The frame-level adaptive ROI video compression method based on deep learning according to claim 1, characterized in that: Based on the identification of each target local image area in each image frame in step B, the minimum circumscribed rectangular area corresponding to each target local image area in each image frame is obtained. In step C, for each image frame in the image frame sequence, the minimum circumscribed rectangular area of ​​the entire minimum circumscribed rectangular area corresponding to each target local image area in the image frame is obtained to constitute the region of interest in the image frame, that is, the region of interest in each image frame is obtained.

7. The frame-level adaptive ROI video compression method based on deep learning according to claim 1, characterized in that: The preset adjustment parameter offset has a value of [-1, 0].

Images