Region of interest based depth video coding method, system and storage medium

Abstract

A region of interest based deep video coding method, system and storage medium, the method comprising: acquiring video frames in real time and preprocessing; inputting the preprocessed real-time video frames into a pre-trained deep target detection network for identification and generating an ROI mask; inputting the real-time video frames, the corresponding ROI mask and the encoding reference frame into a deep video coding network respectively, non-uniformly encoding the real-time video frames in combination with the ROI mask and the encoding reference frame to obtain a plurality of initial video code streams after encoding; packaging the plurality of initial video code streams into video data packets and sending them to a receiving end; and reorganizing the data in the video data packets according to a preset reorganization logic to form complete video code streams. The system is used to implement the above method, and the medium executes the above method through a processor. The region of interest based deep video coding method, system and storage medium can meet the requirements of high-quality video with low delay, high resolution and high frame rate.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and in particular to a depth video encoding and decoding method based on regions of interest. Background Technology

[0002] With the rapid development of mobile internet and communication technologies, video services have been widely used in advertising, healthcare, live streaming, IPTV, security monitoring, and many other fields, consuming the vast majority of network bandwidth. The emergence of massive amounts of video data has also brought extremely serious challenges to video storage and transmission; therefore, how to compress video data more efficiently has become a critical issue.

[0003] The current demand for video compression stems primarily from two aspects: the surge in video data volume and the expanding application areas of video. With increasing network bandwidth and rising user experience demands, the proportion of ultra-high-definition video, high dynamic range video, and immersive application video in the overall network data is growing rapidly. For example, a single frame of uncompressed 4K video image is approximately 23.7MB. Real-time transmission at a frame rate of 60Hz requires a speed of at least 1.4Gb / s, which is difficult for typical 100Mbps or gigabit networks to support. Furthermore, the emergence of new applications such as remote driving, virtual reality, and video-on-demand is creating increasing demands for low-latency, high-resolution, and high-frame-rate high-quality video, rendering traditional video compression methods inadequate for these emerging scenarios. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a depth video encoding and decoding method based on region of interest, which can improve video quality, reduce the time and amount of computation for encoding and decoding the entire video, reduce computational complexity and memory consumption, improve computational efficiency, and reduce latency.

[0005] This invention provides a depth video encoding and decoding method based on regions of interest, comprising the following steps:

[0006] S1, acquire video frames in real time and preprocess the acquired real-time video frames;

[0007] S2, input the pre-processed real-time video frames into the pre-trained deep object detection network for recognition and generate ROI masks;

[0008] S3, input the real-time video frame, the corresponding ROI mask and the coding reference frame into the deep video coding network respectively, and perform non-uniform coding on the real-time video frame in combination with the ROI mask and the coding reference frame to obtain several initial video bitstreams after coding.

[0009] S4, the initial video streams are packaged into a video data packet and sent to the video data receiving module;

[0010] S5 reassembles the data in the video data packet according to the preset reassembly logic to form a complete video stream;

[0011] S6: Input the complete video stream and the decoding reference frame into the deep video decoding network to obtain the decoded video frame.

[0012] In one embodiment, step S1, which involves acquiring video frames in real time and preprocessing the acquired real-time video frames, further includes:

[0013] S101, real-time acquisition of video frames;

[0014] S102 performs noise reduction, correction, and downsampling preprocessing on the acquired real-time video frames.

[0015] In one embodiment, step S2, which involves inputting the preprocessed real-time video frame into a pre-trained deep object detection network for identification and generation of ROI masks, further includes:

[0016] S201, input the pre-processed real-time video frames into the pre-trained deep object detection network;

[0017] S202, Identify the region of interest in the real-time video frame and generate the ROI mask from the region of interest.

[0018] In one embodiment, step S4, which involves packaging several initial video streams into a video data packet and sending it to the receiving end, further includes:

[0019] S401, mark the sequence number and timestamp of the encoded real-time video frames in the initial video stream;

[0020] S402, the marked initial video stream is packaged into video data packets according to the video transmission protocol and sent to the receiving end.

[0021] In one embodiment, step S5, which reassembles the data in the video data packet according to a preset reassembly logic to form a complete video stream, further includes:

[0022] S501, sort the encoded real-time video frames in the initial video stream according to the marked sequence number and timestamp;

[0023] S502 reassembles the encoded real-time video frames according to the arrangement order to form a complete video stream.

[0024] In one embodiment, the encoded reference frame is selected from previously acquired real-time video frames, and the number of encoded reference frames is one or more.

[0025] In one embodiment, the deep video coding network is generated by a pre-trained end-to-end video compression model (DVC).

[0026] In one embodiment, the depth video encoding / decoding method based on region of interest further includes the following steps:

[0027] S7 performs post-processing and rendering operations on the decoded real-time video frames and outputs the processed real-time video frames.

[0028] The present invention also provides a depth video encoding and decoding system based on a region of interest, employing any one of the above-described depth video encoding and decoding methods based on a region of interest, characterized in that it includes:

[0029] The image acquisition and preprocessing module is used to acquire video frames in real time and preprocess the acquired real-time video frames.

[0030] The deep object detection network module is used to identify the input preprocessed real-time video frames and generate ROI masks;

[0031] The depth video encoding module is used to acquire real-time video frames, corresponding ROI masks and encoding reference frames, and perform non-uniform encoding on the real-time video frames in combination with ROI masks and encoding reference frames to obtain several initial video bitstreams after encoding.

[0032] The video data sending module is used to package several of the initial video streams into a video data packet and send it to the video data receiving module;

[0033] The video data receiving module is used to reassemble the data in the received video data packets according to a preset reassembly logic to form a complete video stream;

[0034] The video data receiving module is used to reassemble the data in the received video data packets according to a preset reassembly logic to form a complete video stream;

[0035] The depth video decoding module is used to decode the input complete video bitstream and the decoding reference frame to obtain the decoded video frame.

[0036] The deep video encoding and decoding method based on regions of interest provided by this invention analyzes and extracts regions of interest in a video using a deep learning model, and then uses deep learning-based video encoding and decoding technology for compression. This method can improve video quality while reducing the time and computational load of encoding and decoding the entire video, reducing computational complexity and memory consumption, improving computational efficiency, and reducing latency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the depth video encoding / decoding method based on the region of interest provided in Embodiment 1.

[0039] Figure 2 This is a schematic diagram of the recognition of a pre-trained deep object detection network for the region-of-interest-based deep video encoding and decoding method provided in Example 1.

[0040] Figure 3 A schematic diagram of the ROI mask generated by the region of interest-based depth video encoding / decoding method provided in Example 1;

[0041] Figure 4 This is a schematic diagram of the structure of the video encoding / decoding network for the region-of-interest-based depth video encoding / decoding method provided in Embodiment 1.

[0042] Figure 5 The image shows the encoded result of the depth video encoding / decoding method based on region of interest provided in Example 1.

[0043] Figure 6 This is a schematic diagram of the structure of a depth video encoding and decoding system based on a region of interest, as provided in Embodiment 2. Detailed Implementation

[0044] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. Based on the description of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0045] In the description of this invention, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0046] The terms “upper,” “lower,” “left,” “right,” “front,” “back,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use. They are only for the convenience of description and simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0047] The terms “first,” “second,” “third,” etc., are used merely to distinguish elements with similar properties, not to indicate or imply relative importance or a specific order.

[0048] The terms “include,” “comprising,” or any other variation thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0049] Please refer to Figure 1 The depth video encoding and decoding method based on region of interest provided by this invention includes the following steps:

[0050] S1 acquires video frames in real time and preprocesses the acquired real-time video frames.

[0051] It is understood that step S1 further includes:

[0052] S101, real-time acquisition of video frames.

[0053] It is known that video frames can be acquired in real time through a camera. The camera can be a regular network camera or an industrial camera that can directly read the raw image without encoding or decoding. Industrial cameras include, but are not limited to, area array CCD industrial cameras, line array CCD industrial cameras, three-line sensor CCD industrial cameras, and interleaved transmission CCD industrial cameras.

[0054] S102 performs preprocessing on the acquired real-time video frames, including denoising, correction, and downsampling.

[0055] Understandably, real-time video frames that have undergone preprocessing such as denoising, correction, and downsampling can improve the accuracy and efficiency of subsequent processing and analysis.

[0056] S2 inputs the pre-processed real-time video frames into a pre-trained deep object detection network for identification and generation of ROI masks.

[0057] In this embodiment, step S2 further includes:

[0058] S201 inputs the pre-processed real-time video frames into the pre-trained deep object detection network.

[0059] It is known that the pre-trained deep object detection network is an object detection model trained based on real-world industrial scenarios, referring to... Figure 2 and Figure 3 Pre-trained deep object detection networks can automatically divide regions of interest, which refer to the parts of a video frame that require special attention or processing. The specific division of regions of interest is determined by the actual industrial application scenario. For example, in the autonomous driving scenario, the region of interest is the area containing dynamic objects such as pedestrians and vehicles, as well as the area containing static objects such as lane lines, zebra crossings, and traffic lights.

[0060] The pre-trained object detection model described above was obtained by pre-training the YOLOv5 object detection model. The specific pre-training steps for the object detection model are as follows:

[0061] Data collection: Record a large amount of high-quality video of actual industrial scenes using the deployed cameras. It is necessary to ensure that the video includes a wide variety of targets, covering as many factors as possible such as the shape, size, color, and angle of the targets. At the same time, it is also necessary to cover different time periods, lighting, weather, and background environments as much as possible, in order to prepare for the subsequent construction of the dataset.

[0062] Dataset preprocessing: The acquired video is cut and segmented into frames, converted into images, and images containing a variety of targets are selected. At the same time, preprocessing is performed on the images, including cutting, scaling, rotation, watermark removal, and noise reduction, to ensure image quality.

[0063] Dataset creation and verification: Use appropriate annotation tools (such as LabelImg, VGG Image Annotator, etc.) to annotate the location and category of the target of interest in each image data to obtain the annotated dataset. At the same time, complete the division of the dataset (training set, test set and validation set) and perform multiple verifications to improve the reliability of the annotation.

[0064] Model design and configuration: Select a suitable object detection network structure based on the application scenario and performance requirements. At the same time, consider the actual computing resources to configure relevant parameters for the model, such as learning rate, batch size, and number of training rounds. Also, design relevant loss functions and evaluation metrics, such as cross-entropy loss and accuracy, in order to improve the efficiency of model training.

[0065] Model training: Input predefined training set data, and continuously update network weights and bias parameters through optimization methods such as backpropagation and gradient descent to minimize the loss function and find the optimal model parameters. During the training process, data augmentation and regularization methods can also be used to prevent overfitting, which would result in poor generalization of the model.

[0066] Model testing and optimization: After model training is completed, the performance of the trained and adjusted model is tested using a predefined test set. At the same time, the model's performance is evaluated based on common object detection evaluation metrics such as IOU, FPS, AP, and mAP. Based on the test results and the performance requirements in actual applications, the model's parameters and structure are further adjusted by methods such as increasing training data, modifying the network structure, and adjusting hyperparameters to help the model achieve optimal performance.

[0067] S202, identify the region of interest in the real-time video frame, and generate the ROI (region of interest) mask from the region of interest.

[0068] It is known that the generated ROI mask can represent the location information of the region of interest.

[0069] S3, input the real-time video frame, the corresponding ROI mask and the coding reference frame into the deep video coding network respectively, and perform non-uniform coding on the real-time video frame in combination with the ROI mask and the coding reference frame to obtain several initial video bitstreams after coding.

[0070] It is understandable that the encoding reference frame is selected from previously acquired real-time video frames, and the number of encoding reference frames is one or more. By using deep learning-based video encoding and decoding methods, deep video coding networks can overcome the performance limitations of traditional codecs and achieve higher video quality and lower latency.

[0071] In this embodiment, the deep video coding network uses the end-to-end video compression algorithm DVC, which is based on the existing deep video compression framework. Furthermore, ROI mask information is introduced to train the deep model, thereby achieving deep video encoding and decoding based on the region of interest. A schematic diagram of the model's network structure can be found in [reference needed]. Figure 4 , Figure 4 The deep video coding network in the model includes a motion estimation network, a motion information encoder, a residual information encoder, and a decoding frame buffer. ROI mask information, real-time video frames, and coding reference frames are input to the motion estimation network. After processing by the motion estimation network, they are input to the motion information encoder for transformation and quantization. The reference frame is a real-time video frame that is input to the residual information encoder, processed, and then input to the decoding buffer. The decoding reference frame is output from the decoding buffer.

[0072] The specific pre-training steps for the aforementioned deep video encoding / decoding model are as follows:

[0073] Data collection: Record a large amount of high-quality video of actual industrial scenes using the deployed cameras. It is necessary to ensure that the video includes a wide variety of targets, covering various factors such as the shape, size, color, and angle of the targets. At the same time, it is necessary to cover different time periods, lighting, weather, and background environments as much as possible in order to prepare for the subsequent construction of the dataset.

[0074] Dataset preprocessing: The acquired video is cut and segmented into frames to convert it into images. At the same time, preprocessing is performed on the images, including cutting, scaling, rotation, watermark removal, and noise reduction, to ensure image quality.

[0075] Dataset creation and ROI information acquisition: Seven consecutive image frames are used as a video sequence. The recorded data is made into a dataset of as many video segments as possible. At the same time, a pre-trained object detection model is used to generate ROI mask information corresponding to each video image. The dataset is divided into training set, test set and validation set. Multiple proofreadings are performed to improve the accuracy of ROI information extraction.

[0076] Model Design and Configuration: Design a suitable end-to-end deep video codec network structure based on the application scenario and performance requirements. At the same time, consider the actual computing resources to configure relevant parameters for the model, such as learning rate, batch size, and number of training rounds. In addition, combine ROI information to design relevant loss functions and evaluation metrics, such as MSE, MS-SSIM, and PSNR, to measure the difference between the model-reconstructed frames and the actual frames, so as to improve the efficiency of model training.

[0077] Model training: Input predefined training set data, and continuously update network weights and bias parameters through optimization methods such as backpropagation and gradient descent to minimize the loss function and find the optimal model parameters. During the training process, data augmentation and regularization methods can also be used to prevent overfitting, which would result in poor generalization of the model.

[0078] Model Testing and Optimization: After completing model training, the performance of the trained and adjusted model is tested using a predefined test set. At the same time, the model's performance is evaluated based on common video compression evaluation metrics such as MSE, PSNR, MS-SSIM, VMAF, and encoding / decoding latency. Based on the test results and the performance requirements in practical applications, the model's parameters and structure are further adjusted by increasing training data, modifying the network structure, and adjusting hyperparameters to help the model achieve optimal performance.

[0079] Lightweighting of the model: Existing end-to-end video compression schemes generally have high complexity and lack specific hardware platforms for parallel acceleration, which cannot meet the requirements of real-time encoding and decoding. Therefore, it is necessary to perform lightweighting of the optimized video compression model through methods such as structure optimization, network pruning, and weight quantization to reduce the number of model parameters as much as possible, thereby reducing computational complexity and making it easier to deploy the model in real industrial scenarios.

[0080] S4, the initial video streams are packaged into a video data packet and sent to the video data receiving module.

[0081] It is understood that step S4 above further includes:

[0082] S401 marks the sequence number and timestamp of the encoded real-time video frames in the initial video stream.

[0083] Understandably, the sequence number is used to indicate that the real-time video frame belongs to the same video stream, and the timestamp can be the time information of when the real-time video frame was captured.

[0084] S402, the marked initial video stream is packaged into video data packets according to the video transmission protocol and sent to the video data receiving module.

[0085] It is known that in order to meet the requirements of fast and large-scale transmission, the video transmission protocol chosen is RTP (Real-time Transport Protocol). Other video transmission protocols with better transmission performance can also be selected according to actual needs. Transmission technologies include high-speed Wi-Fi, wired, 5G / 4G, etc. The specific choice can be made based on the actual application scenario.

[0086] Throughout the transmission process, RTCP (Real-Time Transmission Control Protocol) can be used for auxiliary feedback, such as the sender and receiver periodically exchanging the latest status information, thereby further improving the transmission quality.

[0087] S5 reassembles the data in the video data packet according to a preset reassembly logic to form a complete video stream.

[0088] It can be understood that step S5 above further includes:

[0089] S501, sort the encoded real-time video frames in the initial video stream according to the marked sequence number and timestamp.

[0090] It is known that the real-time video frames belonging to the same sequence are first sorted according to their sequence number, and then the encoded real-time video frames are sorted according to their timestamps.

[0091] S502 reassembles the encoded real-time video frames according to the arrangement order to form a complete video stream.

[0092] In this embodiment, the reconstructed complete video stream is arranged entirely in the order of its acquisition time.

[0093] S6: Input the complete video stream and the decoding reference frame into the deep video decoding network to obtain the decoded video frame.

[0094] S7 performs post-processing and rendering operations on the decoded real-time video frames and outputs the processed real-time video frames.

[0095] It is understood that post-processing and rendering operations include, but are not limited to, noise reduction, sharpening, color space transformation, color balance and correction, etc., which aim to improve image quality and enhance visual effects, and help provide users with the best visual experience. The video frames processed in the above way can be directly input to the display for display, or input to other modules, such as the perception module, for further processing of the video images. The aforementioned displays include, but are not limited to, CRT displays, LCD displays, LED displays, etc.

[0096] Example 2

[0097] Please see Figure 6 The depth video encoding and decoding system based on region of interest provided by the present invention employs the aforementioned depth video encoding and decoding method based on region of interest, characterized in that it includes:

[0098] Image acquisition and preprocessing module 1 is used to acquire video frames in real time and preprocess the acquired real-time video frames.

[0099] Deep object detection network module 2 is used to identify the input preprocessed real-time video frames and generate ROI masks;

[0100] The depth video encoding module 3 is used to acquire real-time video frames, corresponding ROI masks and encoding reference frames, and perform non-uniform encoding on real-time video frames in combination with ROI masks and encoding reference frames to obtain several initial video bitstreams after encoding.

[0101] Video data sending module 4: used to package several initial video streams into a video data packet and send it to video data receiving module 5;

[0102] The video data receiving module 5 is used to reassemble the data in the received video data packets according to a preset reassembly logic to form a complete video stream;

[0103] The depth video decoding module 6 is used to decode the input complete video bitstream and the decoding reference frame to obtain the decoded video frame;

[0104] The post-processing and rendering module is used to perform post-processing and rendering operations on the decoded real-time video frames and output the processed real-time video frames.

[0105] Example 3

[0106] The present invention provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, it implements the aforementioned depth video encoding and decoding method based on region of interest. The computer-readable storage medium includes, but is not limited to, various storage media such as ROM, RAM, hard disk, USB flash drive, magnetic disk or optical disk that can be used to store computer programs and support access by the computer device.

[0107] As described above, the depth video encoding and decoding method, system, and storage medium based on regions of interest provided by this invention utilize a pre-trained deep object detection network to extract regions of interest. By focusing on processing regions of interest during encoding, the performance of video compression can be improved while significantly reducing labor costs. Furthermore, while ensuring the compression quality of regions of interest, the bitrate of video stream compression in other regions can be reduced as much as possible to better save encoding resources. This enables high-quality video transmission under conditions of limited latency and bandwidth, thereby meeting the requirements of low latency, high resolution, and high frame rate for high-quality video.

[0108] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A depth video encoding / decoding method based on region of interest, characterized in that, Includes the following steps: S1, acquire video frames in real time and preprocess the acquired real-time video frames; S2, input the pre-processed real-time video frame into the pre-trained deep object detection network for identification to identify the region of interest in the real-time video frame, and generate the ROI mask for the region of interest; S3, the real-time video frame, the corresponding ROI mask, and the encoding reference frame are respectively input into the deep video coding network. The real-time video frame is non-uniformly encoded by combining the ROI mask and the encoding reference frame to obtain several initial video bitstreams after encoding. The deep video coding network includes a motion estimation network, a motion information encoder, a residual information encoder, and a decoding frame buffer. The ROI mask information, the real-time video frame, and the encoding reference frame are input into the motion estimation network. After processing by the motion estimation network, they are input into the motion information encoder for transformation and quantization. The reference frame is the real-time video frame that is input into the residual information encoder and then input into the decoding buffer. The decoding reference frame is output from the decoding buffer. The deep video coding network is generated by a pre-trained end-to-end video compression model (DVC). S4, mark the sequence number and timestamp of the encoded real-time video frames in the initial video bitstream, package the marked initial video bitstream into video data packets according to the video transmission protocol, and send them to the video data receiving module; S5, sort the encoded real-time video frames in the initial video stream according to the marked sequence number and timestamp, and reassemble the encoded real-time video frames according to the sorting order to form a complete video stream; S6, input the complete video stream and the decoding reference frame into the deep video decoding network to obtain the decoded video frame.

2. The depth video encoding and decoding method based on region of interest as described in claim 1, characterized in that, Step S1, which involves acquiring video frames in real time and preprocessing the acquired real-time video frames, further includes: S101, real-time acquisition of video frames; S102 performs preprocessing on the acquired real-time video frames, including denoising, correction, and downsampling.

3. The depth video encoding and decoding method based on region of interest as described in claim 1, characterized in that, The encoded reference frame is selected from previously acquired real-time video frames, and the number of encoded reference frames is one or more.

4. The depth video encoding and decoding method based on region of interest as described in claim 1, characterized in that, The depth video encoding / decoding method based on region of interest further includes the following steps: S7 performs post-processing and rendering operations on the decoded real-time video frames and outputs the processed real-time video frames to the display module.

5. A depth video encoding and decoding system based on a region of interest, used to implement the depth video encoding and decoding method based on a region of interest as described in any one of claims 1 to 4, characterized in that, include: The image acquisition and preprocessing module is used to acquire video frames in real time and preprocess the acquired real-time video frames. The deep object detection network module is used to identify the input preprocessed real-time video frames and generate ROI masks; The depth video encoding module is used to acquire real-time video frames, corresponding ROI masks and encoding reference frames, and perform non-uniform encoding on the real-time video frames in combination with ROI masks and encoding reference frames to obtain several initial video bitstreams after encoding. The video data sending module is used to package several initial video streams into a video data packet and send it to the receiving end; The video data receiving module is used to reassemble the data in the received video data packets according to a preset reassembly logic to form a complete video stream; The depth video decoding module is used to decode the input complete video bitstream and the decoding reference frame to obtain the decoded video frame.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the region-of-interest-based depth video encoding and decoding method as described in any one of claims 1 to 4.

Images