Method for fusing video stream and high-definition background frame

Abstract

The application discloses a fusion method of video code stream and high-definition background frame, comprising the following steps: decoding input code stream, including standard main code stream and self-defined high-definition background frame code stream, obtaining current main video frame and high-definition background frame; carrying out canny boundary detection on the current main video frame, obtaining an image including image content boundary; carrying out Gaussian blurring on the current main video frame and the high-definition background frame; marking the two images after Gaussian blurring as "background block" and "uncertain block" respectively; comparing the boundary image obtained by boundary detection with the images after block marking, determining "background block" and "foreground block"; indexing all 8x8 blocks, replacing all the blocks marked as "background block" with data in the high-definition background frame, carrying out smoothing filtering on the block boundary of adjacent blocks with different types, and finally outputting the fused video image.

Description

Technical Field

[0001] The present invention belongs to the technical field of video decoding, and in particular relates to a method for fusing a video code stream with a high-definition background frame. Background Art

[0002] In the field of video surveillance, a large amount of video data needs to be transmitted and stored, which consumes a lot of network bandwidth resources. If the video bit rate is reduced, the clarity of the surveillance video cannot be guaranteed. If the video bit rate is increased, more network bandwidth resources will be used, resulting in increased operating costs. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for fusing a video stream with a high-definition background frame.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A method for fusing a video stream with a high-definition background frame comprises the following steps:

[0006] Decode the input stream, including the standard main stream and the customized high-definition background frame stream, to obtain the current main video frame and high-definition background frame;

[0007] Perform canny boundary detection on the current main video frame to obtain an image including the boundary of the image content;

[0008] Perform Gaussian blur on the current main video frame and the HD background frame;

[0009] The block types of the main video frame and the high-definition background frame after Gaussian blur are marked as "background block" and "uncertain block" respectively;

[0010] Compare the boundary image obtained by boundary detection with the block-marked image. If most of the blocks within the boundary are "background blocks", all the blocks within the boundary are marked as "background blocks"; if most of the blocks within the boundary are "foreground blocks", all the blocks within the boundary are marked as "foreground blocks";

[0011] Index all 8x8 blocks, replace all blocks marked as "background blocks" with data from the HD background frame, perform smoothing filtering on the block boundaries between adjacent blocks of different types, and finally output the fused video image.

[0012] In a possible implementation manner, most of the thresholds are above 70%.

[0013] In one possible implementation, the block types of the main video frame and the high-definition background frame after Gaussian blur are marked as "background block" and "uncertain block" respectively, specifically including:

[0014] Calculate SSIM by 8x8 block size, get the similarity value of each 8x8 block, preliminarily judge the type of each 8x8 block according to the SSIM value, and mark as "background block" and "uncertain block" respectively;

[0015] Traverse all 8x8 blocks, if it is an "uncertain block", get the type of the adjacent blocks in the up, down, left and right four directions, if all the adjacent blocks are "background blocks", mark the current block as "background block";

[0016] Recursive tracking is performed on all adjacent "uncertain blocks", if the total number is small or the shape is irregular, all the "uncertain blocks" are marked as "background blocks", and all the remaining "uncertain blocks" are marked as "foreground blocks";

[0017] Contract all "foreground blocks" once, if there are "background blocks" adjacent to the current block, the current block is also marked as "background block"; then dilate all "foreground blocks" again, if there are "background blocks" adjacent to the current block, the adjacent "background blocks" are also marked as "foreground blocks";

[0018] Contract all "background blocks" twice, if there are "foreground blocks" adjacent to the current block, the current block is also marked as "foreground block". Then dilate all "background blocks" twice, if there are "foreground blocks" adjacent to the current block, the adjacent "foreground blocks" are also marked as "background blocks".

[0019] In one possible implementation, calculating SSIM by 8x8 block size, getting the similarity value of each 8x8 block, preliminarily judging the type of each 8x8 block according to the SSIM value, and marking as "background block" and "uncertain block" respectively, specifically includes:

[0020] First, traverse all SSIM results according to 64x64 size, if the average SSIM value of the 64x64 block is greater than 0.8 and the minimum SSIM value of the four surrounding blocks is also greater than 0.8, mark the entire 64x64 block as "background block";

[0021] Otherwise, divide the 64x64 block into 4 32x32 blocks according to the quadtree, if the average SSIM value of each 32x32 block is greater than 0.85 and the minimum SSIM value of the four surrounding 32x32 blocks is also greater than 0.85, mark this 32x32 block as "background block";

[0022] Otherwise, divide the 32x32 block into 4 16x16 blocks according to the quadtree, if the average SSIM value of each 16x16 block is greater than 0.88 and the minimum SSIM value of the four surrounding blocks is also greater than 0.88, mark this 16x16 block as "background block";

[0023] Otherwise, the 16x16 block is divided into four 8x8 blocks according to the quadtree. If the SSIM value of each 8x8 block is greater than 0.9, the 8x8 block is marked as a "background block". Finally, all the remaining 8x8 blocks are marked as "uncertain blocks".

[0024] Traverse all 8x8 blocks. If it is an "uncertain block", obtain the types of its adjacent blocks in the four directions of up, down, left and right. If all its adjacent blocks are "background blocks", mark the current block as a "background block".

[0025] In one possible implementation, all adjacent "uncertain blocks" are recursively traced. If the total number is small or the shape is irregular, they are all marked as "background blocks", and all remaining "uncertain blocks" are all marked as "foreground blocks". Specifically, the following steps are performed: starting from the first "uncertain block", the adjacent blocks in 8 directions of up, down, left, right, upper left, upper right, lower left, and lower right are searched respectively. If the adjacent block is also an "uncertain block", the recursive search in 8 directions is continued with it as the new starting point until no adjacent block is of the type of "uncertain block", and a set of "uncertain blocks" is obtained. If the number of this set is less than 8 or the number of sets is less than 64 and the shape of the set is irregular, all of them are marked as "background blocks", and then all remaining "uncertain blocks" in the image are all marked as "foreground blocks".

[0026] The use of the present invention has the following beneficial effects: the surveillance camera is basically stationary, the background in the video is basically fixed, and only the foreground objects of people or cars are moving. The surveillance video is encoded at an extremely low bit rate, and the background frames in the video are extracted at the encoder end, and are encoded at a higher bit rate and an extremely low frame rate and then inserted into the output bit stream. At the video decoder end, the low-quality bit stream and the high-definition background frame are fused and restored to high-definition quality through the video foreground and background fusion algorithm. Without losing image quality, the video bit stream is reduced, which saves network bandwidth. At the same time, compared with complex AI artificial intelligence fusion solutions, similar effects can be achieved without a large amount of hardware resources and complex calculations, saving hardware costs on the decoder end. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the steps of the method for fusing a video stream and a high-definition background frame according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a standard bitstream input in a specific application example;

[0029] Figure 3 For Figure 2 Corresponding high-definition background image;

[0030] Figure 4for Figure 2 The image including the image content boundary obtained after canny edge detection;

[0031] Figure 5 for Figure 2 and Figure 3 The image obtained after fusion. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] Reference Figure 1 , which is a flowchart of a method for fusing a video code stream with a high-definition background frame according to an embodiment of the present invention, includes the following steps:

[0034] S10, decoding the input stream, including the standard main stream and the customized high-definition background frame stream, to obtain the current main video frame and high-definition background frame;

[0035] S20, performing canny boundary detection on the current main video frame to obtain an image including the image content boundary, which is used to assist in optimizing the fusion judgment of the main video frame and the high-definition background frame.

[0036] S30: Gaussian blur the current main video frame and the high-definition background frame to reduce high-frequency noise and details in the image, making the image smoother. Because the main video frame has a low bit rate and poor image quality, Gaussian blurring brings the two frames closer together, allowing the subsequent similarity calculation to produce a closer and more accurate result.

[0037] S40, marking the block types of the main video frame and the high-definition background frame after Gaussian blurring as "background block" and "uncertain block" respectively;

[0038] S50, comparing the boundary image obtained by boundary detection with the block-marked image. If most of the blocks within the boundary are "background blocks", all of the blocks within the boundary are marked as "background blocks". If most of the blocks within the boundary are "foreground blocks", all of the blocks within the boundary are marked as "foreground blocks". The threshold for "most" is 70% or more.

[0039] S60, index all 8x8 blocks, replace all blocks marked as "background blocks" with data in the high-definition background frame, perform smoothing filtering on the block boundaries of adjacent blocks of different types to enable smooth transition, and finally output the fused video image.

[0040] In one embodiment of the present invention, the block type marking of the main video frame and the high-definition background frame after Gaussian blurring is performed, and the marking is respectively marked as "background block" and "uncertain block". Specifically, the following steps are performed:

[0041] Calculate SSIM based on the 8x8 block size to obtain the similarity value of each 8x8 block. Preliminary determine the type of each 8x8 block based on the SSIM value and mark it as "background block" or "uncertain block" respectively.

[0042] Traverse all 8x8 blocks. If it is an "uncertain block", get the types of its adjacent blocks in the four directions of up, down, left, and right. If all its adjacent blocks are "background blocks", mark the current block as a "background block";

[0043] Recursively track all adjacent "uncertain blocks". If the total number is small or the shape is irregular, mark them all as "background blocks" and mark all remaining "uncertain blocks" as "foreground blocks";

[0044] In order to remove discrete "foreground blocks", eliminate calculation errors, and make the shape of adjacent "foreground blocks" more regular, all "foreground blocks" are shrunk. If there are "background blocks" adjacent to the current block, the current block is also marked as a "background block"; then all "foreground blocks" are expanded. If there are "background blocks" adjacent to the current block, the adjacent "background blocks" are also marked as "foreground blocks";

[0045] To remove discrete "background blocks," eliminate computational errors, and make the shape of adjacent "background blocks" more regular, all "background blocks" are contracted twice. If the current block is surrounded by "foreground blocks," the current block is also marked as a "foreground block." Then, all "background blocks" are expanded twice. If the current block is surrounded by "foreground blocks," the adjacent "foreground blocks" are also marked as "background blocks."

[0046] In one embodiment of the present invention, SSIM is calculated based on the 8x8 block size to obtain the similarity value of each 8x8 block. The type of each 8x8 block is preliminarily determined based on the SSIM value and marked as "background block" and "uncertain block". Specifically, the following steps are performed:

[0047] First, traverse all SSIM results according to the 64x64 size. If the average SSIM value of the 64x64 block is greater than 0.8 and the minimum SSIM value of its surroundings is also greater than 0.8, then mark the entire 64x64 block as a "background block";

[0048] Otherwise, the 64x64 block is divided into four 32x32 blocks according to the quadtree. If the average SSIM value of each 32x32 block is greater than 0.85 and the minimum SSIM value of the surrounding 32x32 blocks is also greater than 0.85, then the 32x32 block is marked as a "background block";

[0049] Otherwise, the 32x32 block is divided into four 16x16 blocks according to the quadtree. If the average SSIM value of each 16x16 block is greater than 0.88 and the lowest SSIM value of its surroundings is also greater than 0.88, then the 16x16 block is marked as a "background block";

[0050] Otherwise, the 16x16 block is divided into four 8x8 blocks according to the quadtree. If the SSIM value of each 8x8 block is greater than 0.9, the 8x8 block is marked as a "background block". Finally, all the remaining 8x8 blocks are marked as "uncertain blocks".

[0051] In order to remove individual discrete "uncertain blocks", all 8x8 blocks are traversed. If it is an "uncertain block", the types of its four adjacent blocks in the four directions of up, down, left and right are obtained. If all its adjacent blocks are "background blocks", the current block is marked as a "background block".

[0052] In one embodiment of the present invention, all adjacent "uncertain blocks" are recursively tracked. If the total number is small or the shape is irregular, all of them are marked as "background blocks", and all remaining "uncertain blocks" are marked as "foreground blocks". Specifically, the following steps are performed: starting from the first "uncertain block", adjacent blocks in eight directions, namely, up, down, left, right, upper left, upper right, lower left, and lower right, are searched. If the adjacent block is also an "uncertain block", the adjacent block is used as the new starting point and the recursive search in eight directions is continued until no adjacent block is of the "uncertain block" type, thereby obtaining a set of "uncertain blocks". If the number of such sets is less than 8 or the number of sets is less than 64 and the shape of the set is irregular, all of them are marked as "background blocks", and then all remaining "uncertain blocks" in the image are marked as "foreground blocks". In a specific example, if the aspect ratio of the diagonal coordinate standard difference value of the "uncertain block" is less than 0.5 or greater than 2, it can be understood as an irregular shape.

[0053] After the above method is processed, in a specific application example, Figure 2 is a schematic diagram of the input standard code stream, Figure 3 For Figure 2 Corresponding high-definition background image; Figure 4 for Figure 2 The image including the image content boundary obtained after canny edge detection; Figure 5 for Figure 2 and Figure 3 The image obtained after fusion.

[0054] It can be seen that the above-set method of fusing video code streams with high-definition background frames can improve the picture quality of the main video by detecting and replacing parts of the main video that are similar to the high-definition background frames with pictures in the high-definition background frames. The low-bitrate and low-quality main video can be reproduced and fused back to high-definition quality at a lower computational cost, which not only saves network bandwidth but also improves picture quality, and has a high application prospect.

[0055] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A method for fusing a video stream with a high-definition background frame, characterized in that: The following steps are involved: Decode the input stream, including the standard main stream and the customized high-definition background frame stream, to obtain the current main video frame and high-definition background frame; Perform canny boundary detection on the current main video frame to obtain an image including the boundary of the image content; Perform Gaussian blur on the current main video frame and the HD background frame; The SSIM is calculated for the Gaussian blurred main video frame and the high-definition background frame using 8x8 block size. The main video frame is then marked as "background block" and "uncertain block" respectively. Compare the boundary image obtained by boundary detection with the block-labeled image. If most of the blocks within the boundary are "background blocks", all of them are labeled as "background blocks". If most of the blocks within the boundary are "foreground blocks", all of them are labeled as "foreground blocks". The definition of "most" is more than 70%. Index all 8x8 blocks, replace all blocks marked as "background blocks" with data from the HD background frame, perform smoothing filtering on the block boundaries between adjacent blocks of different types, and finally output the fused video image.

2. The method for fusing a video stream with a high-definition background frame according to claim 1, wherein: The SSIM is calculated for the Gaussian blurred main video frame and the high-definition background frame using 8x8 block size. The block type of the main video frame is marked as "background block" and "uncertain block" respectively. Specifically, Calculate SSIM based on the 8x8 block size to obtain the similarity value of each 8x8 block. Preliminarily determine the type of each 8x8 block based on the SSIM value and mark it as "background block" or "uncertain block" respectively; Traverse all 8x8 blocks. If it is an "uncertain block", get the types of its adjacent blocks in the four directions of up, down, left, and right. If all its adjacent blocks are "background blocks", mark the current block as a "background block"; Recursively track all adjacent "uncertain blocks". If the total number is small or the shape is irregular, mark them all as "background blocks" and mark all remaining "uncertain blocks" as "foreground blocks"; Contract all foreground blocks. If there are "background blocks" around the current block, mark the current block as a "background block"; then expand all foreground blocks. If there are "background blocks" around the current block, mark the adjacent "background blocks" as "foreground blocks"; All "background blocks" are shrunk twice. If there are "foreground blocks" adjacent to the current block, the current block is also marked as a "foreground block"; then all "background blocks" are expanded twice. If there are "foreground blocks" adjacent to the current block, the adjacent "foreground blocks" are also marked as "background blocks".

3. The method for fusing a video stream with a high-definition background frame according to claim 2, wherein: Calculate the SSIM by 8x8 block size and obtain the similarity value of each 8x8 block. Use the SSIM value to preliminarily determine the type of each 8x8 block and mark it as "background block" or "uncertain block". Specifically, First, traverse all SSIM results according to the 64x64 size. If the average SSIM value of the 64x64 block is greater than 0.8 and the minimum SSIM value of its surroundings is also greater than 0.8, then mark the entire 64x64 block as "background block"; Otherwise, the 64x64 block is divided into four 32x32 blocks according to the quadtree. If the average SSIM value of each 32x32 block is greater than 0.85 and the minimum SSIM value of the surrounding 32x32 blocks is also greater than 0.85, then the 32x32 block is marked as a "background block"; Otherwise, the 32x32 block is divided into four 16x16 blocks using a quadtree. If the average SSIM value of each 16x16 block is greater than 0.88 and the lowest SSIM value of its surroundings is also greater than 0.88, then the 16x16 block is marked as a "background block"; Otherwise, the 16x16 block is divided into four 8x8 blocks using a quadtree. If the SSIM value of each 8x8 block is greater than 0.9, the 8x8 block is marked as a "background block". Finally, all remaining 8x8 blocks are marked as "uncertain blocks". Traverse all 8x8 blocks. If it is an "uncertain block", get the types of its adjacent blocks in the four directions of up, down, left, and right. If all its adjacent blocks are "background blocks", mark the current block as a "background block".

4. The method for fusing a video stream with a high-definition background frame according to claim 2, wherein: Recursively trace all adjacent "uncertain blocks". If the total number is small or the shape is irregular, they are all marked as "background blocks", and all remaining "uncertain blocks" are all marked as "foreground blocks". Specifically, starting from the first "uncertain block", search for adjacent blocks in 8 directions, namely, up, down, left, right, upper left, upper right, lower left, and lower right. If the adjacent block is also an "uncertain block", use it as the new starting point to continue recursively searching in 8 directions until no adjacent block is of the type "uncertain block". A set of "uncertain blocks" is obtained. If the number of this set is less than 8, or the shape of the set is irregular and the number of sets is less than 64, they are all marked as "background blocks", and then all remaining "uncertain blocks" in the image are all marked as "foreground blocks".

Images