A surveillance video encoding method based on progressive background frame generation

Abstract

The present invention provides a surveillance video encoding method based on progressively generating background frames. First, a background block detection algorithm is designed. By comparing the similarity between image blocks, background blocks and foreground blocks at the same position in a sequence are effectively distinguished. Simultaneously, the authenticity of the background blocks is further effectively judged by comparing the texture complexity of the image blocks, thereby improving the background block detection accuracy. The background block detection algorithm is then used to detect the actual background blocks in the encoded frames and to perform high-quality encoding on the detected background blocks. Finally, background frames are progressively generated by synthesizing high-quality encoded background blocks from different frames, and these frames are used as long-term reference frames for encoding reference. The method improves the quality of background frame generation, avoids the sudden increase in bit rate caused by the one-time transmission of background frames, and significantly improves the compression efficiency of surveillance video encoding.

Description

Technical Field

[0001] The present invention relates to the field of video coding and decoding, and in particular to a monitoring video coding method based on progressively generated background frames. Background Art

[0002] Thanks to advances in video technology, video services, including surveillance video, are moving towards high definition. One second of uncompressed HD video (30 frames) consumes approximately 1.5G (1920*1080*3*8*30) bits of data. Surveillance cameras operate 24 / 7, generating explosive growth in data volume. In contrast, video compression rates are increasing very slowly, following the "Moore's Law" of doubling every ten years. This gain in compression comes at a significant computational cost. Clearly, current video compression technology is failing to meet the massive demands of surveillance video encoding. Improving video compression technology and increasing the compression rate of surveillance video is a pressing issue.

[0003] Currently, video surveillance services generally use traditional encoders such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding) for compression. These encoders are general video coding standards that do not consider the characteristics of the surveillance service itself and propose targeted solutions. They achieve the purpose of video compression by removing spatial redundancy, temporal redundancy, and visual redundancy information in the video frame sequence, ignoring a special type of redundant information in surveillance videos: background redundancy. Considering the stability of the surveillance video background, the surveillance video compression method based on background reference frames is an effective means to remove its background redundant information and improve the compression rate of surveillance videos. However, the current research on surveillance video coding based on background reference frames still has many problems:

[0004] Background reference frame technology has high requirements on the quality of generated background frames, while traditional background modeling algorithms usually generate background images with poor quality, which seriously affects the improvement of surveillance video compression performance.

[0005] Most methods require encoding and transmitting high-quality background frames at one time after generating a complete background reference image, which will cause a sudden increase in bit rate and lead to network congestion and other problems.

[0006] In related research based on background frame technology, the prior art provides a method for compressing background redundant information. The method uses background modeling technology based on a mixed Gaussian model to obtain a background frame. Then, by segmenting the video frame into a foreground area and a background area, the encoding process of the background area is omitted by replacing the corresponding area of the background frame with the background area of the video frame, thereby achieving the purpose of background redundant information. Although the background frame is obtained by the mixed Gaussian background modeling technology, the quality of the background frame generated by traditional background modeling methods, including the mixed Gaussian background modeling technology, cannot meet the requirements. The generated background frames generally have problems such as foreground misclassification as background and foreground smear. The background frame is severely contaminated by the foreground and cannot maximize the compression rate of the surveillance video. In addition, the generated background frame requires a one-time high-quality encoding and transmission to the decoding end. During the transmission process, a sudden increase in bit rate will occur, leading to network congestion and other problems. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the present invention provides a surveillance video encoding method based on progressively generated background frames.

[0008] A surveillance video encoding method based on progressively generating background frames specifically comprises the following steps:

[0009] Step 1: Initialize the first I frame encoded by the encoder as a background frame and insert it into the encoder's reference frame list as a long-term background reference frame;

[0010] Step 2: Split the current image frame to be encoded into image blocks in the same way as the encoder divides CTUs (Coding Tree Units).

[0011] Step 3: Perform motion measurement on the image blocks of the segmented image frame. The motion measurement value calculation method is shown in formula (1) and formula (2):

[0012]

[0013]

[0014] Formula (1) represents the adjacent frames F T and F T-1 Perform differential operation on the Kth image block to generate a differential image block Formula (2) calculates the motion metric value of the differential image block In the above formula and Represents the video frame F T and F T-1 The pixel value at position (i, j) in the K-th image block, N represents the side length of the image block;

[0015] The position (i, j) represents the length and width of the image block. If it is a*a, then it contains a*a coordinates, and the position (i, j) is the horizontal and vertical coordinates of the corresponding image block;

[0016] Step 4: After obtaining the motion metric value based on step 3, the motion metric value is compared with a threshold. If the motion metric value is less than the threshold, it is determined to be a potential background block; and the image block codebook model is used to determine whether the potential background block is a background block.

[0017] The threshold is a threshold of the motion metric value;

[0018] The image block codebook model determines whether a potential background block is a background block as follows:

[0019] Step A: Maintain a codebook for each image block position of the video frame. The codebook contains a number of code elements, which are a set of candidate background blocks. The codebook is set to empty during the initialization phase.

[0020] Step B: Match the potential background block to be processed with the code elements in the codebook corresponding to the position one by one. If the match is successful, the potential background block is placed in the code element. If the match fails or the codebook is empty, a new code element is created and the potential background block is placed in the new code element.

[0021] The perceptual hash algorithm is used to calculate the hash value of the image block, and the Hamming distance of the hash values between the image blocks is used as the similarity measure between the image blocks. The code element with the minimum Hamming distance to the potential background block to be processed is found in the codebook. If the Hamming distance between the hash value of the image block in the code element and the hash value of the potential background block to be processed is less than a threshold D, the match is successful and the potential background block to be processed is placed in the code element. Otherwise, the match fails, a new code element is created and the potential background block to be processed is placed in the new code element; the threshold D is the threshold of the Hamming distance between the potential background block and the perceptual hash value of the image block in the code element;

[0022] Step C: Perform a symbol analysis on all symbols in the codebook every N frames. The symbols that meet the judgment criteria will be judged as background symbols, and the remaining redundant symbols will be cleared;

[0023] Based on the number of image blocks contained in each codeword in the codebook, find the codeword containing the largest number of image blocks in the codebook and compare it with a threshold value Thf. If the number of image blocks in the codeword is greater than the threshold value Thf, the codeword is determined to be a potential background codeword; the threshold value Thf is the threshold value of the number of image blocks contained in the codeword;

[0024] Alternatively, based on the cumulative value of the time intervals between the image blocks included in each symbol in the codebook, the symbol with the largest cumulative value of the time intervals between the image blocks in the codebook is found and the cumulative value is compared with a threshold value T. If the cumulative value is greater than the threshold value, the symbol is determined to be a potential background symbol; the threshold value T is the threshold value of the cumulative value of the time intervals between the image blocks included in the symbol;

[0025] If there is no background symbol in the codebook, the potential background symbol is directly determined as the background symbol. If there is a background symbol in the codebook, the texture complexity of the potential background symbol and the image block contained in the background symbol are calculated respectively. If the texture complexity of the image block in the potential background symbol is lower than that of the background symbol, the potential background symbol is set as the new background symbol and the original background symbol is cleared. Otherwise, the potential background symbol is cleared.

[0026] Step D: When background symbols already exist in the codebook, the potential background block that successfully matches the background symbol can be determined as a background block. However, in order to prevent repeated high-quality encoding of the same background block during the encoding process and waste of bitstream, only the first potential background block that successfully matches the same background symbol is determined as a background block during the background block detection process;

[0027] Step 5: Add a marking symbol to the CTU structure of the encoder to record whether the image block is a background block. If it is a background block, determine the prediction reference value of the background block and adjust the encoding parameters according to the reference value. If it is a non-background block, use the default parameters for encoding;

[0028] The reference value represents the importance level of the background block to the subsequent coded frame when performing inter-frame prediction;

[0029] The step of determining the reference value includes:

[0030] Counting the Skip coding mode ratios of image blocks in the same position as the background block from the first frame to the frame containing the background block, and determining the prediction reference value of the image block based on the counted Skip coding mode ratios of the image blocks. The higher the Skip coding mode ratio of the image block, the greater the reference value of the background block.

[0031] Step 6: After the codec reconstructs the coded frame, it extracts the background blocks and replaces the corresponding areas of the long-term background reference frame to provide a reference for subsequent coded frames;

[0032] Repeat steps 2 to 6 to encode subsequent image frames until every frame in the video is encoded.

[0033] Beneficial technical effects of the present invention:

[0034] In the surveillance video encoding method based on progressive generation of background reference frames designed by the present invention, background frames are gradually generated based on image blocks, and the generation process of background reference images spans dozens to hundreds of frames. It can improve the compression rate of surveillance video while effectively smoothing the code stream to avoid the problem of sudden increase in code rate caused by the one-time transmission of high-quality background reference frames in traditional optimization methods.

[0035] Since background images generated by traditional background modeling technology generally have the problems of foreground smearing and foreground misjudgment as background, the quality of the generated background images is low. The present invention designs an "image block codebook model" algorithm to detect clean and real background blocks in video frames, and generates high-quality background frames by synthesizing the detected background blocks in different frames. This method avoids the foreground smear problem caused by pixel statistical modeling in traditional background modeling technology. At the same time, in order to address the phenomenon of foreground misjudgment as background in traditional background modeling methods caused by phenomena such as sudden foreground stop, the present invention introduces texture complexity calculation to analyze whether the potential background is the real background, thereby reducing the occurrence of foreground misjudgment as background, improving the overall quality of the background image, and further enhancing the compression performance gain brought by the background image as a reference frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Framework diagram of a surveillance video encoding method based on progressive generation of background reference frames according to an embodiment of the present invention

[0037] Figure 2 Comparison of rate-distortion curve performance of surveillance video compression according to an embodiment of the present invention

[0038] Figure 3 Comparison diagram of bits required per frame for monitoring video encoding according to an embodiment of the present invention

[0039] Figure 4 Comparison of subjective quality of background images generated in Classover sequences by the embodiment of the present invention and the hybrid high background modeling method

[0040] Figure 5 Comparison of subjective quality of background images generated in Campus sequence by the embodiment of the present invention and the hybrid high background modeling method

[0041] Figure 6 Comparison of subjective quality of background images generated in Crossroad sequence by the embodiment of the present invention and the hybrid high background modeling method DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0043] Figure 1This is the overall framework diagram of the monitoring video encoding method based on progressive generation of background reference frames proposed in the technical solution of the present invention. The gray blocks are the modules newly added to the HEVC encoding framework of the present invention, such as Figure 1 As shown, the surveillance video encoding process proposed by the present invention is as follows:

[0044] A surveillance video encoding method based on progressively generating background frames specifically comprises the following steps:

[0045] Step 1: Initialize the first I frame encoded by the HEVC encoder as a background frame and insert it into the HEVC reference frame list as a long-term background reference frame;

[0046] Step 2: Split the current image frame to be encoded into image blocks in the same way as the HEVC encoder divides CTUs (Coding Tree Units);

[0047] Step 3: Perform motion measurement on the image blocks of the segmented video frame. The motion measurement value calculation method is shown in formula (1) and formula (2):

[0048]

[0049]

[0050] Formula (1) represents the adjacent frames F T and F T-1 Perform differential operation on the Kth image block to generate a differential image block Formula (2) calculates the motion metric value of the differential image block In the above formula and Represents the video frame F T and F T-1 The pixel value at position (i, j) in the K-th image block, N represents the side length of the image block;

[0051] The position (i, j) represents the length and width of the image block. If it is a*a, then it contains a*a coordinates, and the position (i, j) is the horizontal and vertical coordinates of the corresponding image block;

[0052] Step 4: After obtaining the motion metric value based on step 3, the motion metric value is compared with a threshold. If the motion metric value is less than the threshold, it is determined to be a potential background block; and the image block codebook model is used to determine whether the potential background block is a background block.

[0053] The threshold is a threshold of the motion metric value;

[0054] The accuracy of the background block detection method in step 4 above is the key to improving the quality of the generated background frame and thus improving the compression performance of the surveillance video. To this end, this paper designs an "image block codebook model" algorithm to detect background blocks. Its specific operation process is as follows:

[0055] Step A: Maintain a codebook for each image block position of the video frame. The codebook contains a number of code elements, which are a set of candidate background blocks. The codebook is set to empty during the initialization phase.

[0056] Step B: Match the potential background block to be processed with the code elements in the codebook corresponding to the position one by one. If the match is successful, the potential background block is placed in the code element. If the match fails or the codebook is empty, a new code element is created and the potential background block is placed in the new code element.

[0057] The perceptual hash algorithm is used to calculate the hash value of the image block, and the Hamming distance of the hash values between the image blocks is used as the similarity measure between the image blocks. The code element with the minimum Hamming distance to the potential background block to be processed is found in the codebook. If the Hamming distance between the hash value of the image block in the code element and the hash value of the potential background block to be processed is less than a threshold D, the match is successful and the potential background block to be processed is placed in the code element. Otherwise, the match fails, a new code element is created and the potential background block to be processed is placed in the new code element; the threshold D is the threshold of the Hamming distance between the potential background block and the perceptual hash value of the image block in the code element;

[0058] Step C: Perform a symbol analysis on all symbols in the codebook every N frames. The symbols that meet the judgment criteria will be judged as background symbols, and the remaining redundant symbols will be cleared;

[0059] Based on the number of image blocks contained in each codeword in the codebook, find the codeword containing the largest number of image blocks in the codebook and compare it with a threshold value Thf. If the number of image blocks in the codeword is greater than the threshold value Thf, the codeword is determined to be a potential background codeword; the threshold value Thf is the threshold value of the number of image blocks contained in the codeword;

[0060] Alternatively, based on the cumulative value of the time intervals between the image blocks included in each symbol in the codebook, the symbol with the largest cumulative value of the time intervals between the image blocks in the codebook is found and the cumulative value is compared with a threshold value T. If the cumulative value is greater than the threshold value, the symbol is determined to be a potential background symbol; the threshold value T is the threshold value of the cumulative value of the time intervals between the image blocks included in the symbol;

[0061] If there is no background symbol in the codebook, the potential background symbol is directly determined as the background symbol. If there is a background symbol in the codebook, the texture complexity of the potential background symbol and the image block contained in the background symbol are calculated respectively. If the texture complexity of the image block in the potential background symbol is lower than that of the background symbol, the potential background symbol is set as the new background symbol and the original background symbol is cleared. Otherwise, the potential background symbol is cleared.

[0062] Step D: When background symbols already exist in the codebook, the potential background block that successfully matches the background symbol can be determined as a background block. However, in order to prevent repeated high-quality encoding of the same background block during the encoding process and waste of bitstream, only the first potential background block that successfully matches the same background symbol is determined as a background block during the background block detection process;

[0063] Step 5: Add a marking symbol to the CTU structure of the encoder to record whether the image block is a background block. If it is a background block, determine the prediction reference value of the background block and adjust the encoding parameters according to the reference value. If it is a non-background block, use the default parameters for encoding;

[0064] The reference value represents the importance level of the background block to the subsequent coded frame when performing inter-frame prediction;

[0065] The step of determining the reference value includes:

[0066] Counting the Skip coding mode ratios of image blocks in the same position as the background block from the first frame to the frame containing the background block, and determining the prediction reference value of the image block based on the counted Skip coding mode ratios of the image blocks. The higher the Skip coding mode ratio of the image block, the greater the reference value of the background block.

[0067] Step 6: After the codec reconstructs the coded frame, it extracts the background block and replaces the corresponding area of the long-term background reference frame to provide a reference for subsequent coded frames.

[0068] Repeat steps 2 to 6 to encode subsequent video frames until every frame in the video is encoded.

[0069] To demonstrate the effectiveness of the surveillance video encoding algorithm in the present invention, the encoding method of the present invention was integrated into the HEVC standard encoding software HM16.20 for verification. The low-delay configuration file (encoder_lowdelay_P_main.cfg) was used as the default configuration. At the same time, some sequences from the public surveillance video dataset PKU-SVD-A released by Peking University were selected as test sequences. The specific information of the sequences is shown in Table 1:

[0070] Table 1 Test sequence information

[0071]

[0072] Table 2 shows the BD-rate comparison of our method with HM16.20. It shows that our method achieves an average coding gain of 17.89% on the Y component. This means that, while achieving the same video quality, our method saves an average of 17.89% bitrate compared to the standard test software. The largest Y-component coding gain is achieved in the Overbridge sequence (29.54%), while the smallest is 8.37% for the more complex Crossroad sequence. Overall, all test sequences achieve varying degrees of positive Y-component gain, demonstrating the effectiveness of our method. The Y-component gain is generally considered the most representative component for coding performance evaluation, with the gains of other components used as a reference. Our method also achieves impressive gains for the U, V, and YUV components, reaching 76.43%, 76.22%, and 26.68%, respectively.

[0073] Table 2 BD-rate gain relative to HM16.20

[0074]

[0075] Figure 2 The rate-distortion curve of the above test video sequence after encoding is given, which is drawn by the bit rate and the PSNR value of the Y component. Figure 2 It can be seen that the coding performance of the proposed method at low bit rates is significantly better than that of the HM1620 standard software. For example, when the Overbridge sequence bit rate is 130kbps, the PSNR-Y value of the proposed method is 3.5dB higher than that of HM16.20.

[0076] like Figure 3 Figure 2 shows a comparison of the per-frame code size changes for the first 150 frames of the Crossroad and Overbridge sequences using the proposed method and the HM1620 standard software. The figure shows that while the proposed method consumes slightly more bits per frame than the HM1620 standard software during background synthesis, the overall bit rate is similar, with no significant bit rate fluctuations.

[0077] At the same time, in order to prove the effectiveness of the "image block codebook model" algorithm proposed in this invention, a subjective quality comparison is made between the background image generated by the algorithm and the background image generated by the mixture Gaussian model. Figure 4 、 5 , 6, Figure 4 、 56 are comparison diagrams of the background images generated by the mixed Gaussian background modeling method and the method of the present invention at the 100th, 200th and 300th frames in three test sequences. Since the background image generation process of the present invention is progressive, the areas where the real background has not been generated during the test are represented by black image blocks. The last column in the figure shows the details of some areas of the background image. Figure 4 、 5 6, it can be seen that there is no smearing or misjudgment in the background image generated by the method of the present invention, and the quality of the generated background frame is higher than that of the mixed Gaussian background modeling method.

Claims

1. A surveillance video encoding method based on progressive generation of background frames, characterized in that: The specific steps include: Step 1: Initialize the first I frame encoded by the encoder as a background frame and insert it into the encoder's reference frame list as a long-term background reference frame; Step 2: Divide the current image frame to be encoded into image blocks in the same way as the encoder divides CTU (CodingTree Unit); Step 3: Perform motion measurement on the image blocks of the segmented image frame; Step 4: After obtaining the motion metric value based on step 3, the motion metric value is compared with a threshold value. If the motion metric value is less than the threshold value, it is determined to be a potential background block; the image block codebook model is used to determine whether the potential background block is a background block; the threshold value is the threshold value of the motion metric value; The specific method of using the image block codebook model to determine whether a potential background block is a background block is as follows: Step A: Maintain a codebook for each image block position of the video frame. The codebook contains a number of code elements, which are a set of candidate background blocks. The codebook is set to empty during the initialization phase. Step B: Match the potential background block to be processed with the code elements in the codebook corresponding to the position one by one. If the match is successful, the potential background block is placed in the code element. If the match fails or the codebook is empty, a new code element is created and the potential background block is placed in the new code element. Step C: Perform a symbol analysis on all symbols in the codebook every N frames. The symbols that meet the judgment criteria will be judged as background symbols, and the remaining redundant symbols will be cleared; Step D: When background symbols already exist in the codebook, the potential background block that successfully matches the background symbol can be determined as a background block. However, in order to prevent repeated high-quality encoding of the same background block during the encoding process and waste of bitstream, only the first potential background block that successfully matches the same background symbol is determined as a background block during the background block detection process; Step 5: Add a marking symbol to the CTU structure of the encoder to record whether the image block is a background block. If it is a background block, determine the prediction reference value of the background block and adjust the encoding parameters according to the reference value. If it is a non-background block, use the default parameters for encoding; Step 6: After the codec reconstructs the coded frame, it extracts the background blocks and replaces the corresponding areas of the long-term background reference frame to provide a reference for subsequent coded frames; Repeat steps 2 to 6 to encode subsequent image frames until every frame in the video is encoded.

2. The method for encoding surveillance video based on progressively generating background frames according to claim 1, characterized in that: Step 3: The motion metric value calculation method is shown in Equation 1 and Equation 2: Wherein formula 1 represents the adjacent frame F T and F T-1 Perform differential operation on the Kth image block to generate a differential image block Formula 2 calculates the motion metric value of the differential image block In the above formula and Represents the video frame F T and F T-1 The pixel value at position (i, j) in the Kth image block in , where N represents the side length of the image block.

3. The method for encoding surveillance video based on progressively generating background frames according to claim 1, characterized in that: Step B is specifically as follows: The perceptual hash algorithm is used to calculate the hash value of the image block, and the Hamming distance of the hash values between the image blocks is used as the similarity measure between the image blocks. The code element with the minimum Hamming distance to the potential background block to be processed is found in the codebook. If the Hamming distance between the hash value of the image block in the code element and the hash value of the potential background block to be processed is less than a threshold D, the match is successful and the potential background block to be processed is placed in the code element. Otherwise, the match fails, a new code element is created and the potential background block to be processed is placed in the new code element; the threshold D is the threshold of the Hamming distance between the potential background block and the perceptual hash value of the image block in the code element.

4. The method for encoding surveillance video based on progressively generating background frames according to claim 1, characterized in that: Step C is specifically as follows: Based on the number of image blocks contained in each codeword in the codebook, find the codeword containing the largest number of image blocks in the codebook and compare it with a threshold value Thf. If the number of image blocks in the codeword is greater than the threshold value Thf, the codeword is determined to be a potential background codeword; the threshold value Thf is the threshold value of the number of image blocks contained in the codeword; Alternatively, based on the cumulative value of the time intervals between the image blocks included in each symbol in the codebook, the symbol with the largest cumulative value of the time intervals between the image blocks in the codebook is found and the cumulative value is compared with a threshold value T. If the cumulative value is greater than the threshold value, the symbol is determined to be a potential background symbol; the threshold value T is the threshold value of the cumulative value of the time intervals between the image blocks included in the symbol; If there is no background codeword in the codebook, the potential background codeword is directly determined as the background codeword. If there is a background codeword in the codebook, the texture complexity of the potential background codeword and the image block contained in the background codeword are calculated respectively. If the texture complexity of the image block in the potential background codeword is lower than that of the background codeword, the potential background codeword is set as the new background codeword and the original background codeword is cleared. Otherwise, the potential background codeword is cleared.

5. The method for encoding surveillance video based on progressively generating background frames according to claim 1, characterized in that: The reference value in step 5 represents the importance level of the background block to the subsequent coded frame when performing inter-frame prediction.

6. The method for encoding surveillance video based on progressively generating background frames according to claim 1, characterized in that: The reference value determined in step 5 is specifically: The skip coding mode ratios of the image blocks at the same position as the background block in the frame from the first frame to the frame where the background block is located are counted, and the prediction reference value of the image block is determined based on the counted skip coding mode ratios of the image blocks. The higher the skip coding mode ratio of the image block, the greater the reference value of the background block.

Images