A multi-objective coding method for surveillance videos

Abstract

The present invention belongs to the field of surveillance video compression, and particularly relates to a multi-target encoding method for surveillance video. The method comprises the following steps: (1) constructing a background frame sequence update module to reconstruct background image frames to obtain reconstructed background frames; (2) constructing a multi-target video compression module to continuously detect and track specific semantic reference objects using a semantic target detection network, converting the semantic reference objects into compact feature representations using a feature extraction network, and encoding the semantic reference objects and the compact features for transmission; and (3) constructing a reconstructed video frame module to combine the reconstructed foreground and reconstructed background frames #imgabs0# to generate a reconstructed frame #imgabs1#. The present invention is aimed at surveillance video data transmission under multi-target conditions, and can transmit surveillance video data information while saving bit rate.

Description

Technical Field

[0001] The invention belongs to the field of monitoring video compression, and in particular relates to a multi-objective encoding method oriented to monitoring videos. Background Art

[0002] With the rapid development of information technology and the integration of various smart devices into daily life, surveillance video data has experienced explosive growth, driven by the successive developments of smart security, smart cities, and the internet industry. To effectively compress video data, the industry has proposed a series of video compression standards and algorithms (such as H.264 / AVC, HEVC, and VVC).

[0003] Surveillance video content typically consists of a continuous sequence of videos shot with a fixed lens, covering a specific indoor or outdoor scene. The key content in the video is often a person or object that appears within the scene and whose position constantly changes. Due to surveillance requirements, the length of the video and the storage space required inevitably increase over time. Therefore, methods for removing redundancy from surveillance video content can effectively save bitrates while reducing transmission bandwidth and storage space requirements.

[0004] In recent years, researchers have made initial explorations in the field of surveillance video compression. For example, a method for compressing apron surveillance videos based on object detection has been proposed. This method divides the apron surveillance video into two parts: a background image and a target image (moving and still), each of which is stored on disk. Similar methods use a Gaussian mixture model to separate the video content into background and foreground images, encoding the foreground image with a traditional encoder while caching the background image on the client. These methods have limited effectiveness and fail to fully utilize the characteristics of surveillance video. To fully utilize the characteristics of surveillance video, a foreground-background parallel transmission method has also been proposed. This scheme uses background information between adjacent frames and utilizes optical flow to estimate the motion of the foreground image. Different from this method, there is also an end-to-end surveillance video compression method based on optical flow prediction and motion compensation networks. Although this method improves the compression efficiency of surveillance video, the diversity of foreground images makes it difficult to compress the foreground image using limited feature compression.

[0005] Existing surveillance video encoding methods have undergone extensive research and have achieved significant performance improvements. However, traditional encoding standards and deep learning-based solutions typically optimize for pixel-level fidelity, failing to fully utilize the inherent information within the video signal during the compression process. Therefore, this paper proposes multi-target surveillance video data compression to reduce data size. Summary of the Invention

[0006] Technical issues solved:

[0007] In traditional video coding schemes or those based on machine learning, kernel density estimation is often used to model backgrounds, estimating the probability density of a given pixel. This method typically performs well in scenes with large lighting variations. However, this method is computationally complex and therefore inefficient. Based on the background characteristics of video surveillance, this paper proposes a background update framework to update the background of surveillance videos.

[0008] The existing surveillance video encoding compression method does not take into account the temporal correlation between adjacent frames. Since each frame can be divided into foreground and background, how to make good use of the characteristics of the foreground and improve compression efficiency is the second technical problem to be solved by the present invention. The present invention uses compact features to represent the extracted foreground objects to improve compression efficiency.

[0009] The reconstructed video frame is obtained by re-synthesizing the separated character foreground and background. The present invention also proposes a solution to the problem of excessive boundary inconsistency during the reconstruction process.

[0010] Technical solution:

[0011] The present invention provides a multi-objective encoding method for surveillance video, comprising the following steps:

[0012] (1) Constructing background frame sequence update module

[0013] Use Vibe algorithm to obtain the background template B in the video frame;

[0014] Get the foreground mask frame (f1, f2...f t-1 、f t ), the background template B and the foreground mask frame (f1, f2...f t-1 、f t ) are subtracted to obtain the background template sequence (B1′, B2′…B t-1 ′、B t ′);

[0015] The source video frame sequence and the foreground mask frame (f1, f2...f t-1 、f t ) are subtracted to obtain the source video frame background sequence (b′1, b′2…b′ t-1 , b′ t );

[0016] Then (B1′, B2′…B t-1 ′、B t′) and (b′1, b′2…b′ t-1 , b′ t ) to compare the values ​​on SSIM; if B′ t and b′ t The SSIM value between is greater than the threshold γ, then B′ t and b′ t Subtract to get the residual r at time t t , residual r t The residual compression module compresses the reconstructed residual, which is recorded as The background image frame at time t is reconstructed according to the following formula:

[0017]

[0018] Among them is the reconstructed background frame at time t, B′ t It represents the background frame obtained by the background template at time t;

[0019] (2) Constructing a multi-objective video compression module

[0020] First, a semantic object detection network is used to continuously detect and track the semantic reference object m in the video; the semantic reference object m refers to the human target appearing in the video frame;

[0021] The semantic reference object m is converted into a compact feature representation (x0, x1, x2...) using a feature extraction network. The compact feature representation includes three types: human key points, limb masks, and side information.

[0022] Then the semantic reference object m and the compact features (x0, x1, x2…) are transmitted and encoded, and decoded using the decoder to obtain the decoded semantic reference object, which is recorded as and compact feature sequences Then and compact feature sequences Together, we reconstruct specific semantic targets through the human posture transfer network and obtain the reconstruction prospects l t ;

[0023] When the semantic reference object is occluded and a sufficient number of compact feature representations cannot be obtained, resulting in the human pose transfer network being unable to generate accurate human targets, the improved deep video compression DCVC is used to compress the human targets;

[0024] (3) Constructing a video frame reconstruction module

[0025] The reconstructed video frame module is used to reconstruct the foreground and background frames Combine to generate reconstructed frame

[0026] First, according to the foreground mask frame (f1, f2, ... f t-1 、f t ) Determine the foreground in the reconstructed background frame The position in the middle, and then use the reconstructed foreground l t Reconstructed background frame The corresponding position in is replaced to obtain a reconstructed frame a, wherein the reconstructed frame a has a transition boundary generated by the replacement, and the reconstructed frame a is input into the reconstruction network to eliminate the transition boundary to obtain a reconstructed frame

[0027] Preferably, the method of obtaining the background template B in the video frame by using the Vibe algorithm comprises the following steps:

[0028] The Vibe algorithm uses a sequence of pixel values ​​of a point to describe the background model of each pixel;

[0029] M(x)={v1,v2,…,v N}

[0030] Where M(x) is the background model, v i is the sampling value, N is the length of the point sequence;

[0031] The pixel value v(x) of the input point and the sample value v in M(x) i In contrast, if the distance in the color space is less than the preset parameter D t , we think that v(x) and v i resemblance;

[0032] v(x)≈v i if||v(x)-v i ||<D t

[0033] After traversing the entire sequence, count the number of sampling points similar to v(x). If the number is greater than the threshold α, v(x) is the background, otherwise it is the foreground.

[0034] Preferably, the semantic object detection network is any one of YOLO and RCNN;

[0035] The Repulison loss function is introduced into the semantic target detection network to improve the tracking effect under occlusion. The formula of the Repulison loss function is:

[0036] L=L Attr +α*L RepGT +β*L RepBox

[0037] Among them, L AttrIt is an attraction item, which requires the prediction box to be close to its specified target; L RepGT and L RepBox is a repulsive term that requires the prediction box to be away from other surrounding groundtruth objects and other prediction boxes that are different from the specified target. The coefficients α and β act as weights to balance the auxiliary loss, and * is multiplication.

[0038] Preferably, a structure-preserving motion estimation network is introduced into the improved deep video compression DCVC;

[0039] The structure-preserving motion estimation network includes motion field computation Δ main , auxiliary motion field calculation Δ aux and the sports field integrate three parts.

[0040]

[0041]

[0042] Among them, C(·,·) represents the concatenation operation, Conv 3×3 represents the convolution operation, represents the element-by-element multiplication of the matrix, and ReLU is the activation function; F t is the input frame, F t-1 is the previous frame, The previous frame to be decoded.

[0043] Preferably, the reconstruction network consists of three convolutional layers, three residual blocks and three transposed convolutional layers; the three convolutional layers are used to capture the local features of the image in the reconstructed frame a, and the three residual blocks are used to learn the foreground l obtained by the semantic target detection network. t and reconstructed background frame The nonlinear relationship between them is achieved by using three transposed convolutional layers for upsampling to restore the spatial resolution of the image.

[0044] Beneficial effects of the present invention:

[0045] The present invention is for monitoring video data transmission under a multi-target situation, and can transmit data information of the monitoring video under the premise of saving bit rate.

[0046] 1. Background update module

[0047] Although the background of surveillance videos is visually static, it is actually subject to changes in lighting, weather, and time, causing fluctuations in brightness and color. To maintain the consistency of the extracted background with the original video background and improve compression performance, the present invention proposes a background update module.

[0048] Structural Similarity (SSIM) is a metric used to measure the similarity between two images and evaluate video quality. Therefore, the present invention utilizes SSIM to reconstruct the background. An initial background template is selected and then compared with the original video background for SSIM. If the SSIM is greater than a threshold γ, the original background and the initial background template are subtracted to obtain a residual. This residual is then added to the initial background template to obtain a new background template.

[0049] 2. Video compression module under multiple targets

[0050] Pixels in video sequences are highly temporally correlated. Traditional coding frameworks remove temporal redundancy in surveillance videos through block motion compensation, but fail to account for the inherent structural information within the video. Therefore, to fully leverage this structural information, this paper introduces an object detection module to detect and crop human subjects. The introduction of a conditional generative network allows the generation of video encoding sequences by simply compressing semantic reference objects and compact feature sequences, improving encoding performance.

[0051] However, due to the interaction between the objects, a series of occlusion problems will occur. Therefore, relying solely on the object detection module cannot detect human objects. Therefore, the present invention introduces a loss function in the detection module to improve detection performance. Due to the occlusion of this part of the human body, it is impossible to generate human sequences through the generative network. For this part, a classic deep learning-based encoding method similar to Deep Video Compression (DCVC) is used to compress the human objects.

[0052] 3. Rough to fine video frame reconstruction module

[0053] Due to the fusion of foreground and background, there will be excessive boundary blurring and distortion in the reconstructed frame. Therefore, the present invention introduces a reconstruction network through deep learning to improve the quality of the reconstructed video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the specific embodiments or the description of the prior art. Hereinafter, some specific embodiments of the present invention will be described in detail in an illustrative and non-limiting manner with reference to the drawings. The same reference numerals in the drawings indicate the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0055] Figure 1 The overall flow chart is shown in Figure 2.

[0056] Figure 2 Schematic diagram of the background update module.

[0057] Figure 3 Schematic diagram of the reconstructed frame network. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] In order to further improve the compression efficiency of surveillance videos, this paper uses background updating, semantic target detection, human posture transfer network, deep learning coding and other means to optimize the performance of video compression coding. The multi-target coding optimization process is divided into the target detection stage and the target coding stage, in which the target detection stage mainly performs target recognition and cropping. The overall process is shown in Figure (1).

[0060] (1) Background frame sequence update

[0061] Although the background of a surveillance video is visually static, it can actually fluctuate in brightness and color due to changes in lighting, weather, and time. Therefore, the present invention first utilizes the Vibe algorithm to obtain the background in the video frame. The Vibe algorithm uses a sequence of points (pixel values) to describe the background model of each pixel.

[0062] M(x)={v1,v2,…,v N} (1)

[0063] Where M(x) is the background model, v i is the sampling value, and N is the length of the point sequence. The pixel value v(x) of the input point and the sampling value v in M(x) i In contrast, if the distance in the color space is less than the specified parameter D t , we think that v(x) and v i resemblance.

[0064] v(x)≈v i if||v(x)-v i ||<D t (2)

[0065] After traversing the entire sequence, the number of sampling points similar to v(x) is counted. If the number is greater than the threshold α, v(x) is considered to be the background, otherwise it is the foreground.

[0066] The present invention first obtains the foreground mask frame (f1, f2...f t-1 、f t ) and a background template B. Through the background template and foreground mask frame (f1, f2...f t-1 、ft ) are subtracted to obtain a new background template sequence (B1′, B2′…B t-1 ′、B t ′). The original video frame sequence is also the same as the foreground mask frame (f1, f2...f t-1 、f t ) to obtain the source video frame background sequence (b′1, b′2…b′ t-1 , b′ t ). Then the background template sequence and the background sequence are compared by SSIM. If B′ t and b′ t If the SSIM value between them is greater than the threshold γ, the two are subtracted to obtain the residual r t , residual r t Compressed by the residual compression module, the reconstructed residual is Therefore the background image at time t will be reconstructed as

[0067]

[0068] Among them is the reconstructed background frame at time t, B′ t It represents the background frame obtained by the background template at time t;

[0069] (2) Multi-target video compression module

[0070] In the video sequence, the present invention first continuously detects and tracks a specific semantic object m through a semantic target detection network (such as YOLO, RCNN). While selecting and encoding the semantic reference object m, the feature extraction network is used to convert the semantic object into a compact feature representation (such as human key points, limb masks, and side information). The semantic reference object m and the compact features (x0, x1, x2...) are then transmitted and decoded, and the semantic reference object obtained by the decoder is recorded as and compact feature sequences Then the semantic reference object containing complete appearance information and compact feature sequences Together, we reconstruct specific semantic targets through a human pose transfer network. This network is based on the concept of Generative Adversarial Networks (GANs). Specifically, given an original image and a new pose represented by key points, the network's goal is to convert the original image's pose into the target pose. This invention generates the remaining foreground of the person in the video frame by inputting a reference image of the person into the network.

[0071] However, because objects interact with each other, partial occlusion may occur. Because of this occlusion, the semantic object detection network cannot continuously track a specific semantic object. Therefore, the present invention introduces the Repulison loss function into the object detection network to improve the tracking effect under occlusion. The formula of the Repulison loss function is:

[0072] L=L Attr +α*L RepGT +β*L RepBox (4)

[0073] Among them, L Attr It is an attraction item, which requires the prediction box to be close to its specified target; L RepGT and L RepBox is an exclusion term, which requires the prediction box to be away from other surrounding groundtruth objects and prediction boxes that are different from other specified targets. The coefficients α and β act as weights to balance the auxiliary losses, and * is multiplication. Due to the occlusion of the tracked semantic objects, a sufficient number of key points cannot be obtained, so the human posture transfer network cannot generate accurate human targets. For this part of the human targets, the present invention chooses to use improved deep video compression (DCVC) to compress human targets. The DCVC algorithm learns in the time domain and generates contextual features. The context features are used as conditional input to help the encoding and decoding of the current frame.

[0074] The present invention introduces a structure-preserving motion estimation network into the DCVC algorithm to improve coding efficiency. The network includes motion field calculation Δ main , auxiliary motion field calculation Δ aux and the sports field integrate three parts.

[0075]

[0076]

[0077] Among them, C(·,·) represents the concatenation operation, Conv 3×3 represents the convolution operation, represents the element-wise multiplication of matrices (also called Hadamard product), and ReLU is the activation function. The generation of these two motion fields involves the input frame F t and decoded previous frame or the current frame F t and the previous frame F t-1 The convolution operation.

[0078] (3) Rough to fine reconstruction video frame module

[0079] The function of the video frame reconstruction module is to reconstruct the foreground l t Background of reconstruction Combined to generate a reconstructed frame The specific operation is based on the foreground mask frame (f1, f2...f t-1 、f t ) Determine the corresponding position of the foreground in the background in the video frame, and use the reconstructed foreground l in the reconstructed frame t Replace the reconstructed background at the corresponding position However, due to the fusion of foreground and background, there will be excessive boundary inconsistencies in the reconstructed frame a.

[0080] In order to improve the quality of the reconstructed frame, a reconstruction network is introduced, as shown in Figure (3). This network is designed to eliminate the excessive boundaries between the foreground and background by learning higher-level feature representations, making them more harmonious and consistent in the reconstructed frame. Specifically, the reconstruction network consists of three convolutional layers, three residual blocks and three transposed convolutional layers. The convolutional layer is used to capture local features in the image, the residual block helps to learn the nonlinear relationship between the foreground and background, and the transposed convolutional layer is used for upsampling to restore the spatial resolution of the image. By introducing the reconstruction network, the reconstructed frame a can be effectively enhanced, improving the fusion effect between the foreground and background, making it more consistent with the visual perception of the human eye. The above is only a partial specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any changes or replacements that can be easily thought of by any person familiar with the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-objective encoding method for surveillance video, characterized in that: The following steps are involved: (1) Constructing background frame sequence update module Use Vibe algorithm to obtain the background template B in the video frame; Get the foreground mask frame (f1, f2...f t-1 、f t ), the background template B and the foreground mask frame (f1, f2...f t-1 、f t ) are subtracted to obtain the background template sequence (B1′, B2′…B t-1 ′、B t ′); The source video frame sequence and the foreground mask frame (f1, f2...f t-1 、f t ) are subtracted to obtain the source video frame background sequence (b′1, b′2…b′ t-1 , b′ t ); Then (B1′, B2′…B t-1 ′、B t ′) and (b′1, b′2…b′ t-1 , b′ t ) to compare the values ​​on SSIM; if B′ t and b′ t The SSIM value between is greater than the threshold γ, then B′ t and b′ t Subtract to get the residual r at time t t , residual r t The residual compression module compresses the reconstructed residual, which is recorded as The background image frame at time t is reconstructed according to the following formula: Among them is the reconstructed background frame at time t, B′ t It represents the background frame obtained by the background template at time t; (2) Constructing a multi-objective video compression module First, the semantic object detection network is used to continuously detect and track the semantic reference object m in the video; The semantic reference object m refers to the human target appearing in the video frame; The semantic reference object m is converted into a compact feature representation (x0, x1, x2...) using a feature extraction network. The compact feature representation includes three types: human key points, limb masks, and side information. Then the semantic reference object m and the compact features (x0, x1, x2…) are transmitted and encoded, and decoded using the decoder to obtain the decoded semantic reference object, which is recorded as and compact feature sequences Then and compact feature sequences Together, we reconstruct specific semantic targets through the human posture transfer network and obtain the reconstruction prospects l t ; When the semantic reference object is occluded and a sufficient number of compact feature representations cannot be obtained, resulting in the human pose transfer network being unable to generate accurate human targets, the improved deep video compression DCVC is used to compress the human targets; (3) Constructing a video frame reconstruction module The reconstructed video frame module is used to reconstruct the foreground and background frames Combine to generate reconstructed frame First, according to the foreground mask frame (f1, f2, ... f t-1 、f t ) Determine the foreground in the reconstructed background frame The position in the middle, and then use the reconstructed foreground l t Reconstructed background frame The corresponding position in is replaced to obtain a reconstructed frame a, wherein the reconstructed frame a has a transition boundary generated by the replacement, and the reconstructed frame a is input into the reconstruction network to eliminate the transition boundary to obtain a reconstructed frame The method of obtaining the background template B in the video frame by using the Vibe algorithm comprises the following steps: The Vibe algorithm uses a sequence of pixel values ​​of a point to describe the background model of each pixel; M(x)={v1,v2,…,v N } Where M(x) is the background model, v i is the sampling value, N is the length of the point sequence; The pixel value v(x) of the input point and the sample value v in M(x) i In contrast, if the distance in the color space is less than the preset parameter D t , we think that v(x) and v i resemblance; v(x)≈v i if‖v(x)-v i ‖<D t After traversing the entire sequence, count the number of sampling points similar to v(x). If the number is greater than the threshold α, v(x) is the background, otherwise it is the foreground.

2. The multi-objective encoding method for surveillance video according to claim 1, characterized in that: The semantic object detection network is any one of YOLO and RCNN; The Repulison loss function is introduced into the semantic target detection network to improve the tracking effect under occlusion. The formula of the Repulison loss function is: L=L Attr +a*L RepGT +β*L RepBox Among them, L Attr It is an attraction item, which requires the prediction box to be close to its specified target; L RepGT and L RepBox is an exclusion term, which requires the prediction box to be away from other surrounding groundtruth objects and prediction boxes different from other specified targets; the coefficients α and β act as weights to balance the auxiliary loss, and * is multiplication.

3. The multi-objective encoding method for surveillance video according to claim 1, characterized in that: Introducing a structure-preserving motion estimation network into the improved deep video compression DCVC; The structure-preserving motion estimation network includes motion field computation Δ main , auxiliary motion field calculation Δ aux and sports fields integrate three parts; Among them, C(·,·) represents the concatenation operation, Conv 3×3 represents the convolution operation, represents the element-by-element multiplication of the matrix, and ReLU is the activation function; F t is the input frame, F t-1 is the previous frame, The previous frame to be decoded.

4. The multi-objective encoding method for surveillance video according to claim 1, wherein: The reconstruction network consists of three convolutional layers, three residual blocks and three transposed convolutional layers; the three convolutional layers are used to capture the local features of the image in the reconstructed frame a, and the three residual blocks are used to learn the foreground l obtained by the semantic target detection network. t and reconstructed background frame The nonlinear relationship between them is achieved by using three transposed convolutional layers for upsampling to restore the spatial resolution of the image.

Images