End-to-end video compression method based on foreground and background segmentation and sub-channel coding

Abstract

The invention discloses an end-to-end video compression method based on foreground and background segmentation and sub-channel coding, and the method comprises the following steps: obtaining a target frame which comprises a current frame and a reconstruction frame of a previous frame; constructing an encoding end-decoding end video compression model, and training based on the target frame to obtain a target end-to-end video compression model; constructing a multi-loss function, and carrying out training optimization on the target coding end-decoding end video compression model to obtain an optimized target coding end-decoding end video compression model; and compressing the video to be compressed based on the target coding end-decoding end video compression model. According to the invention, foreground and background segmentation is realized by using the motion estimation module, the background processing flow is simplified, the calculation complexity is reduced, the coding data volume is reduced, and the compression efficiency is improved; foreground optical flow passes through the motion compensation module and the residual error generation module, foreground motion features are effectively extracted, and the video reconstruction quality is improved while accurate transmission of foreground motion information is ensured.

Description

Technical Field

[0001] The present invention relates to the field of video compression, and in particular to an end-to-end video compression method based on foreground-background segmentation and channel-by-channel coding. Background Art

[0002] In safety and security scenarios like power construction, the widespread deployment of surveillance equipment and its continuous operation have led to explosive growth in video data. This places extremely high demands on storage capacity and network bandwidth. This massive volume of surveillance video not only requires expensive storage devices but also presents challenges such as real-time video freezes and delays during data transmission, severely impacting the normal operation and practical application of surveillance systems. Therefore, efficient video compression technology is crucial for resolving the storage and transmission challenges of surveillance data.

[0003] Traditional video compression standards, such as H264 and H265, have alleviated some of the pressure on data storage and transmission. However, with the rapid development of deep learning technology, deep learning-based video compression methods are gradually emerging. Deep learning possesses powerful feature extraction and data processing capabilities, enabling more accurate capture of complex information in videos. However, some existing deep learning video compression methods rely too heavily on network structures and lack human-designed optimization concepts. These methods differ significantly from traditional compression standards, hindering widespread adoption in practice. Furthermore, existing methods fail to fully exploit the relatively static nature of surveillance video scenes, where moving objects are concentrated in specific areas. Summary of the Invention

[0004] The purpose of the present invention is to address the above-mentioned existing problems and propose an end-to-end video compression method based on foreground and background segmentation and channel coding. It aims to use an end-to-end framework combined with foreground and background segmentation, motion compensation and multi-loss function optimization methods to solve the problem of improving compression efficiency and reducing bit rate while ensuring reconstruction quality.

[0005] The technical solution adopted by the present invention is:

[0006] An end-to-end video compression method based on foreground-background segmentation and channel-by-channel coding, comprising:

[0007] S1. Acquire a target frame, where the target frame includes a current frame and a reconstructed frame of a previous frame;

[0008] S2. Construct an encoder-decoder video compression model, perform training based on the target frame, and obtain a target end-to-end video compression model; the encoder includes a motion estimation module, a motion compensation module, and a residual generation module; the decoder includes a reconstruction module and an optimization module; specifically, the following steps are included:

[0009] S21, obtaining foreground optical flow, background optical flow foreground mask and background mask;

[0010] S22, obtaining the predicted foreground of the current frame;

[0011] S23, obtaining the predicted background of the current frame;

[0012] S24, obtaining the predicted current frame;

[0013] S3. Construct multiple loss functions to train and optimize the target encoder-decoder video compression model to obtain an optimized target encoder-decoder video compression model.

[0014] S4. Compress the compressed video based on the target encoding-decoding video compression model.

[0015] Furthermore, step S1 includes:

[0016] Based on the continuous video stream collected by fixed monitoring equipment in the power safety supervision scenario, the original video stream is obtained;

[0017] Taking frames as units, extract the video sequence from the original video stream to obtain continuous frames;

[0018] Normalizing the continuous frames to obtain normalized continuous frames;

[0019] Select a frame from the normalized continuous frames as the current frame;

[0020] Select the previous frame corresponding to the current frame and reconstruct the previous frame to obtain the reconstructed frame of the previous frame;

[0021] The reconstructed frame of the current frame and the previous frame is used as the target frame.

[0022] Furthermore, step S21 includes:

[0023] Based on the motion estimation module, the target frame is taken as input and the global optical flow field between two consecutive frames is obtained as output: the current frame I t and the reconstructed frame of the previous frame As two consecutive frames, input to the motion estimation module to obtain the current frame I t and the reconstructed frame of the previous frame The global optical flow field between

[0024] Split the global optical flow field to obtain foreground optical flow and background optical flow;

[0025] Based on the optical flow splitting results, the foreground optical flow and background optical flow are marked with binary masks respectively to obtain the foreground mask and background mask.

[0026] Furthermore, step S22 further includes:

[0027] Encode the foreground optical flow to obtain the compressed foreground optical flow;

[0028] Reconstructed frame based on compressed foreground optical flow and previous frame The foreground is used to perform motion compensation through the motion compensation module to obtain the predicted current frame prospects;

[0029] Based on the residual generation module, calculate the previous frame I t-1 The foreground and the reconstructed frame of the previous frame The residual between the foreground and the foreground is obtained;

[0030] Based on the current frame I t The foreground and foreground residual of the current frame I t forecast prospects.

[0031] Furthermore, step S23 further includes:

[0032] Combine the background mask and the current frame I t Multiply and get the current frame I t background;

[0033] Based on the residual generation module, calculate the previous frame I t-1 The background and current frame I t The residual between the background and the background is obtained;

[0034] Based on the previous frame I t-1 The background and background residual of the current frame I t prediction background.

[0035] Furthermore, step S24 further includes:

[0036] Performing entropy coding on the foreground optical flow, foreground residual and background residual to obtain compressed foreground optical flow, foreground residual and background residual, i.e., compressed code stream;

[0037] Based on the reconstruction module, the current frame I t The predicted foreground and current frame I t The predicted background is fused to get the predicted current frame

[0038] Furthermore, step S3 includes:

[0039] 1) Use binary cross entropy loss as mask segmentation loss L mask , constrained foreground mask Consistency with the true segmentation label y:

[0040]

[0041] Where, is the foreground mask; y is the true segmentation label; where y=1 is foreground and y=0 is background;

[0042] 2) Calculate the current frame I t and the previous frame I t-1 True foreground optical flow and predicted foreground optical flow The L2 norm error of the optical flow loss L is obtained flow , the specific calculation formula is as follows:

[0043]

[0044] Where N is the total number of pixels,

[0045] 3) Based on the reconstruction distortion D and the bit rate R, the coding loss L is calculated rate-distortion , the specific calculation formula is as follows:

[0046]

[0047] Where λ is a hyperparameter that adjusts compression efficiency and image quality; D is the L2 norm of the predicted frame and the current real frame; R is the length of the coding module after quantization;

[0048] 4) Calculate the total loss L total , the specific calculation formula is as follows:

[0049] L total =αL flow +βL mask +γL rate-distortion

[0050] Among them, α, β, and γ are hyperparameters used to weigh the impact of each loss term on model training.

[0051] Furthermore, step S4 includes:

[0052] Based on the current frame I t and predict the current frame To address the gap, we train the optimized target encoder-decoder video compression model and determine the compression ratio.

[0053] Compress the video based on the compression ratio.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1) Use the motion estimation module to achieve foreground and background segmentation, use inter-frame residual coding for the background, and take advantage of the static background characteristics of power safety monitoring scenes to simplify the background processing process, reduce computational complexity, reduce the amount of encoded data, and improve compression efficiency;

[0056] 2) The foreground optical flow is further optimized and predicted through the motion compensation module and the residual generation module, effectively extracting foreground motion features and reducing the difference in reconstructed data. This ensures accurate transmission of foreground motion information while improving the quality of video reconstruction.

[0057] 3) Constructing a multi-loss function including mask segmentation loss, optical flow loss and coding loss for end-to-end training can balance the segmentation accuracy, optical flow estimation accuracy and compression performance, enhance the adaptability of the model in power safety monitoring video scenarios, and achieve overall performance optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0059] Figure 1 1 is a flow chart of an end-to-end video compression method based on foreground-background segmentation and channel-by-channel coding according to the present invention;

[0060] Figure 2 It is a schematic diagram of the processing flow of each module of the encoding-decoding end video compression model in the present invention. DETAILED DESCRIPTION

[0061] Example

[0062] like Figures 1 to 2 As shown, an end-to-end video compression method based on foreground-background segmentation and channel-by-channel coding includes:

[0063] S1. Acquire a target frame, which includes a current frame and a reconstructed frame of a previous frame:

[0064] Specifically, step S1 includes:

[0065] Based on the continuous video stream collected by fixed monitoring equipment in the power safety supervision scenario, the original video stream is obtained;

[0066] Taking frames as units, extract the video sequence from the original video stream to obtain continuous frames;

[0067] Normalizing the continuous frames to obtain normalized continuous frames;

[0068] Select a frame from the normalized continuous frames as the current frame I t ;

[0069] Select the previous frame corresponding to the current frame and reconstruct the previous frame to obtain the reconstructed frame of the previous frame

[0070] Current frame I t and the reconstructed frame of the previous frame As the target frame;

[0071] S2. Construct an encoder-decoder video compression model, and perform training based on the target frame to obtain a target end-to-end video compression model; the encoder includes a motion estimation module, a motion compensation module, and a residual generation module; the decoder includes a reconstruction module and an optimization module;

[0072] S21, obtaining foreground optical flow, background optical flow foreground mask and background mask, specifically including the following steps:

[0073] Based on the motion estimation module, the target frame is taken as input and the global optical flow field between two consecutive frames is obtained as output: the current frame I t and the reconstructed frame of the previous frame As two consecutive frames, input to the motion estimation module to obtain the current frame I t and the reconstructed frame of the previous frame The global optical flow field between

[0074] Split the global optical flow field to obtain the foreground optical flow v and background optical flow;

[0075] Based on the optical flow splitting results, the foreground optical flow v and the background optical flow are marked with binary masks to obtain the foreground mask and background mask;

[0076] S22, get current frame I t The forecasting process includes the following steps:

[0077] Encode the foreground optical flow v to obtain the compressed foreground optical flow;

[0078] Reconstructed frame based on compressed foreground optical flow and previous frame The foreground is used to perform motion compensation through the motion compensation module to obtain the predicted current frame prospects;

[0079] Based on the residual generation module, calculate the previous frame I t-1 The foreground and the reconstructed frame of the previous frame The residual between the foreground and the foreground is obtained;

[0080] Based on the current frame I t The foreground and foreground residual r are obtained to obtain the current frame I t forecast prospects;

[0081] S23, get current frame I t The prediction background includes the following steps:

[0082] Combine the background mask and the current frame I t Multiply and get the current frame I t background;

[0083] Based on the residual generation module, calculate the previous frame I t-1 The background and current frame I t The residual between the background and the background is obtained b ;

[0084] Based on the previous frame I t-1 The background and background residual of the current frame I t The forecast background;

[0085] S24. Get the predicted current frame The specific steps include:

[0086] Foreground optical flow v, foreground residual r and background residual r b Perform entropy coding to obtain compressed foreground optical flow, foreground residual and background residual, i.e. compressed code stream;

[0087] Based on the reconstruction module, the current frame I t The predicted foreground and current frame I t The predicted background is fused to get the predicted current frame In order to avoid artifacts after fusion, the artifacts are eliminated based on the optimization module to ensure that the current frame I t The predicted foreground and current frame I t The fusion quality of the predicted background;

[0088] S3. Construct multiple loss functions to train and optimize the target encoder-decoder video compression model to obtain an optimized target encoder-decoder video compression model.

[0089] Specifically, 1) binary cross entropy loss is used as the mask segmentation loss L mask , constrained foreground mask Consistency with the true segmentation label y:

[0090]

[0091] Where, is the foreground mask; y is the true segmentation label; where y=1 is foreground and y=0 is background;

[0092] 2) Calculate the current frame I t and the previous frame I t-1 True foreground optical flow and predicted foreground optical flow The L2 norm error of the optical flow loss L is obtained flow , the specific calculation formula is as follows:

[0093]

[0094] Where N is the total number of pixels,

[0095] 3) Based on the reconstruction distortion D and the bit rate R, the coding loss L is calculated rate-distortion , the specific calculation formula is as follows:

[0096]

[0097] Where λ is a hyperparameter that adjusts compression efficiency and image quality; D is the L2 norm of the predicted frame and the current real frame; R is the length of the coding module after quantization;

[0098] 4) Calculate the total loss L total , the specific calculation formula is as follows:

[0099] L total =αL flow +βL mask +γL rate-distortion

[0100] Among them, α, β, and γ are hyperparameters used to weigh the impact of each loss term on model training.

[0101] S4. Compress the compressed video based on the target encoder-decoder video compression model:

[0102] Specifically, based on the current frame I t and predict the current frame The gap is solved by training an optimized target encoding-decoding video compression model and determining the compression ratio; the video is compressed based on the compression ratio.

[0103] The present invention constructs the encoding end and decoding end into an end-to-end video compression model. The encoding end includes a motion estimation module, a motion compensation module and a residual generation module, and the decoding end includes a reconstruction module and an optimization module. The input is continuous frames, and the parameters of each module are synchronously optimized through gradient back propagation. Multiple loss functions are used to ensure the balance between foreground and background segmentation accuracy, motion estimation accuracy and compression performance, thereby improving the adaptability of the model in surveillance video scenarios.

[0104] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the principles and essence of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. An end-to-end video compression method based on foreground-background segmentation and channel coding, characterized in that: The steps include: S1. Acquire a target frame, where the target frame includes a current frame and a reconstructed frame of a previous frame; S2. Build an encoder-decoder video compression model, train it based on the target frame, and obtain the target end-to-end video compression model; The encoding end includes a motion estimation module, a motion compensation module and a residual generation module; the decoding end includes a reconstruction module and an optimization module; Specifically include: S21, obtaining foreground optical flow, background optical flow foreground mask and background mask; S22, obtaining the predicted foreground of the current frame; S23, obtaining the predicted background of the current frame; S24, obtaining the predicted current frame; S3. Construct multiple loss functions to train and optimize the target encoder-decoder video compression model to obtain an optimized target encoder-decoder video compression model. S4. Compress the compressed video based on the target encoding-decoding video compression model.

2. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 1, characterized in that: Step S1 includes: Based on the continuous video stream collected by fixed monitoring equipment in the power safety supervision scenario, the original video stream is obtained; Taking frames as units, extract the video sequence from the original video stream to obtain continuous frames; Normalizing the continuous frames to obtain normalized continuous frames; Select a frame from the normalized continuous frames as the current frame; Select the previous frame corresponding to the current frame and reconstruct the previous frame to obtain the reconstructed frame of the previous frame; The reconstructed frame of the current frame and the previous frame is used as the target frame.

3. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 1, characterized in that: Step S21 includes: Based on the motion estimation module, the target frame is taken as input and the global optical flow field between two consecutive frames is obtained as output: the current frame I t and the reconstructed frame of the previous frame As two consecutive frames, input to the motion estimation module to obtain the current frame I t and the reconstructed frame of the previous frame The global optical flow field between Split the global optical flow field to obtain foreground optical flow and background optical flow; Based on the optical flow splitting results, the foreground optical flow and background optical flow are marked with binary masks respectively to obtain the foreground mask and background mask.

4. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 3, characterized in that: Step S22 further includes: Encode the foreground optical flow to obtain the compressed foreground optical flow; Reconstructed frame based on compressed foreground optical flow and previous frame The foreground is used to perform motion compensation through the motion compensation module to obtain the predicted current frame prospects; Based on the residual generation module, calculate the previous frame I t-1 The foreground and the reconstructed frame of the previous frame The residual between the foreground and the foreground is obtained; Based on the current frame I t The foreground and foreground residual of the current frame I t forecast prospects.

5. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 4, characterized in that: Step S23 further includes: Combine the background mask and the current frame I t Multiply and get the current frame I t background; Based on the residual generation module, calculate the previous frame I t-1 The background and current frame I t The residual between the background and the background is obtained; Based on the previous frame I t-1 The background and background residual of the current frame I t prediction background.

6. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 5, characterized in that: Step S24 further includes: Performing entropy coding on the foreground optical flow, foreground residual and background residual to obtain compressed foreground optical flow, foreground residual and background residual, i.e., compressed code stream; Based on the reconstruction module, the current frame I t The predicted foreground and current frame I t The predicted background is fused to get the predicted current frame 7. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 1, characterized in that: Step S3 includes: 1) Use binary cross entropy loss as mask segmentation loss L mask , constrained foreground mask Consistency with the true segmentation label y: Where, is the foreground mask; y is the true segmentation label; where y=1 is foreground and y=0 is background; 2) Calculate the current frame I t and the previous frame I t-1 True foreground optical flow and predicted foreground optical flow The L2 norm error of the optical flow loss L is obtained flow , the specific calculation formula is as follows: Where N is the total number of pixels, 3) Based on the reconstruction distortion D and the bit rate R, the coding loss L is calculated rate-distortion , the specific calculation formula is as follows: Where λ is a hyperparameter that adjusts compression efficiency and image quality; D is the L2 norm of the predicted frame and the current real frame; R is the length of the coding module after quantization; 4) Calculate the total loss L total , the specific calculation formula is as follows: L total =αL flow +βL mask +γL rate-distortion Among them, α, β, and γ are hyperparameters used to weigh the impact of each loss term on model training.

8. The end-to-end video compression method based on foreground-background segmentation and channel coding according to claim 1, characterized in that: Step S4 includes: Based on the current frame I t and predict the current frame To address the gap, we train the optimized target encoder-decoder video compression model and determine the compression ratio. Compress the video based on the compression ratio.

Images