Long-term reference and motion spatio-temporal correlation end-to-end monitoring video coding method

Abstract

A long-time reference and motion space-time correlation end-to-end monitoring video coding method, comprising the following steps: step one: in the motion estimation stage, simultaneously using the long-time reference frame (x LTR ) and the short-time reference frame in the monitoring video and the original image x t to perform optical flow estimation, simultaneously generating two motion information, m t1 , m t2 ; step two: sending m t1 , m t2 into a motion information coding module, and the motion information coding module outputs the reconstructed value of the motion information, that is, step three: using the pair to perform motion compensation, obtaining the predicted frame using the pair x LTR to perform motion compensation, obtaining the predicted frame step four: sending into a predicted frame fusion module, obtaining the final predicted frame step five: subtracting the predicted frame from the original image x t to obtain the image residual r t , and then coding the image residual to obtain the reconstructed residual, and adding the reconstructed residual and the predicted frame to obtain the reconstructed frame. The present application can provide stable high-quality reference information.

Description

Technical Field

[0001] The present invention belongs to the technical field of surveillance videos, and in particular relates to a long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding method. Background Art

[0002] As the amount of video data on the internet continues to grow, the demand for video codec performance is becoming increasingly stringent. To address this challenge, new video codec standards are constantly being introduced. With the rise of deep learning, deep learning-based methods have emerged as a new video codec solution. Compared to traditional video codec solutions, deep learning-based methods utilize an end-to-end joint optimization approach, achieving comparable encoding performance. Currently, the main end-to-end video codec methods can be divided into two categories: residual coding and conditional coding.

[0003] Among the existing end-to-end video encoding and decoding methods, some adopt a multi-reference frame strategy, and some utilize the temporal and spatial correlation of motion information, but these methods are not targeted at surveillance videos.

[0004] Prior art, J.Lin, D.Liu, H.Li, and F.Wu, "M-LVC: Multiple Frames Prediction for Learned Video Compression," presented at the Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2020, pp. 3546–3554, involves a method for predicting motion information and using multiple reference frames. Four reference frames are used in the motion compensation stage, which are the first four decoded frames. This reference frame selection method does not fully consider the fact that most surveillance video scenes are stable. Furthermore, the large number of reference frames requires more cache space. The motion information prediction part only utilizes the temporal correlation of motion information, but does not consider its spatial correlation.

[0005] Prior art, J. Li, B. Li, and Y. Lu, “Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression,” in Proceedings of the 30th ACM International Conference on Multimedia, October 2022, pp. 1503–1511, uses a method that jointly applies temporal and spatial correlations to motion information. This method applies temporal and spatial correlations to motion information within an entropy model. However, the input data to the entropy model is not the original motion information, but information that has been processed by a neural network, which may result in the loss of some original information.

[0006] The multi-reference frame method mentioned in the above technical solution does not take into account the characteristics of the surveillance video when selecting the reference frame. The reference frame selected is the first four frames of the currently encoded frame. Considering that the majority of surveillance videos are stable videos, this reference frame selection strategy is not optimal for surveillance videos.

[0007] In terms of the temporal and spatial correlation of motion information, existing methods use the temporal and spatial correlation of motion information during entropy coding. At this time, the motion information is information that has been processed by the neural network, not the original motion information. This may cause some original information to be lost, thereby making the temporal and spatial correlation of motion information insufficiently utilized. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, the present invention aims to provide an end-to-end surveillance video encoding and decoding method with long-term reference and spatiotemporal motion correlation. This method directly exploits the temporal and spatial correlations of raw motion information, making more efficient use of this information. The proposed long-term reference frame offers superior image quality compared to other non-long-term reference frames. Furthermore, the long-term reference frame remains unchanged within a group of coded pictures (GOP), providing stable, high-quality reference information.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A method for encoding and decoding end-to-end surveillance videos with long-term reference and motion spatiotemporal correlation comprises the following steps:

[0011] Step 1: In the motion estimation stage, the long-term reference frame x in the surveillance video is used LTR and short-term reference frames With the original image x t Perform optical flow estimation and generate m t1 and m t2Two motion information, t represents the sequence number in the GOP (t=0, 1, 2, ...), when t=0, it means that the frame is a key frame (I frame), using intra-frame coding, and key frame coding does not use reference frames;

[0012] Step 2: Set m t1 、m t2 Send it to the motion information encoding and decoding module to obtain the reconstructed value of the motion information, that is,

[0013] Step 3: Exploit right Perform motion compensation to obtain the predicted frame use x LTR Perform motion compensation to obtain the predicted frame

[0014] Step 4: Send it to the prediction frame fusion module to get the final prediction frame

[0015] Step 5: Prediction frame and the original image x t Do the difference to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0016] In the step 1, the network used for optical flow estimation is SPyNet, i.e., the optical flow estimation network. The input of SPyNet is two frames of images, a long-term reference frame (x LTR ) and the current frame, the original image x t The generated motion information m t1 , short-term reference frame and the current frame, the original image x t The generated motion information m t2 ;

[0017] In the step 1, the long-term reference frame is selected as a key frame in the coded image.

[0018] Long-term reference frame: It has a long validity period and will be used as a reference frame by multiple frames. In the present invention, the long-term reference frame is the I frame of each GOP;

[0019] Short-term reference frames: These frames are short-lived and are constantly replaced by new frames as the encoding process progresses to reflect the latest changes in the video content. In this invention, the short-term reference frame is the frame preceding the currently encoded frame. The specific process of the motion information encoding and decoding module in step 2 is described as follows:

[0020] Step 1): The motion information is divided into two parts in a checkerboard manner The two parts are coded separately, corresponding to two coding stages. Stage 1 coding Phase 2 Coding The first stage is encoding, and the first encoded part is used to predict the later encoded part, that is, the spatial correlation of motion information is utilized;

[0021] Step 2): Phase 1, using the reconstructed motion information of the first two frames in the cache and Information about the first part of the movement Make predictions and get predicted motion information

[0022] Step 3): The first part of the motion information and predicted motion information Make a difference and get the difference

[0023] Step 4): The motion information residual codec module is sent to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part With the first part of the predicted motion information Add up to get the first part of the reconstructed motion information

[0024] Step 5): Phase 2, the first part of the motion reconstruction Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information prediction network is input at the same time to obtain the predicted motion information

[0025] Step 6): The second part of the motion information and predicted motion information Make a difference and get the difference Will The motion information residual is sent to the encoding and decoding module to obtain the second part of the reconstructed motion information difference Reconstruct the motion information difference from the second part And the second part predicts motion information Add up to get the second part of the reconstructed motion information

[0026] Step 7): Finally, and Add to reconstruct the original motion information

[0027] In step 2), the process of generating the predicted motion information is described as follows: and Input to the feature extraction network to generate features F t-1 and F t-2 , F t-1 、F t-2 and historical characteristics After connecting in the channel dimension, the historical feature extraction network is input to generate historical features Put it into the cache for the next motion information prediction, and then Input motion information predictor generates predicted motion information m t′ ;

[0028] The feature extraction network consists of a convolutional layer and a ResBlocks layer, which consists of three ResBlocks, each of which consists of two convolutional layers and a ReLU activation layer. The history feature extraction network consists of four layers of ConvLSTM. The ResBlocks in the motion information predictor are the same as those in the feature extraction network.

[0029] In step 3, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0030] First use the reference frame and sports information Perform warp operation (sampling in reference frame according to motion information) to generate a preliminary prediction frame; Sports Information After connecting in the channel dimension, a CNN enhancement network is input to obtain the final predicted frame

[0031] In step 4, the prediction frame fusion module is specifically as follows:

[0032] First Divide into 8 pixel × 8 pixel blocks, and calculate the The peak signal-to-noise ratio (PSNR) of each block in the image and the original image is calculated as follows:

[0033]

[0034] The MSE calculation method in formula (1) is formula (2), MAX Iis the maximum pixel value that can be obtained in the image; I and K in formula (2) are the two images used to calculate PSNR, and I in the prediction frame fusion module is the original image x t ,K is the predicted frame or m and n are the value ranges. It is divided into 8×8 pixel blocks, so the values ​​of m and n are both 8. After the PSNR calculation of all blocks is completed, the block with the larger PSNR is selected in the two prediction frames in units of blocks, and a mask is generated based on the selection result. The mask consists of 0 and 1, and the prediction frame is selected. The block in the value is 1, and the predicted frame is selected The value of the block in is 0. Fusion generates the final prediction frame The fusion method is shown in formula (3);

[0035]

[0036] In the step 5, the image residual is encoded and decoded, and the predicted frame and the original image x t Do the difference to get the image residual r t , the calculation formula is (4);

[0037]

[0038] Get the residual r t After that, r t Input residual codec network for encoding and decoding. The residual codec includes residual encoder, residual decoder and super priori codec. The residual encoder and residual decoder encode and decode the residual respectively. The super priori codec is used to generate is a set of parameters to recover y′ t Providing probability estimation, the residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0039] A long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding system, including an optical flow estimation module, a motion information encoding and decoding module, a motion compensation module, a prediction frame fusion module, and a residual encoding and decoding module;

[0040] The optical flow estimation module generates a motion vector, i.e., motion information; the motion information encoding and decoding module encodes and decodes the motion information; the motion compensation module generates a prediction frame using the motion information and the reference frame; and the residual encoding and decoding module encodes and decodes the difference between the original image and the prediction frame, i.e., the residual.

[0041] The optical flow estimation module is used to generate motion information, which describes the displacement of objects between frames, and is used in the motion compensation module to generate predicted frames.

[0042] The optical flow estimation module simultaneously uses long-term reference frames and short-term reference frames in the monitoring video and the original image for optical flow estimation, and simultaneously generates two motion information;

[0043] The motion information coding module is used for data compression of the original motion information generated by the optical flow estimation module, and the motion information coding module includes a motion information encoder and a motion information decoder, the motion information encoder is used for compressing motion information, i.e. encoding, and the motion information decoder is used for recovering motion information from compressed data, i.e. decoding;

[0044] The motion compensation module is used to generate two predicted frames; the predicted frame is a prediction of the current frame to be encoded;

[0045] The predicted frame fusion module is used to fuse the two predicted frames;

[0046] The difference between the original image and the predicted frame is the residual error, and the residual error coding module is used for compressing the residual error.

[0047] The beneficial effects of the present application are:

[0048] The present application optimizes the original end-to-end video coding scheme for the monitoring video scene, and due to the introduction of long-term reference frames, the image quality of the image background part is effectively improved, and compared with the end-to-end scheme not optimized for monitoring video, the PSNR BD-rate is reduced by 1%-3%;

[0049] The motion information spatio-temporal correlation utilization strategy proposed in the present application is more optimal, and the time and space correlation of motion information can be more fully utilized, and compared with the end-to-end video coding scheme not using motion information spatio-temporal correlation, the PSNR BD-rate is reduced by 3%-10%; BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The present application is based on the long-term reference frame and the end-to-end monitoring video coding method framework based on the spatio-temporal correlation of motion information.

[0051] Figure 2 The motion information coding module of the present application is shown in the schematic diagram.

[0052] Figure 3 The motion compensation network of the present application is shown in the schematic diagram.

[0053] Figure 4 The predicted frame reference information source distribution diagram of the present application is shown in the schematic diagram.

[0054] Figure 5This is a schematic diagram of the motion information prediction network structure of the present invention.

[0055] Figure 6 Schematic diagram showing the motion information prediction effect of the present invention.

[0056] Figure 7 Schematic diagram of the self-network structure in the motion information prediction network of the present invention.

[0057] Figure 8 Schematic diagram of the motion information residual encoding and decoding network of the present invention.

[0058] Figure 9 This is the residual codec network structure diagram. DETAILED DESCRIPTION

[0059] The present invention will be described in further detail below with reference to the accompanying drawings.

[0060] The present invention proposes a long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding method, the framework of which is as follows: Figure 1 As shown, Figure 2 for Figure 1 The motion information encoding and decoding module in .

[0061] The method proposed in the present invention is based on an end-to-end residual coding and decoding framework, adds a long-term reference frame and prediction frame fusion module on the basis of the residual coding and decoding framework, and improves the motion information coding and decoding.

[0062] The method proposed in the present invention includes the following modules: an optical flow estimation module, a motion information encoding and decoding module, a motion compensation module, a prediction frame fusion module, and a residual encoding and decoding module.

[0063] The optical flow estimation module is used to generate motion information, which describes the displacement of an object between frames and is generally represented by a motion vector (MV). Motion information is an essential auxiliary information in video encoding and decoding, and is used in the motion compensation module to generate predicted frames.

[0064] The motion information codec module is used to compress the raw motion information data generated by the optical flow estimation module. The raw motion information data generated by the optical flow estimation module is large in size and needs to be compressed for transmission. The motion information codec module includes a motion information encoder and a motion information decoder. The motion information encoder compresses the motion information (i.e., encoding) and the motion information decoder recovers the motion information from the compressed data (i.e., decoding).

[0065] The motion compensation module is used to generate a prediction frame; the prediction frame is a prediction of the current frame to be encoded, and the purpose of using the prediction frame is to reduce the amount of data transmitted.

[0066] The prediction frame fusion module is used to fuse two prediction frames. In this method, two prediction frames will be generated. In order to effectively utilize the two prediction frames, the two prediction frames need to be fused.

[0067] The difference between the original image and the predicted frame is called the residual, and the residual codec module is used to compress the residual. Directly transmitting the residual data, which is the difference between the original image and the predicted frame, requires compression before transmission. The residual codec module includes a residual encoder and a residual decoder. The residual encoder compresses the residual, i.e., encodes it; the residual decoder recovers the residual from the compressed data, i.e., decodes it.

[0068] The optical flow estimation module generates motion vectors, i.e., motion information; the motion information encoding and decoding module encodes and decodes the motion information; the motion compensation module generates a prediction frame using the motion information and the reference frame; and the residual encoding and decoding module encodes and decodes the residual (the difference between the original image and the prediction frame).

[0069] The overall encoding process after adding long-term reference frames is described as follows:

[0070] Step 1: In the motion estimation stage, long-term reference frames (x LTR ) and short-term reference frames Perform motion estimation, such as Figure 1 As shown, two motion information (m t1 、m t2 ), long-term reference frame (x LTR ) and the motion information m generated by the current frame t1 , short-term reference frame And the motion information m generated by the current frame t2 ,The network used for motion estimation is SPyNet.

[0071] Step 2: m t1 、m t2 It will be sent to the motion information codec module, and the motion information codec module outputs the reconstructed value of the motion information, that is,

[0072] Step 3: Use and right and x LTR Perform motion compensation and get two predicted frames at the same time The network structure of motion compensation is as follows Figure 3 As shown in the figure, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0073] Use separately and right and x LTR Perform motion compensation and get two predicted frames at the same time The network structure of motion compensation is as follows Figure 3 As shown in the figure, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0074] Step 4: Send it to the prediction frame fusion module to get the final prediction frame

[0075] Step 5: Prediction frame Difference with the original image to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0076] In step 1, long-term reference frames: Within a group of coded images (GOP), as the number of coded frames increases, coding errors accumulate, resulting in increasingly poor reconstructed image quality. Long-term reference frames must ensure a stable, high-quality reference image. Given the nearly constant background in surveillance video, selecting key frames (I-frames) within each GOP as long-term reference frames is optimal.

[0077] Selecting I-frames as long-term reference frames has the following advantages:

[0078] 1) Using the I frame as a reference frame for each frame after the I frame can effectively reduce the accumulation of coding errors;

[0079] 2) The I-frame image quality is high, and the background remains nearly unchanged, providing higher-quality background information as a reference, effectively improving the quality of the predicted frame. To utilize the information in the long-term reference frame, motion estimation and motion compensation are performed simultaneously on the long-term reference frame and the short-term reference frame, generating two predicted frames. These two reference frames are then fused to generate a more accurate predicted frame.

[0080] The final prediction frame combines the information in the long-term reference frame and the short-term reference frame, so the prediction is more accurate. The prediction frame fusion method will be described below.

[0081] In the step 4, the prediction frame fusion: generate two prediction frames After that, an effective fusion strategy is needed to fuse the two predicted frames so that the final predicted frame can combine the advantages of the two predicted frames. Considering that the long-term reference frame and the short-term reference frame are not good for predicting every region, for example, for the background region in the video, the long-term reference frame can predict more accurately, while for the moving object in the video, the short-term reference frame can predict more accurately, first divide into 8x8 pixel blocks, calculate the peak signal-to-noise ratio (PSNR) of each block in the original image, and the formula for calculating PSNR is as follows:

[0082]

[0083] The MSE calculation method in formula (1) is formula (2), MAX I is the maximum pixel value that can be taken in the image; I and K in formula (2) are two images used to calculate PSNR, I in the predicted frame fusion module is the original image x t , and K is the predicted frame or m and n are the value range, divided into 8x8 pixel blocks, so the values of m and n are both 8. After the PSNR of all blocks is calculated, select the block with larger PSNR in the two predicted frames in units of blocks, and generate a mask according to the selection result, the mask is composed of 0 and 1, the block selected from the predicted frame is 1, and the block selected from the predicted frame is 0. Fuse to generate the final predicted frame The fusion method is shown in formula (3).

[0084]

[0085] Figure 4 Take a monitoring video as an example to show the distribution of different two-frame reference information sources, the green part in the figure indicates that the reference information comes from the long-term reference frame. As can be seen from the figure, the regions where the long-term reference frame works mainly concentrate in the background region or the stationary region in the video, which is consistent with the expectation.

[0086] The step two, motion information coding, is specifically:

[0087] There are temporal and spatial correlations between video frames. Temporal correlation refers to the similarity or correlation between adjacent frames in a video sequence. Generally, the frame rate of a video is high (e.g., 24 or 30 frames per second), so the time interval between adjacent frames is very short, and the motion of objects in the scene is usually continuous and smooth. This means that: 1) the image content changes little, and the image content between adjacent frames usually does not change much, and most pixels remain unchanged in a short time; 2) the continuity of motion, the motion of objects between adjacent frames is usually continuous, i.e., the displacement and velocity of objects do not suddenly change drastically. Spatial correlation refers to the similarity or correlation between adjacent pixels in the same frame. Since the objects and background in the image usually have continuous color, brightness and texture, adjacent pixels in the same frame usually have high similarity.

[0088] Motion information between frames also has temporal and spatial correlations. Motion information is usually represented by motion vectors, which describe the displacement of objects between adjacent frames. Since the motion of objects in the real world is usually smooth and continuous, motion information is also highly correlated in time, and in most cases, the motion vector of an object between adjacent frames changes little, which means that motion information is highly correlated in time, i.e., temporal correlation of motion information. Motion information is not only correlated in time, but also in space, because of the consistent motion of objects and the consistency of local motion. The consistent motion of objects means that adjacent regions belonging to the same object in the same frame usually have similar motion vectors, because these regions usually move together; the consistency of local motion means that the motion vectors of adjacent blocks in a local region usually do not change much, which makes motion information correlated in space.

[0089] Predictive frames are mainly generated using the temporal correlation between frames, and predictive frames can effectively remove the redundancy between frames. Similarly, the temporal and spatial correlations of motion information can also be used to effectively remove the redundancy between motion information in the motion information coding process. How to use the temporal and spatial correlations of motion information can be divided into three problems:

[0090] 1) How to use the temporal correlation of motion information; 2) How to use the spatial correlation of motion information; 3) How to combine the temporal and spatial correlations of motion information.

[0091] 1) How to utilize the temporal correlation of motion information: The existing method is to use historical motion information to predict the current motion information, but the network used is relatively simple and less historical motion information is considered. If only two or three historical motion information are considered, the motion curve fitted by the network can only be a first-order or second-order curve, but in actual situations, the movement of objects is complex, and first-order or second-order curves cannot fit well. The present invention still adopts the strategy of predicting motion information for the use of motion information. Unlike the existing method, the present invention considers more historical motion information in order to improve the prediction accuracy of motion information. In order to achieve this goal, a historical feature extraction module is added, and the extracted features will be used to predict the current motion information and extract the motion information features of the next frame. The whole process is as follows Figure 5 As shown in the figure, the historical feature extraction module is implemented using ConvLSTM. ConvLSTM can effectively capture temporal correlations and memorize more historical information. That is, each motion prediction takes into account all previous historical motion information. The historical feature extraction network and the motion information prediction network use convolutional neural networks.

[0092] 2) How to utilize the spatial correlation of motion information: In existing methods, the spatial correlation of motion information is utilized in the entropy model. The information processed by the entropy model is information that has been processed by the network, not the original motion information, which may cause some information loss. The method proposed in this invention utilizes the spatial correlation of motion information by directly utilizing the original motion information. The process is as follows: Figure 7 As shown, the motion information is predicted using the spatial correlation of the motion information. The whole process can be divided into two stages. The motion information to be encoded is divided into two parts in a checkerboard manner and encoded in two steps, corresponding to two stages. The first stage predicts the motion information using only the temporal correlation of the motion information. The second stage considers the temporal and spatial correlations of the motion information when generating the predicted motion information, that is, the reconstructed motion information of the first stage is also used as the input of the motion information prediction network. The motion information prediction has been described in detail in 1). The inputs of the motion information prediction in stage one and stage two are different. Stage one only inputs two historical motion information, and stage two has the motion information decoded in stage one in addition to the two historical motion information.

[0093] 3) How to combine the temporal correlation and spatial correlation of motion information: The combination of temporal correlation and spatial correlation is Figure 7 In the second stage, the motion information prediction network needs to use the temporal information and spatial information of the motion information at the same time. Considering that the neural network has self-learning properties, the temporal information (i.e. ) and spatial information (i.e. ) to splice in the channel dimension, and the network learns the relationship between the temporal correlation and the spatial correlation of the motion information autonomously.

[0094] In order to more clearly show the role of the motion information encoding module, a video scene with relatively intense motion is selected for testing, and the motion information prediction effect is shown in Figure 6 In order to more intuitively observe the difference between the two stages, the predicted motion information and the motion information residual generated by the two stages are shown in a graph, that is, the graph shows and It can be seen from the motion information residual graph that the motion information prediction network proposed in the application can effectively predict the motion information (the lighter the color, the smaller the residual, and the more accurate the prediction). The enlarged area of the motion information residual is enlarged, and the local enlarged graph presents a chessboard-shaped distribution of light and shade, which is due to the fact that the first stage motion information prediction only utilizes the temporal correlation of the motion information, and the second stage utilizes the temporal and spatial correlation of the motion information. The motion information predicted by the second stage is more accurate than that predicted by the first stage, so the motion information residual of the second stage is smaller (i.e., the color is lighter).

[0095] As shown in Figure 1 , the motion information encoding and decoding module encodes the motion information m t1 , m t2 , and the encoding process of m t1 , m t2 is the same, the following description of m t1 and m t2 is uniformly replaced by m t , and the motion information encoding process based on the space-time correlation is described as follows:

[0096] 1) The motion information is divided into two parts in the form of a checkerboard The two parts are encoded separately, corresponding to two encoding stages, stage one encoding Stage two encoding Stage one is encoded first, and the part encoded first will be used to predict the part encoded later, that is, the utilization of the spatial correlation of the motion information;

[0097] 2) Stage one, using the reconstructed motion information of the previous two frames in the buffer and to predict the first part of the motion information , to obtain the predicted motion information The motion information prediction network structure is shown in Figure 5 , and the process of generating the predicted motion information is described as follows: and Input to the feature extraction network to generate features F t-1 and F t-2, F t-1 、F t-2 and historical characteristics After connecting in the channel dimension, the historical feature extraction network is input to generate historical features Put it into the cache for the next motion information prediction, and then Input motion information predictor generates predicted motion information m t′ . Figure 5 The three sub-network structures in Figure 7 As shown in the figure, the feature extraction network consists of a convolutional layer and a ResBlocks, ResBlocks consists of 3 ResBlocks, and ResBlock consists of two convolutional layers and a ReLU activation layer; the historical feature extraction network consists of 4 layers of ConvLSTM; the specific network structure of the motion information predictor is shown in the figure, where ResBlocks is the same as the ResBlocks in the feature extraction network.

[0098] 3) The first part of the motion information and predicted motion information Make a difference and get the difference

[0099] 4) The motion information residual codec module is sent to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part With the first part of the predicted motion information Add up to get the first part of the reconstructed motion information Motion information residual codec network Figure 8 As shown;

[0100] 5) Phase 2, the first part is to reconstruct motion information Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information prediction network is input at the same time to obtain the predicted motion information

[0101] 6) The subsequent steps of stage 2 are similar to the motion information encoding and decoding process of stage 1;

[0102] 7) Finally, and Add to reconstruct the original motion information

[0103] In step 3, the network structure of motion compensation is as follows: Figure 3As shown in the figure, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0104] The network structure of motion compensation is as follows Figure 3 (Left) As shown, first use the reference frame and sports information Perform warp operation (sampling in the reference frame according to motion information) to generate a preliminary prediction frame. The generated preliminary prediction frame needs to be further enhanced. Sports Information After connecting in the channel dimension, a CNN enhancement network is input. The CNN enhancement network structure is as follows Figure 3 (right) shows the final predicted frame

[0105] In the step 5, the image residual is encoded and decoded, and the predicted frame and the original image x t Do the difference to get the image residual r t , the calculation formula is (4);

[0106]

[0107] Get the residual r t After that, r t Input residual codec network for encoding and decoding. The encoding and decoding process is as follows Figure 9 As shown. The residual codec consists of a codec and a super priori codec. The left part of the figure is the codec, and the right part is the super priori codec. The residual encoder and residual decoder encode and decode the residual respectively, and the super priori codec is used to generate is a set of parameters to recover y ′ Provides probability estimation. AE and AD in the figure refer to arithmetic coding and arithmetic decoding respectively. The residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

Claims

1. A method for end-to-end surveillance video encoding and decoding with long-term reference and motion spatiotemporal correlation, characterized in that: The following steps are included: Step 1: In the motion estimation stage, the long-term reference frame x in the surveillance video is used LTR and short-term reference frames With the original image x t Perform optical flow estimation and generate m t1 and m t2 Two motion information; Step 2: Set m t1 、m t2 Send it to the motion information encoding and decoding module to obtain the reconstructed value of the motion information, that is, Step 3: Exploit right Perform motion compensation to obtain the predicted frame use x LTR Perform motion compensation to obtain the predicted frame Step 4: Send it to the prediction frame fusion module to get the final prediction frame Step 5: Prediction frame and the original image x t Do the difference to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame Add together to obtain the reconstructed frame; In step 2, the specific process of the motion information encoding and decoding module is described as follows: Step 1): The motion information is divided into two parts in a checkerboard manner The two parts are coded separately, corresponding to two coding stages. Stage 1 coding Phase 2 Coding The first stage is encoding, and the first encoded part is used to predict the later encoded part, that is, the spatial correlation of motion information is utilized; t represents the sequence number in the GOP (t=0, 1, 2, ...). When t=0, it means that the frame is a key frame (I frame), which uses intra-frame coding and has no reference frame. Step 2): Phase 1, using the reconstructed motion information of the first two frames in the cache and Information about the first part of the movement Make predictions and get predicted motion information Step 3): The first part of the motion information and predicted motion information Make a difference and get the difference Step 4): The motion information residual codec module is sent to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part With the first part of the predicted motion information Add up to get the first part of the reconstructed motion information Step 5): Phase 2, the first part of the motion reconstruction Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information prediction network is input at the same time to obtain the predicted motion information Step 6): The second part of the motion information and predicted motion information Make a difference and get the difference Will The motion information residual is sent to the encoding and decoding module to obtain the second part of the reconstructed motion information difference Reconstruct the motion information difference from the second part And the second part predicts motion information Add up to get the second part of the reconstructed motion information Step 7): Finally, and Add to reconstruct the original motion information 2. The method for encoding and decoding long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 1, characterized in that: In the step 1, the network used for optical flow estimation is SPyNet, i.e., the optical flow estimation network. The input of SPyNet is two frames of images, a long-term reference frame x LTR and the current frame, the original image x t The generated motion information m t1 , short-term reference frame and the current frame, the original image x t The generated motion information m t2 ; In the step 1, the long-term reference frame selects a key frame in the coded image; The long-term reference frame is the I frame of each GOP; The short-term reference frame is the frame before the currently encoded frame.

3. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 1, characterized in that: In step 2), the process of generating the predicted motion information is described as follows: and Input to the feature extraction network to generate features F t-1 and F t-2 , F t-1 、F t-2 and historical characteristics After connecting in the channel dimension, the historical feature extraction network is input to generate historical features Put it into the cache for the next motion information prediction, and then Input motion information predictor generates predicted motion information m t′ ; The feature extraction network consists of a convolutional layer and a ResBlocks layer, which consists of three ResBlocks, each of which consists of two convolutional layers and a ReLU activation layer. The history feature extraction network consists of four layers of ConvLSTM. The ResBlocks in the motion information predictor are the same as those in the feature extraction network.

4. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 1, characterized in that: In the step 3, first use the reference frame and sports information Perform a warp operation to generate a preliminary prediction frame; The preliminary prediction frame and reference frame Sports Information After connecting in the channel dimension, a CNN enhancement network is input to obtain the final predicted frame 5. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 4, characterized in that: In step 4, the prediction frame fusion module is specifically as follows: First Divide into 8 pixel × 8 pixel blocks, and calculate the The peak signal-to-noise ratio (PSNR) of each block in the image and the original image is calculated as follows: The MSE calculation method in formula (1) is formula (2), MAX I is the maximum pixel value that can be obtained in the image; I and K in formula (2) are two images used to calculate PSNR, and I in the prediction frame fusion module is the original image x t ,K is the predicted frame or m and n are the value ranges. It is divided into 8×8 pixel blocks, and the values ​​of m and n are both 8. After the PSNR calculation of all blocks is completed, the block with the larger PSNR is selected from the two predicted frames in units of blocks, and a mask is generated based on the selection result. The mask consists of 0 and 1, and the predicted frame is selected. The block in the value is 1, select the predicted frame The value of the block in is 0, according to the mask pair Fusion generates the final prediction frame The fusion method is shown in formula (3); 6. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 5, characterized in that: In the step 5, the image residual is encoded and decoded, and the predicted frame and the original image x t Do the difference to get the image residual r t , the calculation formula is (4); Get the residual r t After that, r t Input residual codec network for encoding and decoding. The residual codec includes residual encoder, residual decoder and super priori codec. The residual encoder and residual decoder encode and decode the residual respectively. The super priori codec is used to generate is a set of parameters to recover y′ t Providing probability estimation, the residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

7. A long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding system for implementing the method according to any one of claims 1 to 6, characterized in that: Including optical flow estimation module, motion information encoding and decoding module, motion compensation module, prediction frame fusion module, residual encoding and decoding module; The optical flow estimation module generates a motion vector, i.e., motion information; the motion information encoding and decoding module encodes and decodes the motion information; the motion compensation module generates a prediction frame using the motion information and the reference frame; and the residual encoding and decoding module encodes and decodes the difference between the original image and the prediction frame, i.e., the residual.

8. The long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding system according to claim 7, characterized in that: The optical flow estimation module is used to generate motion information, which describes the displacement of an object between frames and is used to generate a predicted frame in the motion compensation module; The optical flow estimation module uses the long-term reference frame and short-term reference frame in the surveillance video and the original image to perform optical flow estimation, and generates two motion information at the same time; The motion information encoding and decoding module is used to compress the original motion information data generated by the optical flow estimation module. The motion information encoding and decoding module includes a motion information encoder and a motion information decoder. The motion information encoder is used to compress the motion information, that is, encoding, and the motion information decoder is used to recover the motion information from the compressed data, that is, decoding. The motion compensation module is used to generate two prediction frames; The predicted frame is a prediction of the current frame to be encoded; The prediction frame fusion module is used to fuse two prediction frames; The difference between the original image and the predicted frame is a residual, and the residual encoding and decoding module is used to compress the residual.

Images