End-to-end monitoring video coding and decoding method based on long-time reference and motion space-time association

Abstract

The invention discloses a long-time reference and motion space-time correlation end-to-end monitoring video coding and decoding method. The method comprises the following steps: step 1, in a motion estimation stage, simultaneously utilizing a long-time reference frame (xLTR) and a short-time reference frame # imgabs0 # in a monitoring video and an original image xt to carry out optical flow estimation, and simultaneously generating two pieces of motion information, namely mt1 and mt2; step 2, sending mt1 and mt2 into a motion information coding and decoding module, and outputting a reconstruction value of motion information by the motion information coding and decoding module, namely # imgabs1 #; step 3, carrying out motion compensation on # imgabs3 # by using # imgabs2 # to obtain a prediction frame # imgabs4 #, carrying out motion compensation on xLTR by using # imgabs5 # to obtain a prediction frame # imgabs6 #; step 4, sending # imgabs7 # into a prediction frame fusion module, and carrying out motion compensation on xLTR by using # imgabs4 #; and step five, carrying out subtraction on the prediction frame # imgabs9 # and the original image xt to obtain an image residual error rt, carrying out encoding and decoding on the image residual error to obtain a reconstruction residual error # imgabs10 #, and adding the reconstruction residual error # imgabs10 # and the prediction frame # imgabs11 # to obtain a reconstruction frame. According to the invention, stable and high-quality reference information can be provided.

Description

Technical Field

[0001] The invention belongs to the technical field of monitoring videos, and in particular relates to a long-term reference and motion spatiotemporal association end-to-end monitoring video encoding and decoding method. Background Art

[0002] As video data on the Internet continues to grow, the requirements for video codec performance are getting higher and higher. In order to meet this challenge, new video codec standards are constantly being introduced. With the rise of deep learning, deep learning-based methods have become a new video codec solution. Compared with traditional video codec solutions, deep learning-based methods use an end-to-end joint optimization approach, and the encoding performance is comparable to that of traditional video codec methods. The main end-to-end video codec methods can be divided into two categories: residual coding and conditional coding.

[0003] Among the existing end-to-end video encoding and decoding methods, some adopt the strategy of multiple reference frames, and some utilize the temporal correlation and spatial correlation of motion information, but these methods are not targeted at surveillance videos.

[0004] Prior art, J.Lin, D.Liu, H.Li, and F.Wu, "M-LVC: Multiple Frames Predictionfor Learned Video Compression," presented at the Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Reco, 2020, pp.3546–3554, a method for predicting motion information and using multiple reference frames. Four reference frames are used in the motion compensation stage, and the four reference frames are the first four decoded frames. This reference frame selection method does not fully consider the fact that the majority of surveillance video scenes are stable videos, and the number of reference frames is large, requiring a larger cache space; the motion information prediction part only uses the temporal correlation of the motion information, and does not consider the spatial correlation of the motion information.

[0005] Prior art, J.Li, B.Li, and Y.Lu, "Hybrid Spatial-Temporal EntropyModelling for Neural Video Compression," in Proceedings of the 30th ACMInternational Conference on Multimedia, Oct. 2022, pp. 1503–1511, a method for jointly applying temporal correlation and spatial correlation of motion information. The method applies temporal and spatial correlation of motion information in an entropy model, and the data input into the entropy model is not the original motion information, but information that has been processed by a neural network, which may cause some original information to be lost.

[0006] The multi-reference frame method mentioned in the above technical solution does not take into account the characteristics of the surveillance video in its reference frame selection. The reference frame selection is the first four frames of the currently encoded frame. Considering that the majority of surveillance videos are stable videos, this reference frame selection strategy is not optimal for surveillance videos.

[0007] In terms of the temporal and spatial correlation of motion information, existing methods use the temporal and spatial correlation of motion information during entropy coding. At this time, the motion information is information that has been processed by the neural network, not the original motion information, which may cause some original information to be lost, thereby making insufficient use of the temporal and spatial correlation of motion information. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a long-term reference and motion spatiotemporal correlation end-to-end monitoring video encoding and decoding method, which directly utilizes the temporal and spatial correlation of the original motion information, thereby making more effective use of this information. The proposed long-term reference frame has a higher image quality than other non-long-term reference frames, and the long-term reference frame is unchanged in a group of coded images (GOP), which can provide stable and high-quality reference information.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A long-term reference and motion spatiotemporal correlation end-to-end monitoring video encoding and decoding method comprises the following steps;

[0011] Step 1: In the motion estimation stage, the long-term reference frame x in the surveillance video is used LTR and short-term reference frames With the original image x t Perform optical flow estimation to generate m t1 and m t2Two motion information, t represents the sequence number in the GOP (t=0, 1, 2, ...), when t=0, it means that the frame is a key frame (I frame), intra-frame coding is used, and key frame coding does not use reference frames;

[0012] Step 2: m t1 、m t2 Send it to the motion information encoding and decoding module to obtain the reconstructed value of the motion information, that is,

[0013] Step 3: Utilization right Perform motion compensation to get the predicted frame use x LTR Perform motion compensation to get the predicted frame

[0014] Step 4: Send it to the prediction frame fusion module to get the final prediction frame

[0015] Step 5: Prediction frame and the original image x t Do the difference to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0016] In step 1, the network used for optical flow estimation is SPyNet, i.e., the optical flow estimation network. The input of SPyNet is two frames of images, a long-term reference frame (x LTR ) and the current frame, i.e. the original image x t The generated motion information m t1 , short-term reference frame and the current frame, the original image x t The generated motion information m t2 ;

[0017] In the step 1, the long-term reference frame selects a key frame in the encoded image.

[0018] Long-term reference frame: has a longer validity period and will be used as a reference frame by multiple frames of images. In the present invention, the long-term reference frame is the I frame of each GOP;

[0019] Short-term reference frame: The validity period is short. As the encoding process proceeds, the short-term reference frame will be continuously replaced by new frames to reflect the latest changes in the video content. In the present invention, the short-term reference frame is the previous frame of the currently encoded frame. In step 2, the specific process of the motion information encoding and decoding module is described as follows:

[0020] Step 1): The motion information is divided into two parts in a checkerboard manner. The two parts are encoded separately, corresponding to two encoding stages. Stage 1 encoding Phase 2 Encoding In the first stage, the first encoded part is used to predict the later encoded part, that is, the spatial correlation of motion information is utilized;

[0021] Step 2): Phase 1, using the reconstructed motion information of the first two frames in the cache and Information about the first part of the campaign Make predictions and get predicted motion information

[0022] Step 3): The first part of the motion information and predicted motion information Make a difference and get the difference

[0023] Step 4): The motion information residual is sent to the encoding and decoding module to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part Predicted motion information with the first part Add together to get the first part of the reconstructed motion information

[0024] Step 5): Phase 2, the first part of the reconstruction of motion information Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information is input into the motion information prediction network at the same time to obtain the predicted motion information

[0025] Step 6): The second part of the motion information and predicted motion information Make a difference and get the difference Will The motion information residual is sent to the encoding and decoding module to obtain the second part of the reconstructed motion information difference Reconstruct the motion information difference from the second part And the second part predicts motion information Add the second part to reconstruct the motion information

[0026] Step 7): Finally, and Add together to reconstruct the original motion information

[0027] In step 2), the process of generating predicted motion information is described as follows: and Input to the feature extraction network to generate feature F t-1 and F t-2 , F t-1 、F t-2 and historical features After connecting in the channel dimension, input the historical feature extraction network to generate historical features Put it into the cache for the next motion information prediction, and then The input motion information predictor generates the predicted motion information m t′ ;

[0028] The feature extraction network consists of a convolutional layer and a ResBlocks, ResBlocks consists of 3 ResBlocks, and ResBlocks consists of two convolutional layers and a ReLU activation layer; the historical feature extraction network consists of 4 layers of ConvLSTM; the ResBlocks in the motion information predictor are the same as those in the feature extraction network.

[0029] In the step 3, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain a final prediction frame;

[0030] Reference frame and sports information Perform a warp operation (sample in the reference frame according to motion information) to generate a preliminary prediction frame; Sports Information After connecting in the channel dimension, a CNN enhancement network is input to obtain the final predicted frame

[0031] In step 4, the prediction frame fusion module is specifically as follows:

[0032] First Divide into 8 pixel × 8 pixel blocks, and calculate the The peak signal-to-noise ratio (PSNR) of each block in the image and the original image is calculated as follows:

[0033]

[0034] The MSE calculation method in formula (1) is formula (2), MAX Iis the maximum pixel value that can be obtained in the image; I and K in formula (2) are the two images used to calculate PSNR, and I in the prediction frame fusion module is the original image x t ,K is the predicted frame or m and n are value ranges. It is divided into 8×8 pixel blocks, so the values ​​of m and n are both 8; after the PSNR calculation of all blocks is completed, the block with the larger PSNR is selected in the two prediction frames in units of blocks, and a mask is generated according to the selection result. The mask consists of 0 and 1 to select the prediction frame The block in the value is 1, and the predicted frame is selected The value of the block in is 0. Fusion generates the final prediction frame The fusion method is shown in formula (3);

[0035]

[0036] In the step 5, the image residual is encoded and decoded to obtain the predicted frame. and the original image x t Do the difference to get the image residual r t , the calculation formula is (4);

[0037]

[0038] Get the residual r t After that, r t Input residual codec network for encoding and decoding. The residual codec includes residual encoder, residual decoder and super prior codec. The residual encoder and residual decoder encode and decode the residual respectively. The super prior codec is used to generate is a set of parameters to recover y′ t Providing probability estimates, the residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0039] A long-term reference and motion spatiotemporal association end-to-end monitoring video encoding and decoding system, including an optical flow estimation module, a motion information encoding and decoding module, a motion compensation module, a prediction frame fusion module, and a residual encoding and decoding module;

[0040] The optical flow estimation module generates a motion vector, i.e., motion information; the motion information encoding and decoding module encodes and decodes the motion information; the motion compensation module generates a prediction frame using the motion information and a reference frame; and the residual encoding and decoding module encodes and decodes the difference between the original image and the prediction frame, i.e., the residual.

[0041] The optical flow estimation module is used to generate motion information, which describes the displacement of an object between frames and is used to generate a prediction frame in the motion compensation module;

[0042] The optical flow estimation module uses the long-term reference frame and short-term reference frame in the surveillance video and the original image to perform optical flow estimation, and generates two motion information at the same time;

[0043] The motion information encoding and decoding module is used to compress the original motion information data generated by the optical flow estimation module. The motion information encoding and decoding module includes a motion information encoder and a motion information decoder. The motion information encoder is used to compress the motion information, i.e., encoding, and the motion information decoder is used to recover the motion information from the compressed data, i.e., decoding.

[0044] The motion compensation module is used to generate two prediction frames; the prediction frame is a prediction of the current frame to be encoded;

[0045] The prediction frame fusion module is used to fuse two prediction frames;

[0046] The difference between the original image and the predicted frame is a residual, and the residual encoding and decoding module is used to compress the residual.

[0047] Beneficial effects of the present invention:

[0048] Based on the original end-to-end video encoding and decoding solution, the present invention optimizes the surveillance video scenario. Due to the introduction of long-term reference frames, the image quality of the background part of the image is effectively improved. Compared with the end-to-end solution that is not optimized for surveillance video, the PSNR BD-rate is reduced by 1%-3%;

[0049] The motion information spatiotemporal correlation utilization strategy proposed in the present invention is more optimal, and can make fuller use of the temporal and spatial correlation of motion information. Compared with the end-to-end video encoding and decoding solution that does not use the spatiotemporal correlation of motion information, the PSNR BD-rate is reduced by 3%-10%. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention is a framework diagram of an end-to-end surveillance video encoding and decoding method based on long-term reference frames and spatiotemporal correlation of motion information.

[0051] Figure 2 It is a schematic diagram of the motion information encoding and decoding module of the present invention.

[0052] Figure 3 Schematic diagram of the motion compensation network of the present invention.

[0053] Figure 4 This is a distribution diagram of the reference information sources of the prediction frame of the present invention.

[0054] Figure 5This is a schematic diagram of the motion information prediction network structure of the present invention.

[0055] Figure 6 It is a schematic diagram showing the motion information prediction effect of the present invention.

[0056] Figure 7 It is a schematic diagram of the self-network structure in the motion information prediction network of the present invention.

[0057] Figure 8 Schematic diagram of the motion information residual encoding and decoding network of the present invention.

[0058] Fig. 9 This is the residual codec network structure diagram. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0060] The present invention proposes a long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding method, the framework of which is as follows: Figure 1 As shown, Figure 2 for Figure 1 The motion information encoding and decoding module in.

[0061] The method proposed in the present invention is based on an end-to-end residual coding and decoding framework, on the basis of which a long-term reference frame and a prediction frame fusion module are added, and motion information coding and decoding are improved.

[0062] The method proposed by the present invention comprises the following modules: an optical flow estimation module, a motion information encoding and decoding module, a motion compensation module, a prediction frame fusion module, and a residual encoding and decoding module.

[0063] The optical flow estimation module is used to generate motion information, which describes the displacement of an object between frames and is generally represented by a motion vector (MV). Motion information is a necessary auxiliary information in video encoding and decoding, and is used to generate prediction frames in the motion compensation module;

[0064] The motion information encoding and decoding module is used to compress the original motion information data generated by the optical flow estimation module. The original motion information data generated by the optical flow estimation module is large in amount, and the motion information needs to be compressed for transmission. The motion information encoding and decoding module includes a motion information encoder and a motion information decoder. The motion information encoder is used to compress the motion information, that is, encoding, and the motion information decoder is used to recover the motion information from the compressed data, that is, decoding.

[0065] The motion compensation module is used to generate a prediction frame; the prediction frame is a prediction of the frame that currently needs to be encoded, and the purpose of using the prediction frame is to reduce the amount of data transmitted.

[0066] The prediction frame fusion module is used to fuse two prediction frames. In this method, two prediction frames will be generated. In order to effectively use the two prediction frames, the two prediction frames need to be fused.

[0067] The difference between the original image and the predicted frame is the residual, and the residual codec module is used to compress the residual. The residual is the difference between the original image and the predicted frame. The amount of residual data transmitted directly is large, so the residual needs to be compressed before transmission. The residual codec module includes a residual encoder and a residual decoder. The residual encoder is used to compress the residual, that is, encoding; the residual decoder is used to recover the residual from the compressed data, that is, decoding.

[0068] The optical flow estimation module generates motion vectors, i.e., motion information, the motion information encoding and decoding module encodes and decodes the motion information, the motion compensation module generates a prediction frame using the motion information and the reference frame, and the residual encoding and decoding module encodes and decodes the residual (the difference between the original image and the prediction frame).

[0069] The overall encoding process after adding the long-term reference frame is described as follows:

[0070] Step 1: In the motion estimation stage, long-term reference frames (x LTR ) and short-term reference frames To perform motion estimation, Figure 1 As shown, two motion information (m t1 、m t2 ), long-term reference frame (x LTR ) and the motion information m generated by the current frame t1 , short-term reference frame and the motion information m generated by the current frame t2 ,The network used for motion estimation is SPyNet.

[0071] Step 2: m t1 、m t2 It will be sent to the motion information codec module, and the motion information codec module outputs the reconstructed value of the motion information, that is,

[0072] Step 3: Use and right and x LTR Perform motion compensation and get two predicted frames at the same time The network structure of motion compensation is as follows Figure 3 As shown in the figure, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0073] Use respectively and right and x LTR Perform motion compensation and get two predicted frames at the same time The network structure of motion compensation is as follows Figure 3 As shown, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0074] Step 4: Send it to the prediction frame fusion module to get the final prediction frame

[0075] Step 5: Prediction frame Difference with the original image to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

[0076] In the step 1, long-term reference frame: in a group of coded images (GOP), as the number of coded frames increases, the coding error is constantly accumulated, resulting in the quality of the reconstructed image getting worse and worse. The long-term reference frame needs to ensure the provision of a stable and high-quality reference image. At the same time, combined with the fact that the background in the surveillance video is almost unchanged, it is most appropriate to select a key frame (I frame) in each GOP as a long-term reference frame.

[0077] Selecting I frame as long-term reference frame has the following advantages:

[0078] 1) Using the I frame as a reference frame for each frame after the I frame can effectively reduce the accumulation of coding errors;

[0079] 2) The I frame image quality is high, and the background remains almost unchanged, which can provide higher quality background information as a reference, effectively improving the quality of the predicted frame. In order to utilize the information in the long-term reference frame, the long-term reference frame and the short-term reference frame are simultaneously subjected to motion estimation and motion compensation to generate two predicted frames, which are then fused to generate a more accurate predicted frame.

[0080] The final prediction frame combines the information in the long-term reference frame and the short-term reference frame, so the prediction is more accurate. The prediction frame fusion method will be described below.

[0081] In step 4, prediction frame fusion: generating two prediction frames Finally, an effective fusion strategy is needed to fuse the two prediction frames so that the final prediction frame can combine the advantages of the two prediction frames. Considering that the long-term reference frame and the short-term reference frame cannot predict every area well, for example, for the background area in the video, the long-term reference frame may predict more accurately, while for the moving objects in the video, the short-term reference frame may predict more accurately. Divide into 8 pixel × 8 pixel blocks, and calculate the The peak signal-to-noise ratio (PSNR) of each block in the image and the original image is calculated as follows:

[0082]

[0083] The MSE calculation method in formula (1) is formula (2), MAX I is the maximum pixel value that can be obtained in the image; I and K in formula (2) are the two images used to calculate PSNR, and I in the prediction frame fusion module is the original image x t ,K is the predicted frame or m and n are value ranges. It is divided into 8×8 pixel blocks, so the values ​​of m and n are both 8. After the PSNR of all blocks is calculated, the block with the larger PSNR is selected in the two prediction frames in units of blocks, and a mask is generated based on the selection result. The mask consists of 0 and 1 to select the prediction frame. The block in the value is 1, and the predicted frame is selected The value of the block in is 0. Fusion generates the final prediction frame The fusion method is shown in formula (3).

[0084]

[0085] Figure 4 Taking a surveillance video as an example, the distribution of reference information sources of two different frames is shown. The green part in the figure indicates that the reference information comes from the long-term reference frame. As can be seen from the figure, the areas where the long-term reference frame works are mainly concentrated in the background area or static area in the video, which is consistent with expectations.

[0086] The step 2, motion information encoding and decoding is specifically as follows:

[0087] There are temporal and spatial correlations between video frames. Temporal correlation refers to the similarity or correlation between adjacent frames in a video sequence. Usually, the frame rate of the video is high (for example, 24 or 30 frames per second), so the time interval between adjacent frames is very short, and the movement of objects in the scene is usually continuous and smooth. This means: 1) the image content changes little, the image content between adjacent frames usually does not change much, and most pixels remain unchanged for a short time; 2) the continuity of motion, the movement of objects between adjacent frames is usually continuous, that is, the displacement and speed of the object will not suddenly change drastically. Spatial correlation refers to the similarity or correlation between adjacent pixels in the same frame. Since the objects and background in the image usually have continuous color, brightness and texture, there is usually a high similarity between adjacent pixels in the same frame.

[0088] The motion information between frames also has temporal and spatial correlation. Motion information is usually represented by motion vectors, which describe the displacement of objects between adjacent frames. Since the movement of objects in the real world is usually smooth and continuous, the motion information is also highly correlated in time. In most cases, the motion vector of an object changes little between adjacent frames, which means that the motion information is highly correlated in time, that is, the temporal correlation of motion information. Motion information is not only correlated in time, but also in space, because of the consistent motion of objects and the consistency of local motion. The consistent motion of objects means that in the same frame, adjacent areas belonging to the same object usually have similar motion vectors because these areas usually move together; the consistency of local motion means that within a local area, the motion vectors of adjacent blocks usually do not change much, which makes the motion information spatially correlated.

[0089] Prediction frames are mainly generated by using the temporal correlation between frames. Prediction frames can effectively remove redundant information between frames. Similarly, in the process of motion information encoding and decoding, the temporal and spatial correlation of motion information can also be used to effectively remove redundant information between motion information. How to use the temporal and spatial correlation of motion information can be divided into three questions:

[0090] 1) How to utilize the temporal correlation of motion information; 2) How to utilize the spatial correlation of motion information; 3) How to combine the temporal correlation and spatial correlation of motion information.

[0091] 1) How to utilize the temporal correlation of motion information: The existing method is to use historical motion information to predict the current motion information, but the network used is relatively simple and less historical motion information is considered. If only two or three historical motion information are considered, the motion curve fitted by the network can only be a first-order or second-order curve. However, in actual situations, the motion of objects is complex, and first-order or second-order curves cannot fit well. The present invention still adopts the strategy of predicting motion information for the utilization of motion information. Unlike the existing methods, the present invention considers more historical motion information in order to improve the prediction accuracy of motion information. To achieve this goal, a historical feature extraction module is added, and the extracted features will be used to predict the current motion information and extract the motion information features of the next frame. The whole process is as follows: Figure 5 As shown in the figure, the historical feature extraction module is implemented by ConvLSTM, which can capture time correlation well and remember more historical information, that is, the prediction of each motion information takes into account all previous historical motion information. The historical feature extraction network and the motion information prediction network use convolutional neural networks.

[0092] 2) How to use the spatial correlation of motion information: In the existing methods, the spatial correlation of motion information is used in the entropy model. The information processed by the entropy model is information that has been processed by the network, not the original motion information, which may cause some information loss. The method proposed in the present invention uses the spatial correlation of motion information by directly using the original motion information. The process is as follows: Figure 7 As shown, the motion information is predicted using the spatial correlation of the motion information. The whole process can be divided into two stages. The motion information to be encoded is divided into two parts in a checkerboard manner and encoded in two steps, corresponding to two stages. The first stage predicts the motion information using only the temporal correlation of the motion information. The second stage considers the temporal and spatial correlations of the motion information when generating the predicted motion information, that is, the reconstructed motion information of the first stage is also used as the input of the motion information prediction network. The motion information prediction has been described in detail in 1). The inputs of the motion information prediction in stage one and stage two are different. Stage one only inputs two historical motion information, and stage two has the motion information decoded in stage one in addition to the two historical motion information.

[0093] 3) How to combine the temporal correlation and spatial correlation of motion information: The combination of temporal correlation and spatial correlation is Figure 7 In the second stage, the motion information prediction network needs to use both the temporal information and spatial information of the motion information. Considering the self-learning nature of the neural network, the temporal information (i.e. ) and spatial information (i.e. ) are spliced ​​in the channel dimension, and the network autonomously learns the relationship between the temporal correlation and spatial correlation of motion information.

[0094] In order to more clearly demonstrate the role of the motion information encoding module, a video scene with relatively intense motion is selected for testing. The motion information prediction effect is shown as follows: Figure 6 In order to observe the difference between the two stages more intuitively, the predicted motion information and motion information residual generated by the two stages are displayed in one figure, that is, the figure shows and It can be seen from the motion information residual graph that the motion information prediction network proposed in the present invention can effectively predict the motion information (the lighter the color, the smaller the residual, and the more accurate the prediction). When the area with a large motion information residual is enlarged, the color of the local enlarged graph shows a chessboard-like distribution of alternating dark and light colors. This is because the motion information prediction in the first stage only uses the temporal correlation of the motion information, and the second stage uses both the temporal and spatial correlations of the motion information. The motion information predicted in the second stage is more accurate than the motion information predicted in the first stage, so the motion information residual in the second stage will be smaller (that is, the color will be lighter).

[0095] like Figure 1 As shown, the motion information encoding and decoding module processes the motion information m t1 、m t2 Encode m t1 、m t2 The encoding process is the same as that of m. t1 and m t2 The description of m t Instead, the motion information encoding process based on spatiotemporal correlation is described as follows:

[0096] 1) Motion information is divided into two parts in a checkerboard manner The two parts are encoded separately, corresponding to two encoding stages. Stage 1 encoding Phase 2 Encoding The first stage is encoding first, and the first encoded part will be used to predict the later encoded part, that is, the spatial correlation of motion information is utilized;

[0097] 2) Stage 1: Reconstructing motion information of the first two frames in the cache and Information about the first part of the campaign Make predictions and get predicted motion information The motion information prediction network structure is as follows Figure 5 As shown, the process of generating predicted motion information is described as follows: and Input to the feature extraction network to generate feature F t-1 and F t-2, F t-1 、F t-2 and historical features After connecting in the channel dimension, input the historical feature extraction network to generate historical features Put it into the cache for the next motion information prediction, and then The input motion information predictor generates the predicted motion information m t′ . Figure 5 The three sub-network structures in Figure 7 As shown in the figure, the feature extraction network consists of a convolutional layer and a ResBlocks, ResBlocks consists of 3 ResBlocks, and ResBlock consists of two convolutional layers and a ReLU activation layer; the historical feature extraction network consists of 4 layers of ConvLSTM; the specific network structure of the motion information predictor is shown in the figure, where ResBlocks is the same as the ResBlocks in the feature extraction network.

[0098] 3) The first part of the motion information and predicted motion information Make a difference and get the difference

[0099] 4) The motion information residual is sent to the encoding and decoding module to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part Predicted motion information with the first part Add together to get the first part of the reconstructed motion information Motion information residual codec network Figure 8 As shown;

[0100] 5) Phase 2, the first part is to reconstruct motion information Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information is input into the motion information prediction network at the same time to obtain the predicted motion information

[0101] 6) The subsequent steps of stage 2 are similar to the motion information encoding and decoding process of stage 1;

[0102] 7) Finally, and Add together to reconstruct the original motion information

[0103] In step 3, the network structure of motion compensation is as follows: Figure 3As shown, a preliminary prediction frame is first generated using the reference frame and motion information, and then a CNN network is input to correct the preliminary prediction frame to obtain the final prediction frame;

[0104] The network structure of motion compensation is as follows Figure 3 (Left) As shown, first use the reference frame and sports information Perform a warp operation (sampling in the reference frame according to motion information) to generate a preliminary prediction frame. The generated preliminary prediction frame needs to be further enhanced. Sports Information After connecting in the channel dimension, a CNN enhancement network is input. The CNN enhancement network structure is as follows Figure 3 (right) shows the final predicted frame

[0105] In the step 5, the image residual is encoded and decoded to obtain the predicted frame. and the original image x t Do the difference to get the image residual r t , the calculation formula is (4);

[0106]

[0107] Get the residual r t After that, r t Input residual codec network for encoding and decoding. The encoding and decoding process is as follows Fig. 9 As shown in Figure 1, the residual codec consists of a codec and a super-prior codec. The left part of the figure is the codec, and the right part is the super-prior codec. The residual encoder and residual decoder encode and decode the residual respectively, and the super-prior codec is used to generate is a set of parameters to recover y ′ Provides probability estimation. AE and AD in the figure refer to arithmetic coding and arithmetic decoding respectively. The residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

Claims

1. A long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding method, characterized in that: The steps include: Step 1: In the motion estimation stage, the long-term reference frame x in the surveillance video is used LTR and short-term reference frames With the original image x t Perform optical flow estimation to generate m t1 and m t2 Two sports information; Step 2: m t1 、m t2 Send it to the motion information encoding and decoding module to obtain the reconstructed value of the motion information, that is, Step 3: Exploitation right Perform motion compensation to get the predicted frame use x LTR Perform motion compensation to get the predicted frame Step 4: Send it to the prediction frame fusion module to get the final prediction frame Step 5: Prediction frame and the original image x t Do the difference to get the image residual r t , and then encode and decode the image residual to obtain the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

2. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 1, characterized in that: In step 1, the network used for optical flow estimation is SPyNet, i.e., the optical flow estimation network. The input of SPyNet is two frames of images and a long-term reference frame x LTR and the current frame, the original image x t The generated motion information m t1 , short-term reference frame and the current frame, the original image x t The generated motion information m t2 ; In the step 1, the long-term reference frame selects a key frame in the coded image; The long-term reference frame is the I frame of each GOP; The short-term reference frame is the previous frame of the currently encoded frame.

3. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 2, characterized in that: In step 2, the specific process of the motion information encoding and decoding module is described as follows: Step 1): The motion information is divided into two parts in a checkerboard manner. The two parts are encoded separately, corresponding to two encoding stages. Stage 1 encoding Phase 2 Encoding In the first stage, the first encoded part is used to predict the later encoded part, that is, the spatial correlation of motion information is utilized; t represents the sequence number in the GOP (t=0, 1, 2, ...). When t=0, it means that the frame is a key frame (I frame), which uses intra-frame coding and has no reference frame. Step 2): Phase 1, using the reconstructed motion information of the first two frames in the cache and Information about the first part of the campaign Make predictions and get predicted motion information Step 3): The first part of the motion information and predicted motion information Make a difference and get the difference Step 4): The motion information residual is sent to the encoding and decoding module to obtain the first part of the reconstructed motion information difference Reconstruct the motion information difference of the first part Predicted motion information with the first part Add together to get the first part of the reconstructed motion information Step 5): Phase 2, the first part of the reconstruction of motion information Will be used to predict the second part of the motion information, while caching the reconstructed motion information of the first two frames and It will also be used to predict the second part of motion information, namely The motion information is input into the motion information prediction network at the same time to obtain the predicted motion information Step 6): The second part of the motion information and predicted motion information Make a difference and get the difference Will The motion information residual is sent to the encoding and decoding module to obtain the second part of the reconstructed motion information difference Reconstruct the motion information difference from the second part And the second part predicts motion information Add the second part to reconstruct the motion information Step 7): Finally, and Add together to reconstruct the original motion information 4. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 3, characterized in that: In step 2), the process of generating predicted motion information is described as follows: and Input to the feature extraction network to generate feature F t-1 and F t-2 , F t-1 、F t-2 and historical features After connecting in the channel dimension, input the historical feature extraction network to generate historical features Put it into the cache for the next motion information prediction, and then The input motion information predictor generates the predicted motion information m t′ ; The feature extraction network consists of a convolutional layer and a ResBlocks, ResBlocks consists of 3 ResBlocks, and ResBlocks consists of two convolutional layers and a ReLU activation layer; the historical feature extraction network consists of 4 layers of ConvLSTM; the ResBlocks in the motion information predictor are the same as those in the feature extraction network.

5. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 3, characterized in that: In step 3, first use the reference frame and sports information Perform a warp operation to generate a preliminary prediction frame; the preliminary prediction frame and the reference frame Sports Information After connecting in the channel dimension, a CNN enhancement network is input to obtain the final predicted frame 6. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 5, characterized in that: In step 4, the prediction frame fusion module is specifically as follows: First Divide into 8 pixel × 8 pixel blocks, and calculate the The peak signal-to-noise ratio (PSNR) of each block in the image and the original image is calculated as follows: The MSE calculation method in formula (1) is formula (2), MAX I is the maximum pixel value that can be obtained in the image; I and K in formula (2) are two images used to calculate PSNR. I in the prediction frame fusion module is the original image x t ,K is the predicted frame or m and n are value ranges. It is divided into 8×8 pixel blocks, and the values ​​of m and n are both 8. After the PSNR of all blocks is calculated, the block with the larger PSNR is selected in the two prediction frames in units of blocks, and a mask is generated according to the selection result. The mask consists of 0 and 1 to select the prediction frame. The block in the value is 1, and the predicted frame is selected The value of the block in is 0, according to the mask pair Fusion generates the final prediction frame The fusion method is shown in formula (3); 7. The method for encoding and decoding a long-term reference and motion spatiotemporal correlation end-to-end surveillance video according to claim 6, characterized in that: In the step 5, the image residual is encoded and decoded to obtain the predicted frame. and the original image x t Do the difference to get the image residual r t , the calculation formula is (4); Get the residual r t After that, r t Input residual codec network for encoding and decoding. The residual codec includes residual encoder, residual decoder and super prior codec. The residual encoder and residual decoder encode and decode the residual respectively. The super prior codec is used to generate is a set of parameters to recover y′ t Providing probability estimates, the residual decoder outputs the reconstructed residual With predicted frame The reconstructed frame is obtained by adding them together.

8. A long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding system for implementing the method described in claims 1-7, characterized in that: It includes optical flow estimation module, motion information encoding and decoding module, motion compensation module, prediction frame fusion module and residual encoding and decoding module; The optical flow estimation module generates a motion vector, i.e., motion information; the motion information encoding and decoding module encodes and decodes the motion information; the motion compensation module generates a prediction frame using the motion information and a reference frame; and the residual encoding and decoding module encodes and decodes the difference between the original image and the prediction frame, i.e., the residual.

9. The long-term reference and motion spatiotemporal correlation end-to-end surveillance video encoding and decoding system according to claim 8, characterized in that: The optical flow estimation module is used to generate motion information, which describes the displacement of an object between frames and is used to generate a prediction frame in the motion compensation module; The optical flow estimation module uses the long-term reference frame and short-term reference frame in the surveillance video and the original image to perform optical flow estimation, and generates two motion information at the same time; The motion information encoding and decoding module is used to compress the original motion information data generated by the optical flow estimation module. The motion information encoding and decoding module includes a motion information encoder and a motion information decoder. The motion information encoder is used to compress the motion information, i.e., encoding, and the motion information decoder is used to recover the motion information from the compressed data, i.e., decoding. The motion compensation module is used to generate two prediction frames; The predicted frame is a prediction of the current frame that needs to be encoded; The prediction frame fusion module is used to fuse two prediction frames; The difference between the original image and the predicted frame is a residual, and the residual encoding and decoding module is used to compress the residual.

Images