Method and apparatus for compressed reconstruction of a surveillance video

Abstract

This invention provides a method and apparatus for compressing and reconstructing surveillance video. The method includes: sampling keyframes of the surveillance video based on a Gaussian mixture model to filter keyframes; converting the surveillance video into corresponding video snapshots, then compressing and reconstructing the video to obtain a reconstructed video with a low sampling rate; compressing keyframes of the surveillance video based on a deep convolutional network to obtain reconstructed images of keyframe hotspot regions; and synthesizing the reconstructed video and the reconstructed images of keyframe hotspot regions to obtain the final compressed and reconstructed video. This invention uses a video snapshot compression-sensing method to compress all video frames, compresses all or part of the keyframe regions based on a deep convolutional neural network, then reconstructs the compressed representation, uses a higher sampling rate for hotspot regions to obtain better reconstruction quality, and synthesizes the final reconstructed video, balancing compression speed and reconstruction quality.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and apparatus for compressing and reconstructing surveillance video. Background Technology

[0002] Currently, video coding frameworks perform motion estimation and motion compensation operations at the encoding end to capture intra-frame and inter-frame correlations in video and compress video using redundant information. However, in resource-constrained edge applications, performing complex motion estimation and compensation at the edge consumes a lot of computing power and battery energy.

[0003] Compression algorithms based on compressed sensing theory can offload complex motion estimation and compensation operations to the more resource-rich decoding end, while simpler matrix multiplication is used at the edge to obtain a compressed representation of the data. With fewer irrelevant variables, video can be abstracted as a temporal change process of a two-dimensional image. According to the aforementioned CS (Compressive Sensing) theory, for a single image, when the measurement matrix and sparse representation matrix satisfy a certain degree of incorrelation, the original image can be accurately reconstructed by obtaining a small number of measurements.

[0004] However, the aforementioned independent CS reconstruction of a single image, which only utilizes the sparsity prior of the image itself, often fails to achieve the required accuracy when the number of measurements is small. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the first objective of this invention is to propose a method for compressing and reconstructing surveillance videos.

[0006] A second objective of this invention is to provide a device for compressing and reconstructing surveillance videos.

[0007] The technical solution adopted in this invention is as follows:

[0008] An embodiment of the first aspect of the present invention proposes a method for compressing and reconstructing surveillance video, comprising the following steps: sampling keyframes of the surveillance video based on a Gaussian mixture model and obtaining a keyframe sampling sequence for keyframe filtering of the surveillance video; converting the surveillance video into a corresponding video snapshot, and then compressing and reconstructing the surveillance video to obtain a reconstructed video with a low sampling rate; compressing the keyframes of the surveillance video based on a deep convolutional network to obtain a reconstructed image of the keyframe hotspot region; and synthesizing the reconstructed video and the reconstructed image of the keyframe hotspot region according to the sampling sequence to obtain the final compressed and reconstructed video.

[0009] The above-mentioned method for compressing and reconstructing surveillance videos proposed in this invention may also have the following additional technical features:

[0010] According to one embodiment of the present invention, sampling key frames of the surveillance video based on a Gaussian mixture model specifically includes: obtaining the nearest frame n from the current time in the surveillance video. ghistory Estimate the parameters of the Gaussian mixture model; when a new video frame arrives, if the value of a pixel is greater than three times the variance of the model mean, it is marked as a change point; if the number of change points in a specified area is greater than or equal to a set proportion, it is determined that there is an abnormal moving object in the new video frame; when an abnormal moving object is detected, a momentum-based video time unequal interval subsampling algorithm is used to sample keyframes.

[0011] According to one embodiment of the present invention, after converting the surveillance video into a corresponding video snapshot, the surveillance video is compressed and reconstructed to obtain a reconstructed video with a low sampling rate. Specifically, this includes: multiplying the sampling matrices of the surveillance video after color downsampling to convert the surveillance video into a corresponding video snapshot; preprocessing the video snapshot to perform coarse reconstruction of the surveillance video corresponding to the video snapshot; inputting the coarsely reconstructed video into a priori reconstruction network for priori reconstruction; and inputting the priori reconstructed video into an optimized reconstruction network for optimized reconstruction to obtain a reconstructed video with a low sampling rate.

[0012] According to one embodiment of the present invention, the prior reconstruction network includes: an encoding layer, a feature extraction layer, and a decoding layer connected in series. The encoding layer includes: a first three-dimensional convolutional module, a first ReLU (Rectified Linear Unit) module, a second three-dimensional convolutional module, a second ReLU module, a third three-dimensional convolutional module, and a third ReLU module connected in series. The feature extraction layer includes: multiple reversible three-dimensional convolutional modules connected in series. The decoding layer includes: a three-dimensional transposed convolutional module, a fourth ReLU module, a second three-dimensional transposed convolutional module, a fifth ReLU module, a third three-dimensional transposed convolutional module, and a sixth ReLU module connected in series.

[0013] According to one embodiment of the present invention, the optimized reconstruction network is an iterative network, and the optimized reconstruction network includes multiple stages. The network structure of each stage is the same as that of the prior reconstruction network. The input of each stage of the optimized reconstruction network is the output of the network of the previous stage and the input of the network of the previous stage.

[0014] According to one embodiment of the present invention, keyframe compression of the surveillance video based on a deep convolutional network is used to obtain a keyframe hotspot region reconstruction image. Specifically, this includes: compressing the keyframe using a deep convolutional compression network to obtain a compressed representation of the keyframe; obtaining variable regions based on the compressed representation of the keyframe, and calculating the ROI (Region of Interest) matrix based on the values ​​of the variable regions; extracting hotspot regions based on the ROI matrix; compressing several image blocks of the hotspot regions using a hotspot model to obtain a compressed representation of the feature map corresponding to the hotspot region image; and reconstructing the compressed representation of the feature map corresponding to the hotspot region image using a deep convolutional reconstruction network to obtain the keyframe hotspot region reconstruction image.

[0015] According to an embodiment of the present invention, the above-described compression and reconstruction method does not include: if hotspot regions are not extracted, reconstructing based on the compressed representation of the keyframe to obtain a reconstructed image of the keyframe hotspot region.

[0016] According to one embodiment of the present invention, the final compressed and reconstructed video is obtained specifically according to the following formula: in, This refers to the final compressed and reconstructed video. This indicates a reconstructed video with a low sampling rate. Indicates the sampling sequence. This represents the reconstructed image of the hotspot region in the keyframe.

[0017] A second aspect of the present invention provides a compression and reconstruction apparatus for surveillance video, comprising: a sampling module, which samples keyframes of the surveillance video based on a Gaussian mixture model and obtains a keyframe sampling sequence for keyframe filtering of the surveillance video; a first reconstruction module, which converts the surveillance video into a corresponding video snapshot and then compresses and reconstructs the surveillance video to obtain a reconstructed video with a low sampling rate; a second reconstruction module, which compresses keyframes of the surveillance video based on a deep convolutional network to obtain a reconstructed image of keyframe hotspot regions; and a third reconstruction module, which synthesizes the reconstructed video and the reconstructed image of keyframe hotspot regions according to the sampling sequence to obtain the final compressed and reconstructed video.

[0018] The beneficial effects of this invention are:

[0019] This invention implements a momentum-based adaptive video keyframe sampling algorithm based on a Gaussian mixture model. When no abnormal moving targets are present, the sampling frequency of video keyframes is reduced, which is suitable for the characteristics of low-speed surveillance videos and helps to improve the subsequent compression speed. Then, a video snapshot compression sensing method is used to compress all video frames. Based on a deep convolutional neural network, all or part of the keyframe regions are compressed, and then the compressed representation is reconstructed. A higher sampling rate is used for hot spots to obtain better reconstruction quality. Finally, the reconstructed video is synthesized, balancing compression speed and reconstruction quality. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for compressing and reconstructing surveillance video according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram illustrating the principle of acquiring a reconstructed video at a low sampling rate according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of a color video snapshot compressed sensing reconstruction network according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of a priori reconstruction network according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of the t-stage of an optimized reconstruction network according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the structure of a deep convolutional network according to an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram illustrating the principle of acquiring a keyframe hotspot region reconstructed image according to an embodiment of the present invention;

[0027] Figure 8 This is a block diagram of a surveillance video compression and reconstruction apparatus according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Figure 1 This is a flowchart of a method for compressing and reconstructing surveillance video according to an embodiment of the present invention, as follows: Figure 1 As shown, the method includes the following steps S1-S4.

[0030] S1, based on the Gaussian mixture model, samples the key frames of the surveillance video and obtains the key frame sampling sequence for key frame filtering of the surveillance video.

[0031] Furthermore, according to one embodiment of the present invention, sampling key frames of the surveillance video based on a Gaussian mixture model specifically includes: obtaining the closest frame n to the current time in the surveillance video. ghistory Estimate the parameters of the Gaussian mixture model; when a new video frame arrives, if the value of a pixel is greater than three times the variance of the model mean, it is marked as a change point; if the number of change points in a specified area is greater than or equal to a set proportion, it is determined that there is an abnormal moving object in the new video frame; when an abnormal moving object is detected, a momentum-based video time unequal interval subsampling algorithm is used to sample keyframes.

[0032] Specifically, one characteristic of surveillance video is that the image remains relatively unchanged in most cases. Therefore, under normal circumstances, reducing the keyframe sampling rate of the video to compress the original video has little impact on the final reconstruction result. However, in view of the temporal locality and periodicity of abnormal video (abnormal moving objects), this invention adopts a momentum-based video temporal unequal interval subsampling algorithm. When an abnormal moving object is detected, the sampling interval is gradually shortened. The specific sampling interval adopts the following formula (1):

[0033]

[0034] Where Δt i The sampling interval is Δt. i-1 t represents the previous sampling interval. i t is the video frame number for the next sampling. i-1 The last sampled video frame number is η, where T is the period of the lowest sampling rate, and η is the period of the last sampled video frame. t The momentum parameters are sampled over time.

[0035] After sampling using the momentum-based video time-discrete subsampling algorithm, the original surveillance video is obtained. A subsampled video and the corresponding keyframe sampling time sequence Where ∑S key =n key n frame n represents the total number of video frames contained in the monitored video. key Where W is the number of keyframes, H is the width of the video frame, and W is the height of the video frame.

[0036] S2 converts the surveillance video into a corresponding video snapshot, then compresses and reconstructs the surveillance video to obtain a reconstructed video with a low sampling rate.

[0037] According to one embodiment of the present invention, after converting the surveillance video into a corresponding video snapshot, the surveillance video is compressed and reconstructed to obtain a reconstructed video with a low sampling rate. Specifically, this includes: multiplying the sampling matrices of the surveillance video after color downsampling to convert the surveillance video into a corresponding video snapshot; preprocessing the video snapshot to perform coarse reconstruction of the surveillance video corresponding to the video snapshot; inputting the coarsely reconstructed video into a priori reconstruction network for priori reconstruction; and inputting the priori reconstructed video into an optimized reconstruction network for optimized reconstruction to obtain a reconstructed video with a low sampling rate.

[0038] Specifically, the schematic diagram of the principle of this invention for acquiring reconstructed videos at low sampling rates can be found in [reference needed]. Figure 2 As shown, video snapshot compression is first performed to convert the surveillance video into video snapshots. Then, a color video snapshot compression and perceptual reconstruction network is used to reconstruct the video snapshots, resulting in a reconstructed video with a low sampling rate. The structure of the color video snapshot compression and perceptual reconstruction network is shown below. Figure 3 As shown, it includes: a preprocessing module, a priori reconstruction network, and an optimized reconstruction network.

[0039] When obtaining a video snapshot, the principle of the Bayer filter can be used to perform color downsampling, converting the original color image into a single-channel image, reducing the input dimension and number of parameters of the model. The color downsampling formula is expressed as follows (2):

[0040]

[0041] Among them, X bayer ∈R W×H This is a downsampled single-channel image. Color video frames, This refers to the data of the i-th color channel of a color video frame. This is a Bayer color filter matrix arranged in RGGB format, where W and H are the width and height of the video frame, respectively.

[0042] like Figure 2 As shown, the image after color downsampling is multiplied by the sampling matrix to obtain a snapshot of the video. The video snapshot compression process can be represented by the following formula (3):

[0043]

[0044] in For the k-th color downsampled video frame to be compressed,

[0045] C k ∈R W×HLet X be the k-th random sampling matrix, ⊙ be the element-wise matrix multiplication, B be the total number of video frames contained in the snapshot, and X be the random sampling matrix. meas ∈R W×H These are video snapshots corresponding to B video frames.

[0046] For the obtained video snapshot X meas First, a preprocessing module is used to preprocess the video to obtain a coarsely reconstructed video corresponding to the surveillance video frames contained in the snapshot, where X... meas For video snapshots, The preprocessing formula for the coarsely reconstructed video of the surveillance video frame corresponding to the snapshot is as follows (4):

[0047]

[0048] The coarsely reconstructed video is then input into the prior reconstruction network for prior reconstruction.

[0049] According to one embodiment of the present invention, such as Figure 4 As shown, the prior reconstruction network includes: an encoding layer, a feature extraction layer, and a decoding layer connected in series. The encoding layer includes: a first 3D convolutional module Conv3D1, a first ReLU module ReLU1, a second 3D convolutional module Conv3D2, a second ReLU module ReLU2, a third 3D convolutional module Conv3D3, and a third ReLU module ReLU3 connected in series. The feature extraction layer includes: multiple reversible 3D convolutional modules RevConv3D connected in series. The decoding layer includes: a 3D transposed convolutional module ConvT3D1, a fourth ReLU module ReLU4, a second 3D transposed convolutional module ConvT3D2, a fifth ReLU module ReLU5, a third 3D transposed convolutional module ConvT3D3, and a sixth ReLU module ReLU5 connected in series.

[0050] Specifically, such as Figure 4 As shown, the encoding layer consists of the following modules: the feature map has a depth of M. e1 The first 3D convolutional module, Conv3D1, has a kernel size of 5x5x5, a stride of 1, and padding of 2; the first ReLU module, ReLU1, has a feature map depth of M. e2 The convolutional module is a second 3D convolutional module (Conv3D2) with a kernel size of 3x3x3, a stride of 1, and padding of 1; the second ReLU module is ReLU2; and the feature map depth is M. e3 The convolution kernel size is 3x3x3, and the stride is (1, scale). w scale h The third 3D convolutional module, Conv3D3, with padding of 1; and the third ReLU module, ReLU3. The final result is a [M] dimension... e3,B,W / scale w H / scale h The output of ] is then fed into the feature extraction layer.

[0051] Neural networks without special design may lose information at each layer, leading to the need to infer the network input from the network output and weights. This requires saving all intermediate results during the backward differentiation process to update the weights, resulting in a large memory footprint during model training. Therefore, this invention designs a special network module, RevBlock, using the reversible 3D convolution module RevConv3D. This network structure can infer the input from the output. The color video snapshot reconstruction model based on the reversible neural network has a smaller memory footprint during training and can use deeper networks for training.

[0052] Specifically, such as Figure 4 As shown, the feature extraction network 2, constructed using the reversible 3D convolutional module RevConv3D, can be described by the following formula, where F rc3d1 With F rc3d2 Let be a 3D convolutional module, where x is the input and y is the output. The module input x is split along its first dimension to obtain two dimensions [M]. e3 / 2,B,W / scale w H / scale h Given vectors x1, x2, the module outputs y = [y1, y2] based on dimension [M]. e3 / 2,B,W / scale w H / scale h The two vectors y1 and y2 are concatenated along their first dimension.

[0053] y1=x1+F rc3d1 (x2)

[0054] y2=x2+F rc3d2 (x1) (5)

[0055] The reverse calculation process is shown in formula (6):

[0056] x1=y2-F rc3d2 (y1)

[0057] x2=y1-F rc3d1 (y2) (6)

[0058] F rc3d1 With F rc3d2 They have the same structure, as shown below: the feature map depth is M. e3 / 2, a 3D convolutional module with a kernel size of 3x3x3, a stride of 1, and padding of 1; a ReLU module; and a feature map depth of M. e3 / 2, a 3D convolutional module with a kernel size of 3x3x3, a stride of 1, and padding of 1;

[0059] The input and output dimensions of the reversible 3D convolution module RevConv3D are both [M]. e3 ,B,W / scale w H / scale h [It can stack a specified number of reversible 3D convolutional modules to deepen the network.]

[0060] After passing through the feature extraction layer, the output of the feature extraction layer is used as the input to the decoding layer of the prior reconstruction network. The decoding layer consists of the following modules: the feature map depth is M. e3 The convolution kernel is 3x3x3, the stride is 1, and the scale is... w ,scale h The first 3D transpose convolution with padding of 1 is ConvT3D1; ReLU4; the feature map depth is M. e2 The second 3D transposed convolution, ConvT3D2, has a kernel size of 3x3x3, a stride of 1, and padding of 1; ReLU5; and a feature map depth of M. e1 The third-dimensional transposed convolution ConvT3D3 with a kernel size of 1x1x1, a stride of 1, and padding of 1; ReLU6.

[0061] like Figure 3 As shown, this invention adopts an iterative method and proposes an iterative reconstruction method, which is specifically manifested in dividing the reconstruction network into two module parts: a prior reconstruction network and an optimized reconstruction network. The optimized reconstruction network has the same structure as the prior reconstruction network, but the training method and weights are different.

[0062] According to one embodiment of the present invention, the optimized reconstruction network is an iterative network, and the optimized reconstruction network includes multiple stages. The network structure of each stage is the same as that of the prior reconstruction network. The input of each stage of the optimized reconstruction network is the output of the network of the previous stage and the input of the network of the previous stage. Figure 5 This is a schematic diagram of the structure of stage t of an optimized reconstruction network according to an embodiment of the present invention.

[0063] The optimized reconstruction network takes the reconstructed frame as input and the reference reconstruction from the previous step as input, and outputs a reconstructed frame. When training the network for the t-th iteration, the parameters of the previous network are fixed. Multi-step optimized reconstruction improves the network's interpretability and provides variable reconstruction accuracy and speed.

[0064] This invention uses Mean Square Error (MSE) as the loss function during the training of the aforementioned network. MSE is widely used in image processing due to its ease of differentiation. The formula is as follows (7):

[0065]

[0066] Where w is the image width, h is the image height, and c is the color channel. To reconstruct the pixel value at image coordinates (i,j,c), X ijc The pixel value at the actual image coordinates (i,j,c).

[0067] S3 uses a deep convolutional network to compress keyframes in surveillance videos to obtain reconstructed images of keyframe hotspot regions.

[0068] According to one embodiment of the present invention, keyframe compression of surveillance video based on a deep convolutional network is used to obtain a keyframe hotspot region reconstructed image. Specifically, this includes: compressing keyframes using a deep convolutional compression network to obtain a compressed representation of the keyframes; obtaining variable regions based on the compressed representation of the keyframes, and calculating the ROI matrix based on the values ​​of the variable regions; extracting hotspot regions based on the ROI matrix; compressing several images of the hotspot regions using a hotspot model to obtain a compressed representation of the feature maps corresponding to the hotspot region images; and reconstructing the compressed representation of the feature maps corresponding to the hotspot region images using a deep convolutional reconstruction network to obtain a keyframe hotspot region reconstructed image.

[0069] According to one embodiment of the present invention, if a hotspot region is not extracted, reconstruction is performed based on the compressed representation of the keyframe to obtain a reconstructed image of the keyframe hotspot region.

[0070] Specifically, surveillance videos are characterized by a relatively small proportion of changing regions in keyframes, leaving room for further compression. Therefore, this invention employs a keyframe differential compression process based on a deep convolutional network, as follows: compress the keyframe to obtain its compressed representation; extract hotspot regions and compress their images to obtain their compressed representations. If a hotspot image exists, only its compressed representation is transmitted; otherwise, the entire keyframe's compressed representation is transmitted, thereby further improving compression efficiency.

[0071] In embodiments of the present invention, deep convolutional networks such as Figure 6 As shown, the structure of the deep convolutional compression network is as follows: the feature map depth is M. bg The two-dimensional convolutional layer Conv2D is used to obtain [M] bg W mbg H mbgThe feature map of W is used as a compressed representation of the original video frame image, where W mbg H mbg These represent the width and height of the feature map, respectively.

[0072] The structure of a deep convolutional reconstruction network is as follows: Figure 6 As shown, it includes: several RevConv2D modules connected one after another; the feature map depth is M. d1 The convolutional module is a 2D transposed convolutional module (ConvT2D) with a 3x3 kernel, a stride of 1, and padding of 1; a ReLU module is also included; the feature map depth is M. d2 The two-dimensional transposed convolutional module ConvT2D with a kernel size of 3x3, a stride of 1, and padding of 1; and the ReLU module; are also described.

[0073] The RevConv2D module used in this invention can be described by the following formula, where F rc2d1 With F rc2d2 Let be a convolutional block, x be the module input, and y be the module output. The module input x is split into [M] according to the feature map dimension, i.e., the feature map dimension of the encoding layer output. rc2d / 2,W rc2d H rc2d Given two vectors x1 and x2, the module outputs y = [y1, y2], which consists of two vectors with dimensions [M]. rc2d / 2,W rc2d H rc2d The vectors y1 and y2 are concatenated along their first dimension, where M is the vector. rc2d W is the feature map depth input to the module. rc2d H rc2d Here, the width and height of the input feature map are respectively, as shown in the formula below:

[0074] y1=x1+F rc2d1 (x2)

[0075] y2=x2+F rc2d2 (x1) (8)

[0076] The reverse calculation process is as follows:

[0077] x1=y2-F rc2d2 (y1)

[0078] x2=y1-F rc2d1 (y2) (9);

[0079] Where F rc2d1 With F rc2d2They have the same structure, specifically including: feature map depth M rc2d / 2, a 2D convolutional module with a kernel size of 3x3, a stride of 1, and padding of 1; a ReLU module; and a feature map depth of M. rc2d A two-dimensional convolutional module with a kernel size of 3x3, a stride of 1, and padding of 1.

[0080] In a deep convolutional network structure, the encoding layer uses feature maps of depth M. ROI The two-dimensional convolutional layer, the decoding layer is the same as the keyframe general region compression and reconstruction network decoding layer.

[0081] like Figure 7 As shown, for keyframes, a general region encoding layer is used for encoding. Then, the resulting feature map is subtracted from the feature map of the previous keyframe and summed to obtain the delta value of the variable region. The ROI matrix ROI[w,h] is then calculated based on the value of the variable region. After scaling the ROI matrix to the video frame size, a dilation operation with a kernel size of 64x64 and a stride of 64 is performed to obtain the final hotspot regions. For the current keyframe, For the previous keyframe, where,

[0082]

[0083]

[0084] like Figure 7 As shown, several image blocks in the hotspot region are compressed using a hotspot model to obtain a compressed feature map representation of the corresponding hotspot region image. The transmitted data consists of the compressed representation and the corresponding ROI matrix. The reconstruction process involves inputting the compressed feature map representation into the reconstruction network to obtain the reconstructed image of the keyframe hotspot region.

[0085] S4 synthesizes the reconstructed video and keyframe hotspot region reconstructed images according to the sampling sequence to obtain the final compressed reconstructed video.

[0086] Furthermore, according to one embodiment of the present invention, the final compressed and reconstructed video is obtained specifically according to the following formula:

[0087]

[0088] in, This indicates the final compressed and reconstructed video. This indicates a reconstructed video with a low sampling rate.

[0089] Indicates the sampling sequence. This represents the reconstructed image of the hotspot region in the keyframe.

[0090] In summary, the video compression and reconstruction method according to embodiments of the present invention implements a momentum-based adaptive video keyframe sampling algorithm based on a Gaussian mixture model. This reduces the sampling frequency of video keyframes when no abnormal moving targets are present, making it suitable for low-speed surveillance videos and improving subsequent compression speed. Then, a video snapshot compression-sensing method is used to compress all video frames. A deep convolutional neural network is used to compress all or part of the keyframe regions. The compressed representation is then reconstructed, and a higher sampling rate is used for hotspot regions to obtain better reconstruction quality. Finally, the reconstructed video is synthesized, balancing compression speed and reconstruction quality.

[0091] Furthermore, the present invention also proposes a device for compressing and reconstructing surveillance video, including the surveillance video compression and reconstruction method described above.

[0092] Figure 8 This is a block diagram of a surveillance video compression and reconstruction apparatus according to an embodiment of the present invention, as shown below. Figure 8 As shown, the device includes: a sampling module 10, a first reconstruction module 20, a second reconstruction module 30, and a third reconstruction module 40.

[0093] The sampling module 10 is used to sample key frames of the surveillance video based on a Gaussian mixture model and obtain a key frame sampling sequence for key frame selection in the surveillance video; the first reconstruction module 20 is used to convert the surveillance video into a corresponding video snapshot and then compress and reconstruct the surveillance video to obtain a reconstructed video with a low sampling rate; the second reconstruction module 30 is used to compress the key frames of the surveillance video based on a deep convolutional network to obtain a reconstructed image of the key frame hotspot region; and the third reconstruction module 40 is used to synthesize the reconstructed video and the reconstructed image of the key frame hotspot region according to the sampling sequence to obtain the final compressed and reconstructed video.

[0094] The video compression and reconstruction apparatus according to an embodiment of the present invention implements a momentum-based adaptive video keyframe sampling algorithm based on a Gaussian mixture model. When no abnormal moving targets are present, the sampling frequency of video keyframes is reduced, which is suitable for the characteristics of low-speed surveillance videos and helps to improve the subsequent compression speed. Then, a video snapshot compression sensing method is used to compress all video frames. Based on a deep convolutional neural network, all or part of the keyframe regions are compressed. The compressed representation is then reconstructed, and a higher sampling rate is used for hot spots to obtain better reconstruction quality. Finally, the reconstructed video is synthesized, balancing compression speed and reconstruction quality.

[0095] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0096] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples and features of different embodiments or examples described in this specification without contradiction. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples, without contradiction.

[0097] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0099] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0101] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for compressing and reconstructing surveillance video, characterized in that, Includes the following steps: The key frames of the surveillance video are sampled based on a Gaussian mixture model, and the key frame sampling sequence is obtained for key frame filtering of the surveillance video. After converting the surveillance video into a corresponding video snapshot, the surveillance video is compressed and reconstructed to obtain a reconstructed video with a low sampling rate. The keyframes of the surveillance video are compressed using a deep convolutional network to obtain reconstructed images of keyframe hotspot regions. The reconstructed video and the reconstructed images of the keyframe hotspot regions are synthesized according to the sampling sequence to obtain the final compressed reconstructed video.

2. The method for compressing and reconstructing surveillance video according to claim 1, characterized in that, The keyframes of the surveillance video are sampled based on a Gaussian mixture model, specifically including: Get the nearest frame n from the current moment in the surveillance video. ghistory Estimate the parameters of the Gaussian mixture model; When a new video frame arrives, if the value of a pixel is greater than three times the variance of the model mean, it is marked as a point of change. If the number of changing points within the specified area is greater than or equal to the set ratio, then the new video frame is determined to contain an abnormally moving object. When an abnormally moving object is detected, a momentum-based video temporal unequal interval subsampling algorithm is used for keyframe sampling.

3. The method for compressing and reconstructing surveillance video according to claim 1, characterized in that, After converting the surveillance video into corresponding video snapshots, the surveillance video is compressed and reconstructed to obtain a reconstructed video with a low sampling rate, specifically including: The monitoring video is color downsampled and then the sampling matrices are multiplied to convert the monitoring video into a corresponding video snapshot. The video snapshot is preprocessed to coarsely reconstruct the surveillance video corresponding to the video snapshot; The coarsely reconstructed video is input into the prior reconstruction network for prior reconstruction. The previously reconstructed video is input into the optimization reconstruction network for optimization reconstruction to obtain a reconstructed video with a low sampling rate.

4. The method for compressing and reconstructing surveillance video according to claim 3, characterized in that, The prior reconstruction network comprises: an encoding layer, a feature extraction layer, and a decoding layer connected in series, wherein, The encoding layer includes: a first three-dimensional convolutional module, a first ReLU module, a second three-dimensional convolutional module, a second ReLU module, a third three-dimensional convolutional module, and a third ReLU module, which are connected in series. The feature extraction layer includes: multiple interconnected reversible three-dimensional convolutional modules; The decoding layer includes: a three-dimensional transposed convolutional module, a fourth ReLU module, a second three-dimensional transposed convolutional module, a fifth ReLU module, a third three-dimensional transposed convolutional module, and a sixth ReLU module, which are connected in series.

5. The method for compressing and reconstructing surveillance video according to claim 4, characterized in that, The optimized reconstruction network is an iterative network, and it includes multiple stages. The network structure of each stage is the same as that of the prior reconstruction network. The input of each stage of the optimized reconstruction network is the output of the network of the previous stage and the input of the network of the previous stage.

6. The method for compressing and reconstructing surveillance video according to claim 1, characterized in that, The keyframe compression of the surveillance video is performed based on a deep convolutional network to obtain reconstructed images of keyframe hotspot regions, specifically including: The keyframes are compressed using a deep convolutional compression network to obtain a compressed representation of the keyframes; The variable region is obtained based on the compressed representation of the key frame, and the ROI matrix is ​​obtained based on the value of the variable region. Extract hotspot regions based on the ROI matrix; The hotspot region is compressed using a hotspot model to obtain a compressed representation of the feature map corresponding to the hotspot region image; A deep convolutional reconstruction network is used to reconstruct the compressed representation of the feature map corresponding to the hotspot region image to obtain the reconstructed image of the keyframe hotspot region.

7. The method for compressing and reconstructing surveillance video according to claim 6, characterized in that, Not including: If no hotspot region is extracted, reconstruction is performed based on the compressed representation of the keyframe to obtain the reconstructed image of the keyframe hotspot region.

8. The method for compressing and reconstructing surveillance video according to claim 1, characterized in that, The final compressed and reconstructed video is obtained using the following formula: in, This refers to the final compressed and reconstructed video. This indicates a reconstructed video with a low sampling rate. Represents the sampling sequence. This represents the reconstructed image of the hotspot region in the keyframe.

9. A device for compressing and reconstructing surveillance video, characterized in that, include: A sampling module is used to sample key frames of the surveillance video based on a Gaussian mixture model and obtain a key frame sampling sequence for key frame filtering of the surveillance video. The first reconstruction module is used to convert the surveillance video into a corresponding video snapshot, and then compress and reconstruct the surveillance video to obtain a reconstructed video with a low sampling rate. The second reconstruction module is used to compress the key frames of the surveillance video based on a deep convolutional network to obtain a reconstructed image of the key frame hotspot region. The third reconstruction module is used to synthesize the reconstructed video and the reconstructed image of the keyframe hotspot region according to the sampling sequence to obtain the final compressed reconstructed video.

Images