Method and device for improving background quality in low code rate complex scene

Abstract

The application discloses a background quality improving method and device under a low code rate complex scene, wherein the method comprises: a training process, further comprising processing the to-be-trained foreground and background sample sequence and the reference background sequence into a training data pair suitable for a model by using a training data preprocessing module, wherein the data pair comprises a low-quality training sequence, a corresponding high-quality background reference sequence and a high-quality interference-free background under a corresponding scene as an input reference frame; obtaining features by using a foreground and background feature separation module; performing fusion and up-sampling operations on the foreground and background separation features by using a foreground and background feature fusion module to obtain a fusion frame; and an inference process, further comprising loading a to-be-processed video sequence from a video stream or a storage medium and a reference background frame under a corresponding scene; inputting the foreground and background separation features by using an optimized foreground and background feature fusion module to obtain a background enhancement frame sequence, i.e., a processed video.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video enhancement, and particularly relates to a background quality improvement method and device under a low code rate complex scene. BACKGROUND

[0002] In real life, because a video occupies a large amount of bandwidth in the transmission process, a transmission code rate is reduced, and video compression is used to reduce transmission costs, which results in a poor video quality at a playing end. For example, a monitoring video needs to be monitored at all times, and the video data is huge. In order to reduce transmission costs, the video quality at the playing end is inevitably poor, and details are lacking, which is not conducive to the material of calling monitoring evidence.

[0003] For the above demand scene, if the background quality at the playing end can be improved, the quality requirement for the background in the video transmission process can be reduced, transmission resources are all left for important detail areas, and the background information can be optimized and enhanced at the playing end, which not only reduces transmission costs, but also retains key information of the video.

[0004] At present, a super resolution technology is generally used for improving the video image quality. Traditional super resolution is based on the texture of an image, and the texture direction is judged, and then enhancement is performed on the basis, and the processed image is still relatively fuzzy, and the quality is not satisfactory. In recent years, machine learning is increasingly used for super resolution. A neural network structure based on machine learning can better obtain image details, and a generative adversarial network (GAN) technology can generate a high-quality result, but the super resolution model based on the GAN has the following defects: details of an image are generated based on training data, and the obtained image can have a large ambiguity in features and details with an original image, and is not suitable for a scene requiring image details such as a monitoring video. SUMMARY

[0005] In view of the above problems, the present application provides a background quality improvement method and device under a low code rate complex scene, which is used for solving the problem of poor video background quality in various monitoring scenes after high compression rate and ultra-low code rate transmission.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] An aspect of an embodiment of the present application provides a background quality improvement method under a low code rate complex scene, comprising the following steps.

[0008] The training process further comprises: using a training data preprocessing module to process the foreground and background sample sequence to be trained and the reference background sequence into model-applicable training data pairs, the data pairs comprising low-quality training sequences, corresponding high-quality background reference sequences, and high-quality non-interference background under the corresponding scene as input reference frames; using a foreground and background feature separation module to input the sequence as the low-quality training sequence and the input reference frame for 4-pixel resolution decompression downsampling to obtain features, using an M-group double residual dense network in series to extract features for the separation of background features, using a residual network to output foreground and background separation features for the separation of foreground features; using a foreground and background feature fusion module to perform fusion and upsampling operations on the foreground and background separation features to obtain a fusion frame; using a target function optimization module to take the minimum reconstruction loss function between the fusion frame and the corresponding reference image sequence as a target function, and find the optimal parameters of the foreground and background feature separation module and the foreground and background feature fusion module through gradient descent method, and then obtain the optimized foreground and background feature separation module and the foreground and background feature fusion module.

[0009] The inference process further comprises: loading a video sequence to be processed and a reference background frame under the corresponding scene from a video stream or a storage medium, using an inference data preprocessing module to combine the loaded video sequence and the reference background frame sequence into a data pair; obtaining foreground and background separation features by using the optimized foreground and background feature separation module on the data pair obtained by combining the video sequence and the reference background frame sequence; using the optimized foreground and background feature fusion module to input the foreground and background separation features to obtain a background enhancement frame sequence, i.e., a processed video.

[0010] In a possible design, the value range of M is 10-30.

[0011] In a possible design, each residual network comprises a convolutional layer for extracting a foreground region in a low-quality training sequence obtained by the training data preprocessing module or a video sequence obtained by the inference data preprocessing module.

[0012] In a possible design, a double residual dense network is used to align the training sequence and the input reference frame, identify the difference between the edges of the training sequence and the input reference frame, and then extract background region features.

[0013] In a possible design, the foreground and background feature fusion module comprises a plurality of convolutional layers, LeakyReLU activation layers, and upsampling operations.

[0014] Another aspect of the embodiment of the application provides a background quality improvement device under a low code rate complex scene, comprising:

[0015] The training data preprocessing module is configured to process the foreground and background sample sequence to be trained and the reference background sequence into model-applicable training data pairs, the data pairs including a low-quality training sequence, a corresponding high-quality background reference sequence, and a high-quality non-interference background in the corresponding scene as an input reference frame; the foreground and background feature separation module is configured to perform 4-pixel resolution de-reconstruction down-sampling on the input sequence as the low-quality training sequence and the input reference frame to obtain features, use an M-group double residual dense network in series to extract features for the separation of background features, and directly use a residual network to output foreground and background separation features; the foreground and background feature fusion module is configured to perform fusion and up-sampling operations on the foreground and background separation features to obtain a fusion frame; and the objective function optimization module is configured to take a minimum reconstruction loss function between the fusion frame and a corresponding reference image sequence as an objective function, find optimal parameters of the foreground and background feature separation module and the foreground and background feature fusion module through a gradient descent method, and then obtain the optimized foreground and background feature separation module and the foreground and background feature fusion module.

[0016] The inference data preprocessing module is configured to load a video sequence to be processed and a reference background frame in the corresponding scene from a video stream or a storage medium, and combine the loaded video sequence and the reference background frame sequence into a data pair; the optimized foreground and background separation features are configured to obtain foreground and background separation features by processing the data pair of the video sequence and the reference background frame sequence; and the optimized foreground and background feature fusion module is configured to obtain a background enhancement frame sequence by processing the foreground and background separation features, that is, to obtain a processed video.

[0017] In a possible design, the value range of M is 10-30.

[0018] In a possible design, each residual network includes a convolutional layer configured to extract a foreground region in a low-quality training sequence obtained by the training data preprocessing module or a video sequence obtained by the inference data preprocessing module.

[0019] In a possible design, the background region information in the input frame is almost completely derived from the background information in the high-definition background frame, the double residual dense network is used to align the input frame and the reference frame, the difference between the edges of the input frame and the reference frame is identified, and then the background region features are extracted.

[0020] In a possible design, the foreground and background feature fusion module includes a plurality of convolutional layers, LeakyReLU activation layers, and up-sampling operations.

[0021] The low-code-rate complex scene background quality improvement method has the following beneficial effects: by calculating the feature difference between the foreground and the background of the input to-be-processed frame and the reference frame, the foreground and the background feature information are separated and extracted, and then the separated feature information is fused, the fusion of the high-quality background feature information in the reference frame and the foreground feature information in the to-be-processed frame can be completed, under the premise of ensuring the correctness of the details in the frame, the effective background quality is still improved, not only the transmission bandwidth can be reduced under the condition of keeping the video quality unchanged, but also the background quality of the video stream reaching 4K size can be improved in real time under the condition of ultra-low transmission code rate and very poor video quality. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a low-code-rate complex scene background quality improvement method of an embodiment of the present application is shown in FIG.

[0023] Figure 2 A working flowchart of a foreground and background separation module in the low-code-rate complex scene background quality improvement method of an embodiment of the present application is shown in FIG.

[0024] Figure 3 A low-quality high-compression-rate image before an embodiment of the present application is shown in FIG.

[0025] Figure 4 A Figure 3 A low-code-rate complex scene background quality improvement method processed by an embodiment of the present application is shown in FIG. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] Referring to FIG. Figure 1 , a flowchart of a low-code-rate complex scene background quality improvement method of an embodiment of the present application is shown, which includes:

[0028] The training process further includes processing the to-be-trained foreground and background sample sequence and the reference background sequence into a training data pair suitable for the model by using a training data preprocessing module, and the data pair includes a low-quality training sequence img x , a corresponding high-quality background reference sequence GT, and a high-quality non-interference background under the corresponding scene as an input reference frame img ref; utilize the foreground and background feature separation module, input sequence is low quality training sequence and input reference frame for 4 times pixel unshuffle downsampling, get feature, the separation of background feature uses M group double residual dense network in series extraction feature, the separation of foreground feature directly uses residual network, output foreground and background separation feature; utilize foreground and background feature fusion module to make fusion and upsampling operation to foreground and background separation feature, get fusion frame; utilize target function optimization module, the minimum reconstruction loss function between fusion frame and corresponding reference image sequence is taken as target function, the optimal parameters of foreground and background feature separation module and foreground and background feature fusion module are found through gradient descent method, and then the optimized foreground and background feature separation module and foreground and background feature fusion module are obtained;

[0029] The inference process further comprises loading the video sequence to be processed from the video stream or the storage medium x And the reference background frame frame under the corresponding scene ref , utilize the inference data preprocessing module to combine the loaded video sequence and the reference background frame sequence into a data pair <frame x ,frame ref >; the data pair combined by the video sequence and the reference background frame sequence is obtained through the optimized foreground and background feature separation module to obtain foreground and background separation features; utilize the optimized foreground and background feature fusion module, input foreground and background separation features, get background enhancement frame sequence, that is, get the processed video.

[0030] In an embodiment of the application, the foreground and background feature separation processing flow chart is as shown in Figure 2 The input sequence img x And img ref , first, pixel unshuffle is performed, and features feats are obtained, the separation of background features needs to use M group double residual dense network (Res2DenseBlock) in series extraction features, M can be 10-30, the separation of foreground features directly uses residual network (ResNet), and finally the module outputs foreground and background separation features The foreground and background feature separation can be expressed by the formula:

[0031]

[0032] Res2DenseBlock M (Pixel_unshuffle(<img x ,img ref >))

[0033] Each ResNet only contains convolutional layers (Conv, which is a filter used to extract image features), which is suitable for extracting foreground regions in the input frame; the background region information in the input frame is almost entirely derived from the background information in the high-definition background frame. First, the Res2DenseBlock needs to align the input frame img x with the reference frame img ref , identify the differences in the edges of the aligned input frame img x and the reference frame img ref , and then extract the background region features.

[0034] Each group of Res2DenseBlock contains n DenseNets, and n is generally 3, which is sufficient. All DenseNets are connected using the residual formula, and λ is the residual gain, the calculation formula is:

[0035] feats i = DenseNet i (feats i,1 )* λ + feats i,1 , 3≤i≤n, λ∈(0.1, 0.2)

[0036] The DenseNet module uses multiple convolutional layers for stacking processing, and m represents the number of stacked layers. The processing results of all upper layers are input as part of the lower layer, and the formula is as follows:

[0037]

[0038] The final foreground and background feature separation module outputs <F pre , F back > to the foreground and background feature fusion module for further processing.

[0039] In an embodiment of the present application, the features separated by the foreground and background are fused and upsampled, which can fuse the foreground features separated from the input frame and the background features of the reference frame together, maintain smooth edge transition, achieve clear background, and keep the overall color tone consistent with the input frame.

[0040] The foreground and background feature fusion module Fusion includes multiple convolutional layers, LeakyReLU activation layers (a kind of nonlinear function that improves the fitting ability of the model), and upsample operations (upsample). The activation function is represented as:

[0041] α is a learnable parameter

[0042] The overall logic of the module can be represented as:

[0043] Fusion (feat) = LeakyReLU (Conv (upsample (feat) ) )

[0044]

[0045] The fusion frame is handed over to the target function optimization module for parameter updating.

[0046] In an embodiment of the present application, the target function optimization module is used to update the fusion frame and the minimum reconstruction loss function between the corresponding reference image sequence GT is taken as the target function, and the optimal parameters θ of the foreground and background feature separation module Extract and the foreground and background feature fusion module Fusion are found by the gradient descent method, and the process is shown in the following formula:

[0047]

[0048]

[0049] In the training process, the parameters θ obtained through multiple iterations of a large amount of data can improve the quality index of the fusion frame , and the fusion frame obtained by the best parameters θ can be used as the background enhancement frame Y output.

[0050] In an embodiment of the present application, the video sequence frame x to be processed and the reference background frame frame ref under the corresponding scene are obtained by the optimized foreground and background feature separation module to obtain the foreground and background separation features <F p r e , F b a ck >. The formula is as follows:

[0051] <F p r e , F back > = Extract (<frame x , frame ref >)

[0052] Finally, the foreground and background separation features <F pre , F back > are handed over to the optimized foreground and background feature fusion module for processing.

[0053] In an embodiment of the present application, the foreground and background separation features <F pre , F back > are input into the optimized foreground and background feature fusion module to obtain the background enhancement frame sequence Y, and the whole process is as follows:

[0054] Y = Fusion(<F pre ,F back >).

[0055] By the above setting, the method for improving background quality in low code rate complex scene, the feature difference between the foreground and the background of the input frame to be processed and the reference frame is fully utilized when the foreground and background features are separated, and the foreground and background feature information is successfully separated and extracted; the smooth fusion of the high-quality background feature information in the reference frame and the foreground feature information in the frame to be processed is completed when the foreground and background features are fused; through the operation of foreground and background feature separation and foreground and background feature fusion, the processed video frame not only retains the correctness of the key information of the foreground area, but also enhances the background quality, and eliminates the incoherence of different foreground and background edges in fusion, meeting the visual requirements. Not only can the transmission bandwidth be reduced while maintaining the video quality, but also the background quality of the video stream reaching 4K size can be improved in real time under the condition of ultra-low transmission code rate and poor video quality.

[0056] Corresponding to the method embodiment of the application, continuing to refer to Figure 1 The embodiment of the application also provides a device for improving background quality in low code rate complex scene, which comprises:

[0057] The training data preprocessing module is used for processing the foreground and background sample sequence to be trained and the reference background sequence into model applicable training data pairs, and the data pairs comprise a low-quality training sequence, a corresponding high-quality background reference sequence, and a high-quality non-interference background as an input reference frame under the corresponding scene; the foreground and background feature separation module is used for performing 4 times pixel de-reorganization down-sampling on the input sequence as the low-quality training sequence and the input reference frame to obtain features, using M groups of double residual dense network series to extract features for the separation of the background features, and directly using a residual network to output the foreground and background separation features for the separation of the foreground features; the foreground and background feature fusion module is used for performing fusion and up-sampling operations on the foreground and background separation features to obtain a fusion frame; and the target function optimization module is used for taking the minimum reconstruction loss function between the fusion frame and the corresponding reference image sequence as a target function, finding the optimal parameters of the foreground and background feature separation module and the foreground and background feature fusion module through the gradient descent method, and then obtaining the optimized foreground and background feature separation module and the foreground and background feature fusion module.

[0058] The reasoning data preprocessing module is used for loading a video sequence to be processed and a reference background frame under a corresponding scene from a video stream or a storage medium, and combining the loaded video sequence and the reference background frame sequence into a data pair; the optimized foreground and background separation feature is used for processing the data pair of the video sequence and the reference background frame sequence to obtain a foreground and background separation feature; and the optimized foreground and background feature fusion module is used for processing the foreground and background separation feature to obtain a background enhancement frame sequence, i.e. a processed video.

[0059] In an embodiment of the present application, a foreground and background feature separation processing flowchart is as shown in Figure 2 The input sequence img x and img ref are first subjected to 4-pixel unshuffling down-sampling (Pixel_unshuffle), to obtain a feature feats, the separation of the background feature needs to use M groups of Res2DenseBlock in series to extract the feature, M can be 10-30, the separation of the foreground feature directly uses ResNet, and finally the module outputs a foreground and background separation feature The foreground and background feature separation can be expressed by a formula as follows:

[0060]

[0061] Res2DenseBlock M (Pixel_unshuffle(<img x ,img ref >))>

[0062] Each ResNet only contains a convolution layer (Conv, which is a filter for extracting image features), and is suitable for extracting the foreground region in the input frame; the background region information in the input frame is almost completely derived from the background information in the high-definition background frame, first, the Res2DenseBlock needs to align the input frame img x with the reference frame img ref , identify the difference between the edges of the aligned input frame img x and the reference frame img ref , and then extract the background region feature.

[0063] Each group of Res2DenseBlock contains n DenseNets, n is generally 3, all the DenseNets are connected by using a residual formula, and λ is a residual gain, and the calculation formula is as follows:

[0064] feats i = DenseNet i (feats i,1)* lambda + feats i,1 , 3≤i≤n, lambda e (0.1, 0.2)

[0065] DenseNet module adopts multiple convolution layers to stack processing, m represents the number of layers, m is generally taken as 5-6 layers, and the processing results of all upper layers are input as part of the lower layer, and the formula is as follows:

[0066]

[0067] The final foreground and background feature separation module outputs <F pre , F back > is sent to the foreground and background feature fusion module for further processing.

[0068] In an embodiment of the application, the features separated from the foreground and background are fused and upsampled, The foreground features separated from the input frame and the background features of the reference frame can be fused together to maintain smooth edge transition, achieve clear background, and keep the overall tone consistent with the input frame.

[0069] The foreground and background feature fusion module Fusion includes multiple convolution layers, LeakyReLU activation layers (a kind of nonlinear function to improve the fitting ability of the model) and upsample operation (upsample), and the activation function is represented as:

[0070] alpha is a learnable parameter

[0071] The overall logic of the module can be represented as:

[0072] Fusion (feat) = LeakyReLU (Conv (upsample (feat)))

[0073]

[0074] The fusion frame is given to the target function optimization module for parameter update.

[0075] In an embodiment of the application, the target function optimization module uses the minimum reconstruction loss function between the fusion frame and the corresponding reference image sequence GT as the objective function, finds the optimal parameters theta of the foreground and background feature separation module Extract and the foreground and background feature fusion module Fusion through the gradient descent method, and the process is shown in the following formula:

[0076]

[0077] ​​​

[0078] During training, the parameters θ obtained through multiple iterations on a large amount of data can improve the fused frames. The quality index, the optimal parameter θ, yields the fused frame. It can be used as a background enhancement frame Y output.

[0079] In one embodiment of the present invention, the video sequence frame to be processed x and the corresponding background frame ref Foreground and background separation features are obtained through the optimized foreground and background feature separation module. <F pre ,F back The formula is as follows:

[0080] <F pre ,F back >=Extract( <frame x ,frame ref >)

[0081] Finally, the foreground and background separation features are analyzed. <F pre ,F back The optimized foreground and background feature fusion module will then handle the processing.

[0082] In one embodiment of the present invention, the optimized foreground and background feature fusion module is input with foreground and background separation features. <F pre ,F back The background enhancement frame sequence Y is obtained, and the entire process is as follows:

[0083] Y = Fusion( <F pre ,F back >).

[0084] The background quality improvement device under low code rate complex scene is set above, the training data preprocessing module is used for loading the foreground and background sample sequence to be trained and the reference background sequence, and the training data pair is obtained after data processing, and the training data pair is given to the foreground and background feature separation module and the target function optimization module; the inference data preprocessing module is used for reading the real-time video stream or the video saved in the storage medium and loading as the sequence to be processed, reading the reference frame in the real-time video stream or the reference image existing in the storage medium and loading as the reference frame, so as to obtain the inference data pair, and the inference data pair is given to the foreground and background feature separation module; the foreground and background feature separation module firstly calculates the feature difference between the foreground and the background between the input frame to be processed and the reference background frame, separates and extracts the foreground and background feature information, the foreground and background separation processing can improve the background quality while ensuring the correctness and integrity of the important foreground area, and finally the processed foreground and background features are transmitted to the foreground and background feature fusion module; the foreground and background feature fusion module fuses the separated foreground and background information, can complete the fusion of the high-quality background feature information in the reference frame and the foreground feature information in the frame to be processed, obtains the prediction frame, and finally the prediction frame is given to the target function optimization module for parameter updating or saved to the storage medium; the target function optimization module performs gradient descent optimization on the loss function between the image output by the foreground and background feature fusion module and the training sample provided by the training data preprocessing module, and is used for training the model. In the foreground and background feature separation, the feature difference between the foreground and the background between the input frame to be processed and the reference frame is fully utilized, and the foreground and background feature information is successfully separated and extracted; in the foreground and background feature fusion, the smooth fusion of the high-quality background feature information in the reference frame and the foreground feature information in the frame to be processed is completed; through the operation of the foreground and background feature separation and the foreground and background feature fusion, the processed video frame not only retains the correctness of the key information of the foreground area, but also enhances the background quality, and at the same time eliminates the incoherence of different foreground and background edges in fusion, meets the visual requirements. Not only can the transmission bandwidth be reduced under the condition of keeping the video quality unchanged, but also the background quality of the video stream reaching 4K size can be improved in real time under the condition of ultra-low transmission code rate and very poor video quality.

[0085] Referring to Figure 3 and Figure 4 , respectively, a low-quality high-compression-rate image before using the embodiment of the application and an image processed by the background quality improvement method under low code rate complex scene using the embodiment of the application, the effect image after the overall image quality is improved, the processed video frame not only retains the correctness of the key information of the foreground area, but also enhances the background quality, and at the same time eliminates the incoherence of different foreground and background edges in fusion.

[0086] It is to be understood that the example embodiments described herein are illustrative rather than limiting. Although one or more embodiments of the application are described in connection with the accompanying drawings, it will be understood that various modifications in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A method for improving background quality in a low bit rate complex scene, characterized in that, The training process further comprises: using a training data preprocessing module to process the foreground and background sample sequence to be trained and the reference background sequence into a model-applicable training data pair, the data pair comprising a low-quality training sequence, a corresponding high-quality background reference sequence, and a high-quality non-interference background in the corresponding scene as an input reference frame; using a foreground and background feature separation module to input the sequence as the low-quality training sequence and the input reference frame for 4-pixel de-reconstruction downsampling to obtain features, using M sets of double residual dense network series to extract features for separation of background features, the value range of M being 10-30, using a residual network for separation of foreground features, and outputting foreground and background separation features; using a foreground and background feature fusion module to perform fusion and upsampling operations on the foreground and background separation features to obtain a fusion frame; using a target function optimization module to take the minimum reconstruction loss function between the fusion frame and the corresponding background reference sequence as a target function, finding the optimal parameters of the foreground and background feature separation module and the foreground and background feature fusion module through gradient descent method, and then obtaining the optimized foreground and background feature separation module and the foreground and background feature fusion module; The inference process further comprises: loading a video sequence to be processed and a reference background frame sequence in the corresponding scene from a video stream or a storage medium, using an inference data preprocessing module to combine the loaded video sequence and the reference background frame sequence into a data pair; using the optimized foreground and background feature separation module to obtain foreground and background separation features from the data pair obtained by combining the video sequence and the reference background frame sequence; using the optimized foreground and background feature fusion module to input the foreground and background separation features to obtain a background enhancement frame sequence, i.e., a processed video. Each residual network comprises a convolutional layer for extracting a foreground region in a low-quality training sequence obtained by the training data preprocessing module or a video sequence obtained by the inference data preprocessing module.

2. The method of claim 1, wherein the background quality is improved in a low bit rate complex scene. The double residual dense network is used to align the training sequence and the input reference frame, identify the difference between the edges of the training sequence and the input reference frame, and then extract background region features.

3. The method of claim 1, wherein the background quality is improved in a low bit rate complex scene. The foreground and background feature fusion module comprises a plurality of convolutional layers, LeakyReLU activation layers, and upsampling operations.

4. The method of claim 1, wherein the background quality is improved in a low bit rate complex scene. The training data preprocessing module is configured to process the foreground and background sample sequence to be trained and the reference background sequence into a model-applicable training data pair, the data pair comprising a low-quality training sequence, a corresponding high-quality background reference sequence, and a high-quality non-interference background in the corresponding scene as an input reference frame; the foreground and background feature separation module is configured to input the sequence as the low-quality training sequence and the input reference frame for 4-pixel de-reconstruction downsampling to obtain features, use M sets of double residual dense network series to extract features for separation of background features, the value range of M being 10-30, directly use a residual network for separation of foreground features, and output foreground and background separation features; 5. A device for improving background quality in a low bit rate complex scene, characterized in that, the foreground and background feature fusion module is configured to perform fusion and upsampling operations on the foreground and background separation features to obtain a fusion frame; ​ ​ ​ The target function optimization module is configured to take the minimized reconstruction loss function between the fusion frame and the corresponding background reference sequence as a target function, find the optimal parameters of the foreground and background feature separation module and the foreground and background feature fusion module through the gradient descent method, and then obtain the optimized foreground and background feature separation module and the foreground and background feature fusion module. The inference data preprocessing module is configured to load a video sequence to be processed and a reference background frame sequence in a corresponding scene from a video stream or a storage medium, and combine the loaded video sequence and the reference background frame sequence into a data pair; and the optimized foreground and background separation feature is configured to process the data pair of the video sequence and the reference background frame sequence to obtain the foreground and background separation feature. The optimized foreground and background feature fusion module is configured to process the foreground and background separation feature to obtain a background enhancement frame sequence, that is, a processed video.

6. The device for improving background quality in a low bit rate complex scene according to claim 5, wherein, Each residual network includes a convolutional layer configured to extract a foreground region in a low-quality training sequence obtained by the training data preprocessing module or a video sequence obtained by the inference data preprocessing module.

7. The device for improving background quality in a low bit rate complex scene according to claim 5, wherein, The background region information in the input frame is derived from the background information in the high-definition background frame. The input frame and the reference frame are aligned using the double residual dense network, the differences between the edges of the input frame and the reference frame are identified, and then the background region feature is extracted.

8. The device for improving background quality in a low bit rate complex scene according to claim 5, wherein, The foreground and background feature fusion module includes a plurality of convolutional layers, LeakyReLU activation layers, and up-sampling operations.

Images