Data enhancement method for optimizing robustness of background quality in enhanced lens shake scene

Abstract

This invention discloses a data augmentation method for enhancing the robustness of background quality optimization in scenes with camera shake, comprising: S10, obtaining a background sequence. <bgr>and foreground sequence and mask sequence <fgr, pha>; S20, from the background sequence <bgr>If a frame bgr1 is randomly selected from the dataset, and bgr1 has no other relevant background frames, then an affine transformation is directly performed on bgr1 to obtain bgr2; if bgr1 is in... <bgr>When there are correlated frames, first randomly select a frame from all correlated sequences, then perform an affine transformation operation to obtain bgr2; thus obtaining the background pair.<bgr1,bgr2> S30, combine foreground and background sequences; S40, process the reference background sequence gt. bgr Perform augmentation transformation on the training data to obtain training data pairs. <lq,gt bgr S50, training input sequence l q and reference background sequence gt bgr After processing by the background quality optimization module, the background-enhanced fusion sequence S60 is obtained. The parameter θ, obtained through multiple iterations of training on a large amount of data, is used to improve the quality index of the fusion sequence.< / bgr> < / bgr> < / bgr>

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a data augmentation method for enhancing the robustness of background quality optimization in scenes with camera shake. Background Technology

[0002] In real life, in order to reduce the cost of video transmission bandwidth, it is necessary to use methods such as reducing the transmission bit rate and video compression. As a result, the video quality at the playback end is poor, details are lost, and it is not conducive to viewing.

[0003] The proposed background quality optimization algorithm addresses this problem by optimizing the background of poor-quality videos, achieving both low transmission costs and maintaining the integrity of video details at the playback end. This algorithm can save more transmission resources to improve the quality of foreground details. Background optimization can be prioritized less during transmission, with quality enhancement performed only at the receiving end, minimizing the loss of background details.

[0004] This algorithm is suitable for surveillance videos, where the camera lens needs to be kept fixed; otherwise, the processed video frames will have background distortion and other issues. On the other hand, the algorithm has high requirements for the image quality of the reference background frame; if a reference frame of insufficient quality is provided, the processing effect will not be ideal. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention provides a data augmentation method to enhance the robustness of background quality optimization in lens shake scenarios. This method can still achieve a significant improvement in background quality based on a reference background frame even when the video footage is shaken, distorted, or blurred due to lens shake, without losing foreground details, thus improving the robustness of the background quality optimization algorithm.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A data augmentation method for enhancing the robustness of background quality optimization in camera shake scenarios includes:

[0008] S10: Acquire video frames or images (fgr) and corresponding masks (pha) including the foreground region, as well as high-quality background video frames or images (bgr). After obtaining the foreground and background sequences and masks, the background sequence is obtained. <bgr>and foreground sequence and mask sequence<fgr,pha> ;

[0009] S20, from the background sequence <bgr>Randomly select a frame bgr1 from the data, and then... <bgr>The process involves searching for other relevant background frames within bgr1. If bgr1 has no other relevant background frames, an affine transformation is directly performed on bgr1 to obtain bgr2. If bgr1 is within a certain range, the process continues. <bgr>When there are correlated frames, first randomly select a frame from all correlated sequences, then perform an affine transformation operation to obtain bgr2; thus obtaining the background pair.<bgr1,bgr2> ;

[0010] S30, perform foreground and background sequence combination, input data includes...<fgr,pha> and<bgr1,bgr2> If the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence If the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence ;

[0011] S40, for the reference image sequence Perform augmentation transformation on the training data to obtain training data pairs. >;

[0012] S50, training input sequence lq and reference background sequence After processing by the background quality optimization module, the background-enhanced fusion sequence is obtained.

[0013] S60, fusion sequence Minimize the reconstruction loss function between the corresponding baseline sequence gt and the target sequence. The optimal parameters of the background quality optimization module are found using gradient descent as the objective function. During the training process, the parameters are obtained through multiple iterations of training on a large amount of data. Used to improve fusion sequences Quality indicators;

[0014] S70, the fusion sequence obtained by the optimized background quality enhancement module. The background enhancement sequence Y is output as a video stream.

[0015] Preferably, in S20, the affine transformation includes at least one of cropping, flipping, scaling, stretching, rotating, translating, color transformation, and contrast enhancement.

[0016] Preferably, in S30, if the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence The combination formula is as follows:

[0017]

[0018] .

[0019] Preferably, in S50, the reference sequence gt is obtained in the following way:

[0020] .

[0021] Preferably, in S30, if the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence The combination formula is as follows:

[0022]

[0023] .

[0024] Preferably, in S50, the reference sequence gt is obtained in the following way:

[0025] .

[0026] Preferably, in S40, the data augmentation transformation uses multi-stage cascading, with each stage's operation called... The process involves sequentially performing blurring, scaling, noise addition, and compression operations, where k represents the blur kernel. This represents the convolution operation, where r represents the scaling ratio, n represents the added noise, and JPEG represents compression. The specific formula is as follows:

[0027]

[0028] The final Transforms consist of n Cascading, where n is at least two, follows the formula:

[0029] .

[0030] The present invention has the following beneficial effects:

[0031] (1) In the affine transformation of the background sequence, the correlation of the background sequence is processed separately, which not only increases the authenticity of the training data, but also expands the scale of the training data, which can improve the overall training effect.

[0032] (2) By using a variety of special combination methods in the combination of foreground and background sequences, the relative movement of background and foreground in real-world video scenes or the overall shaking and distortion of the image can be simulated, thereby improving the robustness of the model. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the steps of a data augmentation method for enhancing the robustness of background quality optimization in scenarios with improved lens shake, as described in an embodiment of the present invention.

[0034] Figure 2 This is a low-quality, high-compression-ratio image from a specific application example.

[0035] Figure 3 This is a reference background image for a specific application example;

[0036] Figure 4 The image shows the effect of the background quality optimization algorithm before improvement in a specific application example.

[0037] Figure 5 This is a diagram showing the effect of using the improved method of this invention in a specific application example. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Reference Figure 1 The image shows a data augmentation method for enhancing the robustness of background quality optimization in scenarios with lens shake, according to an embodiment of the present invention, comprising:

[0040] S10: Acquire video frames or images (fgr) and their corresponding masks (pha) for the foreground region, as well as high-quality background video frames or images (bgr). These can be obtained from the internet or other platforms. After obtaining the foreground and background sequences and masks, the background sequence is obtained. <bgr>and foreground sequence and mask sequence<fgr,pha> Each of these<fgr,pha> Each of these can uniquely identify a foreground area. <bgr>The background frames can be consecutive, related background frames in a video sequence, or they can be independent background images that are not related at all.

[0041] S20, from the background sequence <bgr>Randomly select a frame bgr1 from the data, and then... <bgr>The process involves searching for other relevant background frames within bgr1. If bgr1 has no other relevant background frames, an affine transformation is directly performed on bgr1 to obtain bgr2. If bgr1 is within a certain range, the process continues. <bgr>When there are correlated frames, first randomly select a frame from all correlated sequences, then perform an affine transformation operation to obtain bgr2; thus obtaining the background pair.<bgr1,bgr2> Here, correlation refers to the temporal continuity or spatial similarity between sequences.

[0042] The problem that the S20 aims to solve is that, due to camera shake in real-world scenarios, the background in the captured video may sway, and foreground objects may move, causing the background to be obscured to varying degrees and angles. Furthermore, the background between the frame being processed and the provided reference background frame may become misaligned due to camera shake. Therefore, a reference background with slight displacement, distortion, or rotation is obtained through transformation.<bgr1,bgr2> .

[0043] S30, perform foreground and background sequence combination, input data includes...<fgr,pha> and<bgr1,bgr2> If the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence If the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence ;

[0044] S40, for the reference image sequence Perform augmentation transformation on the training data to obtain training data pairs. >;

[0045] S50, training input sequence lq and reference background sequence After processing by the background quality optimization module, the background-enhanced fusion sequence is obtained.

[0046] S60, fusion sequence Minimize the reconstruction loss function between the corresponding baseline sequence gt and the target sequence. The optimal parameters of the background quality optimization module are found using gradient descent as the objective function. During the training process, the parameters are obtained through multiple iterations of training on a large amount of data. Used to improve fusion sequences Quality indicators;

[0047] S70, the fusion sequence obtained by the optimized background quality enhancement module. The background enhancement sequence Y is output as a video stream.

[0048] In one embodiment of the present invention, in S20, the affine transformation includes at least one of cropping, flipping, scaling, stretching, rotating, translating, color transformation, and contrast enhancement.

[0049] In one embodiment of the present invention, in S30, if the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence The combination formula is as follows:

[0050]

[0051] .

[0052] In response to the above situation, in S50, the reference sequence gt is obtained as follows:

[0053] .

[0054] In one embodiment of the present invention, in S30, if the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence The combination formula is as follows:

[0055]

[0056] .

[0057] In response to the above situation, in S50, the reference sequence gt is obtained as follows:

[0058] .

[0059] In one embodiment of the present invention, a data augmentation transformation module is used to process the received data. After performing data augmentation transformations, we obtain the training sequence lq, i.e.:

[0060]

[0061] Transforms operations are necessary processing steps performed to simulate the image quality of video sequences that have undergone various compression processes, transmission channel noise interference, and other noise introduced at the receiving end.

[0062] In S40, the data augmentation transformation uses a multi-stage cascaded approach, with each stage's operation called... The process involves sequentially performing blurring, scaling, noise addition, and compression operations, where k represents the blur kernel. This represents the convolution operation, where r represents the scaling ratio, n represents the added noise, and JPEG represents compression. The specific formula is as follows:

[0063]

[0064] The final Transforms consist of n Cascading, where n is at least two, follows the formula:

[0065] .

[0066] In one embodiment of the present invention, the background quality optimization module Enhance includes multiple steps such as feature separation and feature refusion, and trains the input sequence lq and the reference background sequence. After processing by the background quality optimization module, a fused sequence with enhanced background can be obtained. The entire process can be represented as follows:

[0067] .

[0068] In one embodiment of the present invention, the fusion sequence Minimize the reconstruction loss function between the corresponding baseline sequence gt and the target sequence. As the objective function, the optimal parameters of the background quality optimization module Enhance are found using gradient descent. The process is shown in the following formula:

[0069]

[0070]

[0071] The parameters are obtained through multiple iterations of training on a large amount of data during the training process. It can improve the fusion sequence Quality indicators.

[0072] See Figures 2 to 5 A specific application example is used to illustrate the effect comparison after the implementation of the method of the present invention. Figure 2 This is a low-quality, high-compression-ratio image from a specific application example. Figure 3 This is a reference background image for a specific application example; Figure 4 The image shows the effect of the background quality optimization algorithm before improvement in a specific application example. The image quality is improved, but because the car window in the reference background image has a reflection that does not actually exist, the car window in the optimized image does not recognize this incorrect reflection. Figure 5 This is a specific application example of the improved effect of the method of this invention. It can be seen that the overall image quality is improved without being disturbed by the incorrect reflection on the car window in the reference background image.

[0073] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.< / bgr> < / bgr> < / bgr> < / bgr> < / bgr> < / bgr> < / bgr> < / bgr> < / bgr>

Claims

1. A data augmentation method for enhancing the robustness of background quality optimization in scenes with camera shake, characterized in that, include: S10: Acquire video frames or images (fgr) and corresponding masks (pha) including the foreground region, as well as high-quality background video frames or images (bgr). After obtaining the foreground and background sequences and masks, the background sequence is obtained. <bgr> and foreground sequence and mask sequence<fgr,pha> ;< / bgr> S20, from the background sequence <bgr>Randomly select a frame bgr1 from the data, and then... <bgr>The process involves searching for other relevant background frames within bgr1. If bgr1 has no other relevant background frames, an affine transformation is directly performed on bgr1 to obtain bgr2. If bgr1 is within a certain range, the process continues. <bgr> When there are correlated frames, first randomly select a frame from all correlated sequences, then perform an affine transformation operation to obtain bgr2; thus obtaining the background pair.<bgr1,bgr2> ;< / bgr> < / bgr> < / bgr> S30, perform foreground and background sequence combination, input data includes...<fgr,pha> and<bgr1,bgr2> If the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence If the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence ; S40, for the reference image sequence Perform augmentation transformation on the training data to obtain training data pairs. >; S50, training input sequence lq and reference background sequence After processing by the background quality optimization module, the background-enhanced fusion sequence is obtained. S60, fusion sequence Minimize the reconstruction loss function between the corresponding baseline sequence gt and the target sequence. The optimal parameters of the background quality optimization module are found using gradient descent as the objective function. ; The parameters are obtained through multiple iterations of training on a large amount of data during the training process. Used to improve fusion sequences Quality indicators; S70, the fusion sequence obtained by the optimized background quality enhancement module. The background enhancement sequence Y is output as a video stream. In S30, if the foreground does not need to exist in the reference background frame, only one copy is needed.<fgr,pha> The base image sequence is obtained by combining the data. With reference background sequence The combination formula is as follows: ； In S40, the method for obtaining the reference sequence gt is as follows: ； In S30, if the foreground exists in the reference background frame, in<fgr,pha> Select two foreground sequences that have foreground relevance.<fgr1,pha1,fgr2,pha2> Then, the reference image sequence is obtained through combination. With reference background sequence The combination formula is as follows: ； In S40, the method for obtaining the reference sequence gt is as follows: ； In S40, lq is the gt received by the data augmentation transformation module src The to-be-trained sequence obtained after data augmentation transformation.

2. The data augmentation method for enhancing the robustness of background quality optimization in scenes with lens shake as described in claim 1, characterized in that, In S20, the affine transformation includes at least one of cropping, flipping, scaling, stretching, rotating, translating, color transformation, and contrast enhancement.

3. The data augmentation method for enhancing the robustness of background quality optimization in scenes with lens shake as described in claim 1, characterized in that, In S40, the data augmentation transformation uses a multi-stage cascade, with each stage performing blurring, scaling, noise addition, and compression operations in sequence.

Images