A video behavior data enhancement method based on sliding mask

Abstract

The present invention proposes a video behavior data enhancement method based on a sliding mask. This method first enhances the original video behavior training samples to construct new enhanced samples. The specific implementation method is as follows: first, continuous video images are randomly extracted from the entire data set and preprocessed as training samples; then, the training round information of the model training is obtained, and the trigger probability of the data enhancement algorithm is calculated; then, based on the trigger probability, it is determined whether to perform data enhancement on the training samples; if it is determined that enhancement is not necessary, the training samples are directly used to train the video behavior recognition model; if it is determined that enhancement is necessary, the length, width, and number of frames of the training samples are obtained, and a corresponding random mask is generated accordingly; finally, the random mask is merged with the training samples to obtain enhanced samples, and finally, the enhanced samples are used to train the video behavior recognition model. Compared with the existing technology, the present invention has the advantages of good enhancement effect, excellent efficiency, and simplicity.

Description

Technical Field

[0001] The present invention relates to the technical field of data enhancement and deep learning, and particularly to a video behavior data enhancement method based on sliding masks. Background Art

[0002] Recently proposed data augmentation methods have been used to address the overfitting problem of neural networks and have gradually become a research focus in deep learning. These data augmentation methods have been widely applied in tasks such as image recognition, object detection, and image segmentation. However, existing data augmentation methods for overfitting neural networks on video data are limited by feature dimensionality and can only affect the spatial characteristics of training samples. Furthermore, existing methods lack the ability to filter the temporal information of training samples. After data augmentation, training samples still contain a significant amount of redundant information, and video action recognition models are still susceptible to overfitting.

[0003] Recent research shows that "Cubuk ED, Zoph B, Mane D, et al. AutoAugment: Learning Augmentation Strategies From Data[C] / / 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2019." improves the preprocessing method and uses reinforcement learning to search existing strategies to find the optimal combination. However, it takes a long search time and often has a significant negative impact on training. There are also data enhancement methods based on information deletion, such as "Devries T, Taylor GW. Improved Regularization of Convolutional Neural Networks with Cutout[J]. CoRR, abs / 1708.04552, 2017.", which can randomly generate square deletion blocks with zero pixel values on training samples, and "Zhong Z, Zheng L, Kang G, et al. Random Erasing Data Augmentation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 34(7).", which can randomly generate rectangular deletion blocks with different pixel values on training samples. The regularization effect of the above two methods is weak, and the effect on complex models is limited. "Singh KK, Hao Y, Sarmasi A, et al. Hide-and-Seek: A DataAugmentation Technique for Weakly-Supervised Localization and Beyond[J].2018.CoRR,abs / 1811.02545,2018.》Generate a mask consisting of a block of pixels consisting of zeros and ones. The mask is divided into multiple rectangular regions. Within each region, randomly delete regions with zero pixels. The mask is combined with the training samples to generate enhanced samples. This method has stronger regularization capabilities, but has little impact on the temporal dimension of three-dimensional data. By deleting information about the input image, these methods allow convolutional neural networks to learn information that was initially less sensitive or important, thereby increasing the receptive field. However, these data augmentation methods also have some shortcomings. Random deletion only affects the spatial characteristics of video data and cannot effectively take into account the temporal characteristics of video data. They also generally have limitations in terms of feature dimensions. Summary of the Invention

[0004] The present invention discloses a video behavior data enhancement method based on a sliding mask, comprising the following steps:

[0005] (1) Extract continuous video images from a data set; (2) Perform size transformation, random cropping, and normalization on the continuous video images to obtain training samples; (3) Obtain the training round and the maximum training round and calculate the trigger probability of the enhancement algorithm; (4) If the random probability is less than the trigger probability, return the training sample and send it to the model for training; (5) If the random probability is greater than the trigger probability, the algorithm starts to construct a sliding mask and synthesizes the training sample and the corresponding sliding mask to obtain an enhanced sample; (6) Send the enhanced sample to the video behavior recognition model for training; The video behavior data enhancement method disclosed in the present invention is applicable to different video understanding tasks and has achieved good enhancement effects on multiple data sets. While reducing the overfitting problem of the network model, it can also improve the robustness of the network model.

[0006] The technical solution provided by the present invention is: a video behavior data enhancement method based on sliding mask, comprising the following steps:

[0007] 1. A video behavior data enhancement method based on sliding mask, which is characterized by the following steps:

[0008] Step 1: Create training sample X. The specific steps are as follows:

[0009] (1) Randomly select n video samples X′ with a frame number of f from the dataset, where f∈(1, 64]; X′ is a c×f a×b matrix, where a∈(224, 1080); b∈(224, 1080); c is the number of channels of the video image, c∈{1, 3};

[0010] (2) Perform resizing on n video samples X′ with a frame number of f to obtain resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values of 224; then perform maximum and minimum value normalization on X″ to obtain normalized video samples X″′, where X″′ is a c×f w×h matrix; the calculation formula is as follows:

[0011]

[0012] Where x″′ wh Represents a single element in the matrix X″′, x″′ wh ∈(-1, 1); x″′ wh Represents a single element in the matrix X″′, x″ wh ∈(0, 255); x min represents the normalized minimum value, x min =-1;x maxrepresents the normalized maximum value, x max =1;

[0013] (3) Finally, the n normalized video samples X″′ are grouped into a set X as training samples, X = {X″′1, X″′2, X″′3, …, X″′ n}; The training sample X is an n×c×f w×h matrix; n is the number of normalized video samples X″′ in the training sample X, n∈N;

[0014] Step 2: Data enhancement algorithm is based on the training round e i and the maximum number of training rounds e m Calculate the trigger probability p i , and get each frame image x in the training sample X i and the corresponding mask M i , the specific steps are as follows:

[0015] (1) According to the training round e of the model i , and the maximum number of training rounds e m , calculate the trigger probability p of the data enhancement algorithm i , the calculation formula of trigger probability is:

[0016]

[0017] Where p i is the trigger probability of the i-th round of data enhancement algorithm, p i ∈(0,1];e m represents the maximum number of training rounds, e m ∈N;e i represents the current i-th round of training, e i ∈(1,e m ];

[0018] (2) Set the random probability q of the i-th training round i ,q i =random(0,1); when the random probability q of the i-th training round i Greater than the trigger probability p i When the algorithm uses three parameters l, j, r to generate the mask M1, M1 is composed of c identical w×h matrices, m ik Represents the elements of the i-th row and k-th column of the c identical w×h matrices in mask M1; m ik ∈{0, 1}, i∈(0, 224), k∈(0, 224); m in mask M1 ik The calculation formula is as follows:

[0019]

[0020] M1 is the mask corresponding to the first frame image x1 in the training sample X, and M1 is a c w×h matrix; m wh ∈{0, 1}, m wh Represents a single element in the mask M1; l represents the number of grids in the mask M1, l∈[4, 14]; j represents the width of the grid in the mask M1, j∈[16, 56]; r represents the angle of random rotation of the mask M1, r∈(1°, 360°]; the calculation formula of the three parameters is:

[0021] l=rand(l min , l max )

[0022] j=random(j min ,j max )

[0023] r = random(1°, 360°)

[0024] Where j max Represents the upper bound of the random value of the width j of the grid in M1, j max ∈[16, 56]; j min Represents the lower bound of the random value of the width j of the grid in M1, j min ≤j max ;j min ,j max The calculation formula is:

[0025]

[0026]

[0027] Where h represents the height of the mask M1, l represents the number of grids in the mask M1;

[0028] (3) The data enhancement algorithm performs a multi-step process on each frame image x in the training sample X. i and the corresponding mask M i To synthesize, M i is c w×h matrices; mask M i The calculation formula is:

[0029]

[0030] Where i is the i-th frame image in the training sample X, w and h represent the mask M i The length and width, p i is the trigger probability of the i-th round of data enhancement algorithm, f is the number of frames of training sample X, w d Represents mask M i The horizontal coordinate range of the deleted area, w d ∈(0,w);hd Represents mask M i The vertical coordinate range of the deleted area, h d ∈(0, h); (w1, h1) represents the upper left corner coordinate of the deleted area in mask M1, (w2, h2) represents the lower right coordinate of the deleted area in mask M1, {w1, h1, w2, h2}∈(0, 224); d w d h Represents mask M i The moving distance, d w d h The calculation formula is as follows:

[0031]

[0032]

[0033] Where i is the i-th frame image in the training sample X, and f is the number of frames of the training sample X; Step 3, use the mask M generated in step 2 i Synthesize with the training sample X to obtain the enhanced sample The specific steps are as follows:

[0034] (1) When the random probability q of the i-th training round i ,q i =random(0,1); when the random probability q of the i-th training round i Greater than the trigger probability p i When the algorithm performs data enhancement on n normalized video samples X″′, it obtains enhanced video samples is a c×f w×h matrix; The calculation formula is as follows:

[0035]

[0036] Where f is the number of frames of the normalized video sample X″′, Z i ″′; i = {1, 2, ..., f}, represents the i-th frame image in the normalized video sample X″′, Z i ″′ is c w×h matrices; M i ; i = {1, 2, ..., f}, representing the i-th frame image Z in the normalized video sample X″′ i ″′ multiplied mask, M i is c w×h matrices;

[0037] (2) For n enhanced video samples Combined enhanced samples The calculation formula of the enhancement algorithm is:

[0038]

[0039] Where q i is the random probability of the i-th training round, q i =random(0,1);p i is the trigger probability of the enhancement algorithm in the i-th round of training; X is the training sample, and the training sample X is an n×c×f w×h matrix; Represents enhanced video sample, enhanced video sample is a c×f w×h matrix, represents the enhanced sample, is n enhanced video samples A collection of The innovation of the present invention is as follows:

[0040] (1) Compared with methods such as Cutout, Randomearsing, GridMask, and HaS, the aforementioned enhancement methods all use a fixed trigger probability. The difference of the data enhancement method of the present invention is that the trigger probability is associated with the training round and has an independent calculation method;

[0041] (2) Compared with methods such as Cutout, Randomearsing, GridMask, and HaS, which all use random spatial masks, the data enhancement method of the present invention is different in that it uses a spatiotemporal sliding mask. The mask generated by the present invention is correlated in the temporal and spatial dimensions.

[0042] Beneficial effects:

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention studies the different learning characteristics of deep learning models at different stages of training, designs round-related trigger probabilities, and enhances the algorithm to adjust the regularization effect in real time according to the different stages of model training, making the enhancement method of the present invention more robust;

[0045] (2) The method of moving the spatiotemporal mask proposed in the present invention can better adapt to the spatiotemporal characteristics of video data, produce a better regularization effect in the training of the video behavior recognition model, and improve the recognition accuracy of the video behavior recognition model; BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the specific implementation of the present invention. The main process is to establish training samples, then calculate the trigger probability, generate a mask by algorithm, synthesize the image and mask to obtain an enhanced sample, and finally perform model training;

[0047] Figure 2The comparison of the enhancement method of the present invention with other enhancement methods is shown in the figure: a) original image; b) Cutout effect diagram, c) Randomearsing effect diagram, d) effect diagram of the present invention, e) GridMask effect diagram, and f) HaS effect diagram.

[0048] Figure 3 The initial mask shape generated by the present invention is composed of black horizontal and vertical stripes with zero pixels rotated;

[0049] Figure 4 This is a comparison of different mask movement methods in three-dimensional space used in the present invention: a) Random-Spatial; b) Random Temporal-Spatial, c) Temporal-Spatial, and d) Shift Temporal-Spatial used in the present invention.

[0050] Figure 5 This is the enhancement effect of the present invention on the training samples, specifically, mask synthesis is performed on a batch of pictures to obtain new enhanced samples;

[0051] The symbols in the figure are explained as follows:

[0052] p: indicates the trigger probability

[0053] X: represents the training sample;

[0054] : represents the enhanced sample;

[0055] l: represents the number of grids in the mask;

[0056] j: represents the width of the grid in the mask;

[0057] r: represents the angle of random rotation of the mask; DETAILED DESCRIPTION

[0058] The present invention is a video behavior data enhancement method based on sliding mask. The technical solution of the present invention is further described below with reference to the accompanying drawings. Figure 1 , the flowchart of the present invention includes the following steps:

[0059] 1. A video behavior data enhancement method based on sliding mask, which is characterized by the following steps:

[0060] Step 1: Create training sample X. The specific steps are as follows:

[0061] (1) Randomly select n video samples X′ with a frame number of f from the dataset, where f∈(1, 64]; X′ is a c×f a×b matrix, where a∈(224, 1080); b∈(224, 1080); c is the number of channels of the video image, c∈{1, 3};

[0062] (2) Perform resizing on n video samples X′ with a frame number of f to obtain resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values of 224; then perform maximum and minimum value normalization on X″ to obtain normalized video samples X″′, where X″′ is a c×f w×h matrix; the calculation formula is as follows:

[0063]

[0064] Where x′ w ″ h Represents a single element in the matrix X″′, x′ w ″ h ∈(-1, 1); x′ w ' h Represents a single element in the matrix X″′, x″ wh ∈(0, 255); x min represents the normalized minimum value, x min =-1;x max represents the normalized maximum value, x max =1;

[0065] (3) Finally, the n normalized video samples X″′ are grouped into a set X as training samples, X = {X″′1, X″′2, X″′3, …, X″′ n}; The training sample X is an n×c×f w×h matrix; n is the number of normalized video samples X″′ in the training sample X, n∈N;

[0066] Step 2: Data enhancement algorithm is based on the training round e i and the maximum number of training rounds e m Calculate the trigger probability p i , and get each frame image x in the training sample X i and the corresponding mask M i , the specific steps are as follows:

[0067] (1) According to the training round e of the model i , and the maximum number of training rounds e m , calculate the trigger probability p of the data enhancement algorithm i , the calculation formula of trigger probability is:

[0068]

[0069] Where p i is the trigger probability of the i-th round of data enhancement algorithm, p i ∈(0,1];e m represents the maximum number of training rounds, e m ∈N;e i represents the current i-th round of training, e i ∈(1,e m ];

[0070] (2) Set the random probability q of the i-th training round i ,q i =random(0,1); when the random probability q of the i-th training round i Greater than the trigger probability p i When the algorithm uses three parameters l, j, r to generate the mask M1, M1 is composed of c identical w×h matrices, m ik Represents the elements of the i-th row and k-th column of the c identical w×h matrices in mask M1; m ik ∈{0, 1}, i∈(0, 224), k∈(0, 224); m in mask M1 ik The calculation formula is as follows:

[0071]

[0072] M1 is the mask corresponding to the first frame image x1 in the training sample X, and M1 is a c w×h matrix; m wh ∈{0, 1}, m wh Represents a single element in the mask M1; l represents the number of grids in the mask M1, l∈[4, 14]; j represents the width of the grid in the mask M1, j∈[16, 56]; r represents the angle of random rotation of the mask M1, r∈(1°, 360°]; the calculation formula of the three parameters is:

[0073] l=rand(l min , l max )

[0074] j=random(j min ,j max )

[0075] r = random(1°, 360°)

[0076] Where j max Represents the upper bound of the random value of the width j of the grid in M1, j max ∈[16, 56]; j min Represents the lower bound of the random value of the width j of the grid in M1, j min ≤j max ;j min ,jmax The calculation formula is:

[0077]

[0078]

[0079] Where h represents the height of the mask M1, l represents the number of grids in the mask M1;

[0080] (3) The data enhancement algorithm performs a multi-step process on each frame image x in the training sample X. i and the corresponding mask M i To synthesize, M i ∈M C×W×H The present invention has explored various masking methods in the examples, such as Figure 4 The first one is random spatial mask. Figure 4 (a), the mask is randomly generated in the time and space dimensions, and its calculation formula is as follows:

[0081]

[0082] The second type is random spatiotemporal masking. Figure 4 (b) Generate a cube mask of variable length on the video segment. The calculation formula is as follows:

[0083]

[0084] The third type is space-time masking. Figure 4 (c) Generates a fixed-size cube mask that runs through the entire video. The calculation formula is as follows:

[0085]

[0086] The fourth type is the moving spatiotemporal mask proposed and used in the present invention. Figure 4 (d) First, determine the starting point and end point of the deletion block on the image, and then move it evenly in the spatial dimension to generate a continuous mask area, mask M i The calculation formula is:

[0087]

[0088] Where i is the i-th frame image in the training sample X, w and h represent the mask M i The length and width, p i is the trigger probability of the i-th round of data enhancement algorithm, f is the number of frames of training sample X, and w d Represents mask M i The horizontal coordinate range of the deleted area, w d ∈(0,w);h dRepresents mask M i The vertical coordinate range of the deleted area, h d ∈(0, h); (w1, h1) represents the upper left corner coordinate of the deleted area in mask M1, (w2, h2) represents the lower right coordinate of the deleted area in mask M1, {w1, h1, w2, h2}∈(0, 224); d w d h Represents mask M i The moving distance, d w d h The calculation formula is as follows:

[0089]

[0090]

[0091] Where i is the i-th frame image in the training sample X, and f is the number of frames of the training sample X;

[0092] Step 3: Use the mask M generated in step 2 i Synthesize with the training sample X to obtain the enhanced sample The specific steps are as follows:

[0093] (1) When the random probability q of the i-th training round i ,q i =random(0,1); when the random probability q of the i-th training round i Greater than the trigger probability p i When the algorithm performs data enhancement on n normalized video samples X″′, it obtains enhanced video samples is a c×f w×h matrix; The calculation formula is as follows:

[0094]

[0095] Where f is the number of frames of the normalized video sample X″′, Z i ″′; i = {1, 2, ..., f}, represents the i-th frame image in the normalized video sample X″′, Z i ″′ is c w×h matrices; M i ; i = {1, 2, ..., f}, representing the i-th frame image Z in the normalized video sample X″′ i ″′ multiplied mask, M i is c w×h matrices;

[0096] (2) For n enhanced video samples Combined enhanced samples The calculation formula of the enhancement algorithm is:

[0097]

[0098] Where q i is the random probability of the i-th training round, q i =random(0,1);p i is the trigger probability of the enhancement algorithm in the i-th round of training; X is the training sample, and the training sample X is an n×c×f w×h matrix; Represents enhanced video sample, enhanced video sample is a c×f w×h matrix, represents the enhanced sample, is n enhanced video samples A collection of It is an n×c×f w×h matrix.

[0099] (2) Enhance the sample Input into the I3D behavior recognition network model;

[0100] (3) Error back propagation is used, and the cross entropy loss function is used to measure the distance between the true value and the predicted value. Through each iteration of the data set, the SGD stochastic gradient descent function is used to adjust the weights between neurons to reduce the cost function and thus optimize the network;

[0101] (3) Save the parameter with the highest Acc∈(0,1) during model training;

[0102]

[0103] Where TP is the number of positive samples predicted correctly by the model, TN is the number of negative samples predicted correctly by the model, FP is the number of positive samples predicted incorrectly by the model, and FN is the number of negative samples predicted incorrectly by the model.

[0104] (4) Demonstration of the effect of the implementation method of the present invention. The following table presents the optimization effect of the present invention on the model recognition rate, as shown in Table 1:

[0105] Table 1 Improvement of recognition rate of various models by our method

[0106] Tab.1 Ourmethod improves cognitionrate of various models

[0107]

[0108]

[0109] As can be seen from the above table, after using the data enhancement method of the present invention, the recognition accuracy of models such as I3D, S3D, R3D, T3D, R2+1D, CSN26, and SlowFast are all improved. The following Table 2 shows the comparison of the optimization effects of the present invention and different data enhancement methods on the three models of I3D, S3D, and T3D, as shown in the following table:

[0110] Table 2 Accuracy of different approaches on the HMDB51 dataset

[0111]

[0112] As can be seen from the above table, the data enhancement method proposed in this invention has an effect on improving the recognition accuracy of I3D, S3D, T3D and other models.

Claims

1. A video behavior data enhancement method based on sliding mask, characterized by the following steps: Step 1: Create training sample X. The specific steps are as follows: (1) Randomly select n video samples X′ with a frame number of f from the dataset, where f∈(1, 64]; X′ is a c×f a×b matrix, where a∈(224, 1080); b∈(224, 1080); c is the number of channels of the video image, c∈{1, 3}; (2) Perform resizing on n video samples X′ with a frame number of f to obtain resized video samples X″, where X″ is c×f w×h matrices; w and h are fixed values of 224; then perform maximum and minimum value normalization on X″ to obtain normalized video samples X″′, where X″′ is a c×f w×h matrix; the calculation formula is as follows: Where x″' wh Represents a single element in the matrix X″′, x″′ wh ∈(-1, 1); x″' wh Represents a single element in the matrix X″′, x″ wh ∈(0, 255); x min represents the normalized minimum value, x min =-1;x max represents the normalized maximum value, x max =1; (3) Finally, the n normalized video samples X″′ are grouped into a set X as training samples, X = {X1″′, X2″′, X3″′, …, X″′ n }; The training sample X is an n×c×f w×h matrix; n is the number of normalized video samples X″′ in the training sample X, n∈N; Step 2: Data enhancement algorithm is based on the training round e i and the maximum number of training rounds e m Calculate the trigger probability p i , and get each frame image x in the training sample X i and the corresponding mask M i , the specific steps are as follows: (1) According to the training round e of the model i , and the maximum number of training rounds e m , calculate the trigger probability p of the data enhancement algorithm i , the calculation formula of trigger probability is: Where p i is the trigger probability of the i-th round of data enhancement algorithm, p i ∈(0,1];e m represents the maximum number of training rounds, e m ∈N; e i represents the current i-th round of training, e i ∈(1,e m ]; (2) Set the random probability q of the i-th training round i ,q i =random(0,1); when the random probability q of the i-th training round i Greater than the trigger probability p i When the algorithm uses three parameters l, j, r to generate the mask M1, M1 is composed of c identical w×h matrices, m ik Represents the elements in row i and column k of the c identical w×h matrices in mask M1; m ik ∈{0, 1}, i∈(0, 224), k∈(0, 224); m in mask M1 ik The calculation formula is as follows: M1 is the mask corresponding to the first frame image x1 in the training sample X, and M1 is a c w×h matrix; m wh ∈{0, 1}, m wh Represents a single element in the mask M1; l represents the number of grids in the mask M1, l∈[4, 14]; j represents the width of the grid in the mask M1, j∈[16, 56]; r represents the angle of random rotation of the mask M1, r∈(1°, 360°]; the calculation formula of the three parameters is: l=rand(l min ,L max ) j=random(j min ,j max ) r = random(1°, 360°) Where j max Represents the upper bound of the random value of the width j of the grid in M1, j max ∈[16, 56]; j min Represents the lower bound of the random value of the width j of the grid in M1, j min ≤j max ;j min ,j max The calculation formula is: Where h represents the height of the mask M1, l represents the number of grids in the mask M1; (3) The data enhancement algorithm performs a multi-step process on each frame image x in the training sample X. i and the corresponding mask M i To synthesize, M i is c w×h matrices; mask M i The calculation formula is: 0<i<f,w1+d w <w d <w2+d w , h1+d h <h d <h2+d h Where i is the i-th frame image in the training sample X, w and h represent the mask M i The length and width, p i is the trigger probability of the i-th round of data enhancement algorithm, f is the number of frames of training sample X, w d Represents mask M i The horizontal coordinate range of the deleted area, w d ∈(0,w);h d Represents mask M i The vertical coordinate range of the deleted area, h d ∈(0, h); (w1, h1) represents the upper left corner coordinate of the deleted area in mask M1, (w2, h2) represents the lower right coordinate of the deleted area in mask M1, {w1, h1, w2, h2}∈(0, 224); d w d h Represents mask M i The moving distance, d w d h The calculation formula is as follows: Where i is the i-th frame image in the training sample X, and f is the number of frames of the training sample X; Step 3: Use the mask M generated in step 2 i Synthesize with the training sample X to obtain the enhanced sample The specific steps are as follows: (1) When the random probability q of the i-th training round i Greater than the trigger probability p i When q i =random(0,1); the algorithm performs data enhancement on n normalized video samples X″′ to obtain enhanced video samples is a c×f w×h matrix; The calculation formula is as follows: Where f is the number of frames of the normalized video sample X″′, Z i ″′; i = {1, 2, ..., f}, represents the i-th frame image in the normalized video sample X″′, Z i ″′ is c w×h matrices; M i ; i = {1, 2, ..., f}, representing the i-th frame image Z in the normalized video sample X″′ i ″′ multiplied mask, M i is c w×h matrices; (2) For n enhanced video samples Combined enhanced samples The calculation formula of the enhancement algorithm is: Where q i is the random probability of the i-th training round, q i =random(0,1);p i is the trigger probability of the enhancement algorithm in the i-th round of training; X is the training sample, and the training sample X is an n×c×f w×h matrix; Represents enhanced video sample, enhanced video sample is a c×f w×h matrix, represents the enhanced sample, is n enhanced video samples A collection of It is an n×c×f w×h matrix.

Images