Real-time foreground detection method and device based on full convolutional network and attention mechanism
By designing a lightweight U-Net-like encoder-decoder network, combining batch normalization, bilinear interpolation and CBAM attention mechanism, the problems of large parameter scale and poor generalization performance of mainstream prospect detection network are solved, and the problem of large-scale parameters and poor generalization performance of mainstream prospect detection network is realized. Efficient real-time prospect detection, significantly improving processing efficiency and detection performance.
Patent Information
- Application Number
- CN202210508603.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The mainstream prospect detection network has large parameters and poor generalization performance, making it difficult to be competent for real-time video prospect detection tasks.
Adopting a lightweight U-Net-like encoder-decoder network architecture, combining batch normalization and bilinear interpolation, adding a CBAM attention mechanism, using a fusion cross entropy and Dice loss function, optimize the model to reduce over-context Combining and improving generalization.
It has achieved excellent prospect detection capabilities and good generalization in public data sets, especially in terms of detection speed, the requirements of real-time processing are met, the processing efficiency is high, the F value reaches more than 98%, and the FPS reaches 73.
Smart Images

Figure CN114820711B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mobile target detection, and in particular to a real-time foreground detection method and device based on a fully convolutional network and an attention mechanism. Background Art
[0002] With the popularity of video surveillance in road traffic, home security, financial security, industrial production and other fields, how to use computer vision algorithms to extract key information from video data has become particularly important. Video foreground detection is one of the basic research directions in the field of computer vision. Its task is to separate the foreground and background areas from the video captured by the surveillance camera. In most intelligent video surveillance systems, video foreground detection is regarded as an indispensable preprocessing step, which can provide a focus for advanced processing tasks such as tracking, classification, recognition, and behavior analysis.
[0003] In the past five years, a large number of foreground detection methods based on deep learning have been proposed. Although these methods have strong foreground detection capabilities and robustness, they face the problem of model overfitting and poor generalization due to insufficient training data. More seriously, the parameter scale of the current mainstream foreground detection network is large, and such models are not competent for real-time video foreground detection tasks. Summary of the invention
[0004] The main technical problem solved by the present invention is that the parameters of the current mainstream foreground detection network are large and the generalization performance is poor. In order to solve the above problems, the technical solution adopted by the present invention is: to provide a real-time foreground detection method and device based on a full convolutional network and an attention mechanism, and the network adopts a lightweight U-Net-like encoder-decoder network architecture. In the convolution layer of the network, batch normalization operation is used to speed up the training speed of the model and reduce overfitting. In the upsampling process, bilinear interpolation is used instead of transposed convolution, which avoids the disadvantage of the "chessboard effect" and further reduces the number of network parameters. In order to enhance the model's ability to select channel features and spatial features, a CBAM attention mechanism module is added to the network.
[0005] According to one aspect of the present invention, the present invention provides a real-time foreground detection method based on a fully convolutional network and an attention mechanism, comprising the following steps:
[0006] S1, capturing a video, and using the video data to construct a foreground detection dataset, and selecting a training set and a validation set from the foreground detection dataset;
[0007] S2. Add the CBAM attention module to the fully convolutional neural network to build a fully convolutional network model integrating the attention mechanism.
[0008] S3, setting certain hyperparameters, using the training set to train the full convolutional network model, and using the verification set to verify the fitting effect of the full convolutional network model. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained;
[0009] S4. Perform real-time foreground detection on the currently captured video using the real-time foreground detection model to obtain a foreground detection result of the current video.
[0010] Preferably, step S1 comprises:
[0011] S11. Capture a video using a surveillance camera;
[0012] S12, select M frames of video images from the captured video sequence according to a uniform time distribution, use any pixel-level annotation software (such as Image Labeler, LabelMe, CVAT, etc.) to annotate the foreground area of the video image, and after the annotation is completed, randomly select N frames of annotated images and the corresponding original images as a validation set, and the remaining MN frames of annotated images and the corresponding original images as a training set;
[0013] S13, performing standardization processing on the M frames of video images to obtain a standardized result, wherein the standardization formula is as follows:
[0014]
[0015] In the formula, y i is the standardized result, x i is the current frame image, is the mean of the training set images, and s is the standard deviation of the training set images.
[0016] Preferably, step S2 comprises:
[0017] S21, constructing the input layer of the fully convolutional network model; inputting a 3-channel RGB image of any size, and performing an edge adaptive zero padding operation for images whose length or width does not meet 16 times;
[0018] S22, constructing an encoder network of the full convolutional network model; the encoder network includes 4 modules, wherein module 1, module 2, and module 3 each include 2 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer; module 4 includes 4 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer, and the maximum pooling layer is located after the second convolution layer;
[0019] S23. Construct a decoder network of the fully convolutional network model; the decoder network includes 5 modules, among which module 5, module 6, module 7, and module 8 each include an upsampling layer and two convolutional layers with a convolution kernel size of 3×3. The upsampling is first completed using the bilinear interpolation method, and then the features are spliced. The spliced features enter two convolutional layers with a convolution kernel size of 3×3 to improve the upsampling features; module 9 includes a CBAM attention mechanism module and a convolutional layer with a convolution kernel size of 1×1.
[0020] Preferably, step S3 comprises:
[0021] S31. Use the fusion cross entropy loss function CE and Dice loss function to calculate the loss. The calculation formula is as follows:
[0022]
[0023]
[0024] loss=CE+Dice
[0025] In the formula, p i is the prediction result of the current pixel i, y i is the true result corresponding to the current pixel i, n represents the number of image pixels, p is the predicted pixel set for an image, and y is the pixel set of the labeled image;
[0026] S32. Use a small batch stochastic gradient descent optimizer as the training optimizer, set the initial learning rate to 0.01, the momentum coefficient to 0.9, the batch-size to 4, the learning rate decays following the Poly strategy with a coefficient set to 0.9, and the number of training rounds to 200; in addition, a monitor is added to the training program to save the parameter weights of the model with the best effect on the validation set, and the model with the best effect is used as the real-time foreground detection model.
[0027] Preferably, S4 includes:
[0028] S41, performing standardization processing on the currently captured video according to the same parameters as step S13 to obtain a standardized video image;
[0029] S42: Input the standardized video image into the real-time foreground detection model, and output the foreground detection result of the current video.
[0030] According to another aspect of the present invention, the present invention provides a real-time foreground detection device based on a full convolutional network and an attention mechanism, comprising:
[0031] A data set construction module, used for capturing a video, and constructing a foreground detection data set using the video data, and selecting a training set and a validation set from the foreground detection data set;
[0032] The network building module is used to add the CBAM attention module to the fully convolutional neural network and build a fully convolutional network model that integrates the attention mechanism;
[0033] A network training module is used to set certain hyperparameters, train the full convolutional network model using the training set, and verify the fitting effect of the full convolutional network model using the verification set. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained;
[0034] The foreground detection module is used to use the real-time foreground detection model to perform real-time detection of the foreground area of the currently captured video to obtain the foreground detection result of the current video.
[0035] The technical solution provided by the present invention has the following beneficial effects:
[0036] In order to solve the problems of large parameter scale and poor generalization performance of the current mainstream foreground detection network, the present invention designs a lightweight U-Net-like encoder-decoder network architecture. In the convolutional layer of the network, batch normalization is used to speed up the training speed of the model and reduce overfitting. In the upsampling process, bilinear interpolation is used to replace transposed convolution, which avoids the disadvantage of "chessboard effect" and further reduces the number of network parameters. In order to enhance the model's ability to select channel features and spatial features, a CBAM attention mechanism module is added to the network. In addition, in order to avoid the adverse effects of imbalanced positive and negative samples, the network uses a loss function that integrates cross entropy and DiceLoss. Experiments on some subsets of the public data set CDnet2014 show that the network not only exhibits excellent foreground detection capabilities and good generalization, but also meets the requirements of real-time processing in terms of detection speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0038] Figure 1 is a flow chart of a real-time foreground detection method based on a fully convolutional network and an attention mechanism provided by an embodiment of the present invention;
[0039] Figure 2 is a structural diagram of a fully convolutional network integrating an attention mechanism provided by an embodiment of the present invention;
[0040] Figure 3It is a structural diagram of a real-time foreground detection device based on a fully convolutional network and an attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.
[0042] Please refer to Figure 1 , an embodiment of the present invention provides a real-time foreground detection method based on a fully convolutional network and an attention mechanism, comprising the following steps:
[0043] S1, capturing a video, and using the video data to construct a foreground detection dataset, and selecting a training set and a validation set from the foreground detection dataset;
[0044] Step S1 specifically includes:
[0045] S11. Use a surveillance camera to capture a video. In principle, the video should not be less than 200 frames after being decomposed into images. The longer the video, the better the application effect of the method proposed in the present invention.
[0046] S12, select 200 frames of video images from the captured video sequence according to a uniform time distribution, and use any pixel-level annotation software (such as Image Labeler, LabelMe, CVAT, etc., in this embodiment, LabelMe is preferred) to segment the foreground area of the video image (where the foreground area is represented by 255 of the grayscale image, and the background area is represented by 0). After the annotation is completed, randomly select 20 frames of annotated images and their corresponding original images as the verification set, and the remaining 180 frames of annotated images and their corresponding original images are used as the training set;
[0047] S13, calculating the mean and standard deviation of the 180 frames of video images in the training set as standardization parameters, performing standardization processing on the 200 frames of video images, and obtaining the standardized results, wherein the standardization formula is as follows:
[0048]
[0049] In the formula, y i is the standardized result, x i is the current frame image, is the mean of the training set images, s is the standard deviation of the training set images, where y i is the standardized result, x i is the current frame image, is the mean of the training set images, s is the standard deviation of the training set images, and s are normalization parameters.
[0050] S2. Add the CBAM attention module to the fully convolutional neural network to build a fully convolutional network model integrating the attention mechanism.
[0051] Step S2 specifically includes:
[0052] S21. Construct the input layer of the fully convolutional network model; a 3-channel RGB image of any size can be input. For images whose length or width does not meet 16 times, perform edge adaptive zero padding (Padding).
[0053] S22. Construct the encoder network of the full convolutional network model; as shown in Table 1, the encoder network can be divided into 4 modules. Module 1, Module 2, and Module 3 contain 2 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer. Module 4 contains 4 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer, and the pooling layer is located after the second convolution layer. In the table, "Conv" represents the convolution layer, "Batch Norm" represents the batch normalization operation, and "ReLU" represents the ReLU activation function. "F" is the convolution layer filter size, "S" is the convolution step size, and "P" is the number of edge zero padding layers. "W" and "H" are the width and height of the input image, respectively. In the encoder network, the output results of module 1, module 2, module 3, and "Conv8" will be saved for use in the decoder network, and the feature map output by "Conv10" will be used as the input of the decoder network.
[0054] Table 1 Encoder network of the fully convolutional network model
[0055]
[0056] S23. Construct a decoder network of the full convolutional network model; as shown in Table 2, the decoder network can be divided into 5 modules. Modules 5, 6, 7, and 8 first use bilinear interpolation to complete upsampling, and then perform feature splicing. The spliced features will enter two convolutional layers with a convolution kernel size of 3×3 to improve the upsampling features. In module 9, the features output by module 8 will be input into a CBAM attention mechanism module and a convolutional layer with a convolution kernel size of 1×1. In the table, "BI" is the bilinear interpolation method, "Concate" is the feature splicing operation, "ChannelAttention" is the channel attention mechanism, "SpatialAttention" is the spatial attention mechanism, and "Sigmoid" is the Sigmoid activation function.
[0057] Table 2 Decoder network of the fully convolutional network model
[0058]
[0059] S3, setting certain hyperparameters, using the training set to train the full convolutional network model, and using the verification set to verify the fitting effect of the full convolutional network model. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained;
[0060] Step S3 specifically includes:
[0061] S31. In order to overcome the problem of uneven positive and negative samples in the foreground detection field, the fusion cross entropy loss function CE and Dice loss function are used to calculate the loss. The calculation formula is as follows:
[0062]
[0063]
[0064] loss=CE+Dice
[0065] In the formula, p i is the prediction result of the current pixel i, y i is the true result corresponding to the current pixel i, n represents the number of image pixels, p is the predicted pixel set for an image, and y is the pixel set of the labeled image.
[0066] S32. Mini-batch stochastic gradient descent optimizer (Mini-batch SGD) is used as the training optimizer. The initial learning rate is set to 0.01, the momentum coefficient is set to 0.9, the batch-size is set to 4, the learning rate is decayed following the Poly strategy with a coefficient set to 0.9, and the number of training epochs is set to 200. In addition, a monitor is added to the training program to save the parameter weights of the model with the best effect on the validation set, and the best model is used as the real-time foreground detection model.
[0067] S4. Perform real-time foreground detection on the currently captured video using the real-time foreground detection model to obtain a foreground detection result of the current video.
[0068] S41, performing a preprocessing operation on the image collected by the same surveillance camera as in S11 according to the same standardization method as in S13 (the standardization parameters are also the same as in S13).
[0069] S42, inputting the processing result of S41 into the trained network to detect the foreground detection result.
[0070] In order to verify the beneficial effects of the present invention, this embodiment selects a subset of the ChangeDetection2014 data set (i.e., videos copyMachine, corridor, office, sofa, diningRoom, PETS2006 and library collected in seven different environments), and conducts comparative experiments on the method of the present invention (named FgUNet) and the following mainstream supervised methods. The evaluation indicators are the percentage of wrong classifications PWC (Percentage of Wrong Classifications), F-measure, and FPS. Several mainstream supervised methods are: BSPVGAN (Background subtraction algorithm based on Bayesian generative adversarial networks and parallelvision theory), MU-Net1 (MotionU-Net motion U-type network), Cascade CNN (Cascade convolutional neural network), DeepBS (Deep ConvolutionalNeural Network for Video Sequence Background Subtraction background difference deep convolutional network), and the experimental results are shown in Table 3-5:
[0071] Table 3 F-value performance of each method on the dataset
[0072]
[0073] Table 4 PWC performance of each method on the dataset
[0074]
[0075] Table 5 FPS comparison of various methods on the dataset
[0076]
[0077] refer to Figure 3 In some embodiments, a real-time foreground detection device based on a full convolutional network and an attention mechanism is also provided, comprising the following modules:
[0078] A data set construction module 1 is used to capture a video and construct a foreground detection data set using the video data, and select a training set and a verification set from the foreground detection data set;
[0079] Network construction module 2 is used to add the CBAM attention module to the fully convolutional neural network to build a fully convolutional network model integrating the attention mechanism;
[0080] The network training module 3 is used to set certain hyperparameters, train the full convolutional network model using the training set, and verify the fitting effect of the full convolutional network model using the verification set. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained;
[0081] The foreground detection module 4 is used to perform real-time detection of the foreground area of the currently captured video using the real-time foreground detection model to obtain a foreground detection result of the current video.
[0082] This embodiment provides a real-time foreground detection method and device based on a fully convolutional network and an attention mechanism. The method has strong detection performance, good generalization ability, and can achieve real-time detection. The foreground detection method establishes a network structure in the field of foreground detection by designing a lightweight encoder-decoder network model and a network optimization method. The method is an end-to-end method. Compared with traditional methods, the present invention does not require complex preprocessing links and achieves a high accuracy rate on the test set. The video sequence image is input into the network to obtain the foreground detection result, and the F value reaches more than 98%. And the processing efficiency of this method is extremely high. When using a GTX1060 (6G) graphics card as the acceleration hardware, the processing of a video image with a size of 320*240 can reach 73FPS.
[0083] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0084] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.
[0085] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A real-time foreground detection method based on a fully convolutional network and an attention mechanism, characterized in that: The following steps are involved: S1, capturing a video, and using the video data to construct a foreground detection dataset, and selecting a training set and a validation set from the foreground detection dataset; S2. Add the CBAM attention module to the fully convolutional neural network to build a fully convolutional network model integrating the attention mechanism. S3, setting certain hyperparameters, using the training set to train the full convolutional network model, and using the verification set to verify the fitting effect of the full convolutional network model. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained; S4, using the real-time foreground detection model to perform real-time detection of the foreground area of the currently captured video to obtain a foreground detection result of the current video; Step S2 includes: S21, constructing the input layer of the fully convolutional network model; inputting a 3-channel RGB image of any size, and performing an edge adaptive zero padding operation for images whose length or width does not meet 16 times; S22, constructing an encoder network of the full convolutional network model; the encoder network includes 4 modules, wherein module 1, module 2, and module 3 each include 2 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer; module 4 includes 4 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer, and the maximum pooling layer is located after the second convolution layer; S23, constructing a decoder network of the full convolutional network model; the decoder network includes 5 modules, wherein module 5, module 6, module 7, and module 8 each include an upsampling layer and two convolutional layers with a convolution kernel size of 3×3, first use bilinear interpolation to complete upsampling, then perform feature splicing, and the spliced features enter two convolutional layers with a convolution kernel size of 3×3 to improve the upsampling features; module 9 includes a CBAM attention mechanism module and a convolutional layer with a convolution kernel size of 1×1; Step S3 includes: S31. Use the fusion cross entropy loss function CE and Dice loss function to calculate the loss. The calculation formula is as follows: loss=CE+Dice In the formula, p i is the prediction result of the current pixel i, y i is the true result corresponding to the current pixel i, n represents the number of image pixels, p is the predicted pixel set for an image, and y is the pixel set of the labeled image; S32. Use a small batch stochastic gradient descent optimizer as the training optimizer, set the initial learning rate to 0.01, the momentum coefficient to 0.9, the batch-size to 4, the learning rate decays following the Poly strategy with a coefficient set to 0.9, and the number of training rounds to 200; in addition, a monitor is added to the training program to save the parameter weights of the model with the best effect on the validation set, and the model with the best effect is used as the real-time foreground detection model.
2. The real-time foreground detection method based on a fully convolutional network and an attention mechanism as claimed in claim 1, characterized in that: Step S1 includes: S11. Capture a video using a surveillance camera; S12, select M frames of video images from the captured video sequence according to a uniform time distribution, use any pixel-level annotation software to annotate the foreground area of the video image, and after the annotation is completed, randomly select N frames of annotated images and the corresponding original images as a validation set, and the remaining MN frames of annotated images and the corresponding original images as a training set; S13, performing standardization processing on the M frames of video images to obtain a standardized result, wherein the standardization formula is as follows: In the formula, y i is the standardized result, x i is the current frame image, is the mean of the training set images, s is the standard deviation of the training set images, and s are normalization parameters.
3. The real-time foreground detection method based on a fully convolutional network and an attention mechanism as claimed in claim 2, characterized in that S4 include: S41, performing standardization processing on the currently captured video according to the same standardization parameters as step S13 to obtain a standardized video image; S42: Input the standardized video image into the real-time foreground detection model, and output the foreground detection result of the current video.
4. A real-time foreground detection device based on a fully convolutional network and an attention mechanism, characterized in that: Includes the following modules: A data set construction module, used for capturing a video, and constructing a foreground detection data set using the video data, and selecting a training set and a validation set from the foreground detection data set; The network building module is used to add the CBAM attention module to the fully convolutional neural network and build a fully convolutional network model that integrates the attention mechanism; A network training module is used to set certain hyperparameters, train the full convolutional network model using the training set, and verify the fitting effect of the full convolutional network model using the verification set. When the preset effect is achieved, the training is completed and a real-time foreground detection model is obtained; A foreground detection module is used to perform real-time detection of the foreground area of the currently captured video using the real-time foreground detection model to obtain a foreground detection result of the current video; The workflow of the network building module is as follows: S21, constructing the input layer of the fully convolutional network model; inputting a 3-channel RGB image of any size, and performing an edge adaptive zero padding operation for images whose length or width does not meet 16 times; S22, constructing an encoder network of the full convolutional network model; the encoder network includes 4 modules, wherein module 1, module 2, and module 3 each include 2 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer; module 4 includes 4 convolutional layers with a convolution kernel size of 3×3 and 1 maximum pooling layer, and the maximum pooling layer is located after the second convolution layer; S23, constructing a decoder network of the full convolutional network model; the decoder network includes 5 modules, wherein module 5, module 6, module 7, and module 8 each include an upsampling layer and two convolutional layers with a convolution kernel size of 3×3, first use bilinear interpolation to complete upsampling, then perform feature splicing, and the spliced features enter two convolutional layers with a convolution kernel size of 3×3 to improve the upsampling features; module 9 includes a CBAM attention mechanism module and a convolutional layer with a convolution kernel size of 1×1; The workflow of the network training module is as follows: S31. Use the fusion cross entropy loss function CE and Dice loss function to calculate the loss. The calculation formula is as follows: loss=CE+Dice In the formula, p i is the prediction result of the current pixel i, y i is the true result corresponding to the current pixel i, n represents the number of image pixels, p is the predicted pixel set for an image, and y is the pixel set of the labeled image; S32. Use a small batch stochastic gradient descent optimizer as the training optimizer, set the initial learning rate to 0.01, the momentum coefficient to 0.9, the batch-size to 4, the learning rate decays following the Poly strategy with a coefficient set to 0.9, and the number of training rounds to 200; in addition, a monitor is added to the training program to save the parameter weights of the model with the best effect on the validation set, and the model with the best effect is used as the real-time foreground detection model.
