A YOLOv3-based video detection method for computer rooms

Abstract

A YOLOv3-based target detection method for computer room videos includes the following steps: S1, determining the target to be detected and constructing a training dataset; S2, constructing a YOLOv3-based target detection model; S3, acquiring the current image frames of the computer room surveillance video in real time, wherein the surveillance video is real-time video recorded by surveillance cameras pre-deployed in the computer room; S4, inputting the acquired current image frames into the constructed YOLOv3 computer room video detection model for detection and outputting the detection results. This invention is simple, improves the accuracy of target bounding box localization and the detection effect on highly overlapping images, further enhances its detection speed and real-time performance, and demonstrates excellent target detection performance in computer room video surveillance; it greatly ensures the security of unattended computer rooms, improves computer room maintenance efficiency, and makes computer room security management more automated and efficient, resulting in significant economic and social benefits.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method for detecting computer room videos based on YOLOv3. Background Technology

[0002] With the rapid development of network technology, the number and scale of data centers are also increasing exponentially, making their maintenance and security crucial. Traditional data center maintenance relies on camera monitoring supplemented by manual inspections. However, due to limitations in current technology, this monitoring method only provides video playback and real-time scene viewing; it cannot promptly locate and alert on abnormal behavior within the data center. It heavily depends on manual supervision, resulting in poor security and real-time performance, serving only as evidence after an incident. Therefore, researching more secure and reliable intelligent video detection is of great significance.

[0003] The rapid development of computer vision in recent years has led to the emergence of many deep learning-based object detection algorithms. Representative examples include the one-stage YOLO series and the two-stage R-CNN series. One-stage algorithms, due to their high detection speed, are widely used in industrial applications with high real-time requirements. However, in practical applications, the diversity of detection targets and the complexity of detection backgrounds often make it difficult to simultaneously achieve both detection speed and accuracy, causing unnecessary problems. Therefore, how to balance detection speed and accuracy in diverse targets and complex backgrounds is a major challenge in data center video surveillance. Summary of the Invention

[0004] In view of the above situation and to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a data center video inspection method based on YOLOv3, which can effectively solve the problem that the existing technology cannot simultaneously take into account the detection speed and accuracy in the face of diverse detection targets and complex detection backgrounds.

[0005] To achieve the above objectives, the technical solution provided by this invention is a data center video detection method based on YOLOv3, comprising the following steps:

[0006] S1. Determine the target to be detected and construct the training dataset. The method is as follows:

[0007] S11. Identify the target to be detected within the computer room environment;

[0008] S12. Collect images containing the objects to be detected and other images: Use images from the Wider Face dataset and historical representative image frames from the computer room video as images for the dataset, and use image annotation tools to annotate them to construct a new training dataset;

[0009] S13. Determine the model training tool: Use PyTorch, a third-party deep learning tool library for Python, as the YOLO model training tool;

[0010] S2. Construct an object detection model based on YOLOv3, using the following method:

[0011] S21. The lightweight network MobileNetv3 is used as the feature extraction network, and the AE module is added to the prediction layer to build the YOLOv3 network architecture, thereby improving the speed and real-time performance of object detection.

[0012] S22. Construct a weighted nonmaximum suppression algorithm to enhance the accuracy of target box localization and improve the detection efficiency for highly overlapping images;

[0013] S23. Introduce a new loss function to improve the imbalance in the ratio of positive to negative samples in the original YOLOv3 network architecture;

[0014] S24. Employ a combination of data augmentation strategies to address data sample imbalance;

[0015] S25. Use the constructed dataset and Python experimental platform to train the network. The constructed dataset is the new training data aggregate constructed in step S12. Select 60% of the dataset as the training set, 30% as the test set, and 10% as the validation set.

[0016] S26. Output the improved YOLOv3 object detection model;

[0017] S3. Real-time acquisition of the current image frame of the monitoring video in the computer room, wherein the monitoring video is real-time video recorded by monitoring cameras pre-deployed in the computer room;

[0018] S4. Input the acquired current data center image frame into the YOLOv3 data center video detection model constructed in step S26 for detection, and output the detection result. The detection result includes: the monitoring camera information of the current image frame, whether it contains the detection target, the classification result of the detection target, and the coordinate position information of the detection target. If the image frame does not contain the detection target, return to step S3, acquire the next frame of data center video image, and then continue to execute step S4 for detection; if the image frame contains the detection target, output the detection result, and then return to step S3 to acquire the next frame of image for detection, and return the detection result of this image frame to the data center management backend.

[0019] The method of this invention is simple, improves the accuracy of target bounding box localization and the detection effect of highly overlapping images, and further enhances its detection speed and real-time performance. It performs well in target detection in computer room video surveillance. It can also classify and alarm when abnormal behavior is detected and notify maintenance personnel for handling, which greatly ensures the security of the computer room when it is unattended, improves the maintenance efficiency of the computer room, and makes the computer room security management work more automated and efficient, with huge economic and social benefits. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the YOLOv3-based video detection method for computer rooms in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the novel YOLOv3 network architecture constructed in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the bneck module structure of MobileNetv3 in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the network structure of MobileNetv3 in an embodiment of the present invention;

[0024] Figure 5 This is a detailed diagram of the network structure connection module of MobileNetv3 in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the AE module in an embodiment of the present invention;

[0026] Figure 7 This is a flowchart illustrating the specific process of target detection based on YOLOv3 in an embodiment of the present invention. Detailed Implementation

[0027] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0028] Example

[0029] like Figure 1 As shown, a data center video detection method based on YOLOv3 includes steps S1, S2, S3, and S4, specifically:

[0030] S1. Determine the target to be detected and construct the training dataset. The method is as follows:

[0031] S11. Identify the targets to be detected in the computer room environment: including but not limited to people, doors and windows, alarm lights, flames, and intruders, such as unauthorized entry by non-staff members, intrusion by people outside of maintenance hours, abnormal opening and closing of doors and windows, abnormal alarm lights, flame intrusion, and rat intrusion.

[0032] S12. Collect images containing the object to be detected and other images not containing the object to be detected: Select a portion of images from the WiderFace dataset and historical representative image frames from computer room videos as images for the dataset, and label them using the image annotation tool labelImg to construct a new training dataset. This dataset contains a total of 12,680 images, of which 10,000 contain images of the object to be detected and 2,680 contain background images;

[0033] The Wider Face dataset contains 62 scenes and a total of 32,203 images, of which 393,703 are labeled with variations in face size, lighting, pose, expression, and occlusion.

[0034] S13. Determine the model training tool: Select PyTorch, a third-party deep learning tool library for Python, as the YOLO model training tool.

[0035] S2. Construct an object detection model based on YOLOv3, using the following method:

[0036] S21. Improve the original YOLOv3 network architecture: Adopt the excellent lightweight network MobileNetv3 as the feature extraction network, and introduce the AE (Assisted excitation of Activations) module in the prediction layer to build a brand-new YOLOv3 network architecture, improve the problem of inaccurate target box localization, and improve the target detection speed and real-time performance. Figure 2 This is a schematic diagram of the novel YOLOv3 network architecture constructed in this embodiment of the invention.

[0037] Furthermore, MobileNetv3 is a very good lightweight network that includes a new bneck module, introduces a new non-linear activation function h-swish, and uses two advanced search techniques, NetAdapt and NAS (platform-aware NAS), to build the search model.

[0038] The bneck module integrates depthwise separable convolution, inverse residual structure with linear bottleneck, and SE (Squeeze-and-Excitation) module, and can include SE-Net as part of it, placed after the depthwise convolution kernel. Figure 3This is a schematic diagram of the bneck module structure of MobileNetv3 in an embodiment of the present invention.

[0039] The nonlinear activation function h-swish can improve network accuracy while reducing the computational cost of model training. Its calculation formula is as follows:

[0040]

[0041] Where swish[x] = x*sigmoid(βx), ReLu = min(max(0,x),6).

[0042] NetAdapt is a fine-tuning of each network layer from a local perspective after each module is determined; NAS is a search on the modules, optimizing the structure of each module to determine the layout of the design modules for the entire neural network architecture when the amount of computation and parameters are constrained.

[0043] It should be noted that the MobileNetv3 network is designed in two versions, MobileNetv3-Small and MobileNetv3-Large, according to the level of computer configuration resources. This invention uses the MobileNetv3-Large version (hereinafter, MobileNetv3 refers to MobileNetv3-Large).

[0044] Furthermore, combined Figure 4 and Figure 5 A detailed explanation of MobileNetv3 is provided. Figure 4 This is a schematic diagram of the network structure of MobileNetv3 in an embodiment of the present invention; Figure 5 This is a detailed diagram of the network structure connection module of MobileNetv3 in an embodiment of the present invention.

[0045] like Figure 4 and 5 As shown, RE represents the ReLU activation function, and HS represents the h-swish activation function. The SE module indicates whether the bend contains an SE module; if SE is not marked in the network structure, it means the area within the dashed box is not present. NBN indicates the absence of a normalized BN layer, and BN indicates the presence of a normalized BN layer. Avg pooling represents a global average pooling layer. The (Conv2d 1×1, HS, NBN) module represents a structure consisting only of 1×1 convolutional layers and the h-swish activation function, and the (Conv2d 1×1, NBN) module represents a structure consisting only of 1×1 convolutional layers.

[0046] Figure 5 Only for Figure 4 The (benck3×3, RE) module, (benck3×3, SE, HS) module, and (Conv2d, HS) module are broken down in detail, and all other modules can be derived from these three modules. Specifically, Figure 4 The (benck3×3, HS) module replaces the ReLU activation function in the (benck3×3, RE) module with h-swish; the (benck5×5, SE, HS) module replaces the 3×3 depthwise separable convolutional kernels in the (benck3×3, SE, HS) module with 5×5 convolutional kernels; the (benck5×5, SE, RE) module first replaces the 3×3 depthwise separable convolutional kernels in the (benck3×3, SE, HS) module with 5×5 convolutional kernels, transforming it into (benck5×5, SE, HS), and then replaces the h-swish activation function with ReLU, thus transforming it into (benck5×5, SE, RE). The (Conv2d 1×1, HS) module represents a module consisting of a 1×1 convolutional layer, a normalized layer (BN), and an h-swish activation function.

[0047] Furthermore, the AE module is a method that assists detection by providing additional information. This embodiment of the invention introduces this module to reduce localization errors in the model algorithm. The AE module mainly enhances the network's learning ability by introducing a ground truth mask. This method is only used during training and gradually reduces the activation factor, thus not affecting the detection speed. Figure 6 This is a schematic diagram of the AE module in an embodiment of the present invention. Details are as follows:

[0048] 1) Use the feature map of the prediction layer as the input to the AE layer;

[0049] 2) Perform global average pooling on the input features across the channel to obtain a single-channel average tensor map;

[0050] 3) Perform a dot product operation between the average tensor image of the single channel and the GT mask image to obtain the excitation tensor image;

[0051] 4) Multiply the activation tensor graph by the activation factor;

[0052] 5) Add the value obtained from operation 4) to each channel of the input feature map of the AE layer.

[0053] In other words, the implementation process of the AE layer can be viewed as a network layer that controls network activity, and the calculation of the ground truth (GT) mask map is also simpler. Specifically, the original GT is first mapped to the same size as the feature map, and then all pixels within the target bounding box region are represented by 1s and the background by 0s, resulting in the following 0-1 mask matrix:

[0054]

[0055] The formula for the AE layer can be expressed as:

[0056] f l+1 =f l +αe

[0057] Among them, f l Let be the input feature map, e be the activation tensor, and α be the activation factor, which controls the intensity decay of training and gradually decreases with the increase of the number of iterations. Its calculation formula is defined as:

[0058]

[0059] Where t is the current epoch value, Max_Iteration is the maximum number of iterations, and .t represents t / Max_Iteration.

[0060] S22. Construct a weighted non-maximum suppression (NMS) algorithm to enhance the accuracy of target box localization and improve the detection efficiency for highly overlapping images;

[0061] The original NMS algorithm essentially searches for local maxima and suppresses non-maximum elements. While this approach is simple to implement, when two detection boxes are close together, the detection box with the lower score is easily mistakenly deleted due to excessive overlap, leading to the failure of true object detection and significantly impacting detection accuracy and recall.

[0062] To address the problems of the original NMS algorithm, this invention constructs a weighted NMS algorithm. Specifically, it improves upon the original NMS algorithm's forced zeroing approach. For detection boxes with the same cluster center, when their repetition exceeds a set threshold, their overlap is reduced according to a Gaussian weighted rule, rather than being directly set to zero. Assuming M represents the detection boxes with the highest confidence scores, b... i The bounding box to be processed has an IoU of M and b. i Interaction ratio, S i Let represent the score of the i-th detection box. Therefore, the improved Gaussian-weighted NMS algorithm can be expressed as:

[0063]

[0064] S23. Introduce a new loss function to address the imbalance in the positive and negative sample ratio in the original YOLOv3. Specifically, improve the coordinate prediction error by introducing GIoU bounding box regression loss and improve the confidence prediction error by introducing Focal loss, thereby improving the overall loss function of YOLOv3. The improved overall loss function is expressed as:

[0065] LOSS'=LOSS coord '+LOSS IoU '+LOSS class

[0066] Among them, LOSS coord To improve coordinate prediction error, LOSS IoU To improve confidence prediction error, LOSS class Original category prediction error.

[0067] Furthermore, coordinate prediction error is improved by introducing GIoU bounding box regression loss.

[0068] This invention addresses the optimization problem of IoU for non-overlapping parts by introducing GIoU bounding box regression loss, thereby optimizing the coordinate loss component of the overall loss function. The formula for calculating GIoU is:

[0069] GIoU=IoU-|C(A∩B)| / |C|

[0070] Here, A and B are any two bounding boxes, and A,B∈S∈R n Let C be the minimum closure region of A and B, and S be the space containing A and B. The GIoU bounding box regression loss can be expressed as:

[0071] LOSS GIoU =1-GIoU

[0072] Therefore, the improved coordinate prediction error can be expressed as:

[0073]

[0074] Where S represents grid size and B represents box. This indicates that the j-th anchor box in the i-th grid contains an object and is responsible for prediction, and Otherwise, set to zero.

[0075] Furthermore, the confidence prediction error is improved by introducing the Focal loss function.

[0076] Since the original confidence prediction error is calculated for each sample using the standard cross-entropy loss function and assumes all samples have equal weight, this does not solve the problem of imbalanced positive and negative samples. Therefore, this embodiment of the invention introduces the Focal loss function to improve this, and its calculation formula is as follows:

[0077]

[0078] Where p represents the model output class probability, and p∈[0,1]; y represents the class label, y∈{0,1}; p γ Let be the modulation coefficient and α be the balance parameter. Referring to the above equation, the improved coordinate prediction error can be expressed as:

[0079]

[0080] in, Let be the confidence score of the j-th anchor box in the i-th grid. Represents the confidence level of the true bounding box. λ represents the confidence level of the predicted bounding box. noobj There is no weight coefficient for the object.

[0081] Furthermore, without improving the category prediction error, the original formula for calculating the category prediction error remains:

[0082]

[0083] in, This represents the class probability of the bounding box generated by the j-th anchor box of the i-th grid. This represents the class probability of the true bounding box. This represents the class probability of the predicted bounding box.

[0084] S24. Employ a data augmentation strategy that combines multiple types to augment the image data;

[0085] The combined data augmentation strategy refers to a method that combines color operations, geometric operations, and the addition of image noise. Specifically, color operations involve distorting color channels by enhancing contrast, changing image chroma and brightness, and equalizing histograms; this operation does not affect the position of the bounding box. Geometric operations involve geometrically distorting the image by cutting, translating, and rotating; this operation changes the position and size of the bounding box. Adding image noise involves filtering the image by adding Gaussian noise, salt-and-pepper noise, or other image noise.

[0086] S25. Train the improved YOLOv3 object detection model using the constructed dataset and Python experimental platform.

[0087] The constructed dataset refers to the new training data aggregation constructed in step S12. 60% of the dataset is selected as the training set, 30% as the test set, and 10% as the validation set.

[0088] The Python experimental platform refers to an experimental platform that uses Python 3.8 as the scripting language, Caffe as the training framework, and PyTorch, a third-party deep learning tool library in Python, as the learning library.

[0089] The experimental parameters for network training also include: the operating system is Ubuntu 18.04, the CPU is an Intel(R) Core(TM) i7-3770 CPU@3.40GHz, and the GPU is an NVIDIA GTX 1080.

[0090] S26. Output the improved YOLOv3 object detection model.

[0091] S3. Real-time acquisition of current image frames from the monitoring video in the computer room;

[0092] The surveillance video is real-time video recorded by surveillance cameras pre-installed in the computer room; the surveillance cameras are Hikvision DS network cameras, which support real-time monitoring, image capture, video recording, infrared night vision, intelligent motion detection, H.265 compression and other functions.

[0093] S4. Input the acquired current computer room image frame into the constructed YOLOv3 target detection model for detection, and output the detection results.

[0094] The detection results include the surveillance camera information of the current image frame, whether it contains a detection target, the classification result of the detection target, and the coordinate position information of the detection target.

[0095] Furthermore, Figure 7 This is a flowchart illustrating the specific process of target detection based on YOLOv3 according to an embodiment of the present invention. Figure 7 If the image frame does not contain the target, the process returns to step S3 to acquire the next frame of the data center video image, and then continues to step S4 for detection. If the image frame contains the target, the detection result is output, and the process returns to step S3 to acquire the next frame for detection, and the detection result of this image frame is returned to the data center management backend. Furthermore, different levels of alarms can be triggered based on abnormal behavior in the detection results, and maintenance personnel can be notified to handle the issue.

[0096] This invention provides a YOLOv3-based method for detecting video in computer rooms. Compared with existing technologies, the advantages of this invention are:

[0097] (1) This invention improves the original YOLOv3 network architecture, adopts MobileNetv3 as the feature extraction network, and adds an AE module to the prediction layer, which reduces the complexity of parameters and the amount of computation, and improves the detection speed and real-time performance of the original YOLOv3.

[0098] (2) This invention optimizes the original YOLOv3 algorithm, improves the non-maximum suppression algorithm, improves the accuracy of target box localization and the detection effect of highly overlapping images, further improves its detection speed and real-time performance, and performs well in target detection in computer room video surveillance.

[0099] (3) This invention improves the overall loss function of YOLOv3 and draws on a combination of data augmentation strategies to alleviate the problem of imbalance in the ratio of positive and negative samples in the original YOLOv3 algorithm;

[0100] (4) Based on the original YOLOv3 technology, this invention designs a brand-new YOLOv3 target detection model and a YOLOv3-based data center video detection method, which realizes real-time detection and identification of specific targets in data center videos, and can classify alarms and notify maintenance personnel when abnormal behavior is detected. This greatly ensures the security of the data center when it is unattended, reduces the data center operation and maintenance cost, improves the data center maintenance efficiency, and can meet the detection needs of actual scenarios in the data center. This makes the data center security management work more automated and efficient, and has profound practical application value and huge economic and social benefits.

Claims

1. A method for detecting computer room video based on YOLOv3, characterized in that, Includes the following steps: S1. Determine the target to be detected and construct the training dataset. The method is as follows: S11. Identify the target to be detected within the computer room environment; S12. Collect images containing the objects to be detected and other images: Use images from the Wider Face dataset and historical representative image frames from the computer room video as images for the dataset, and use image annotation tools to annotate them to construct a new training dataset; S13. Determine the model training tool: Use PyTorch, a third-party deep learning tool library for Python, as the YOLO model training tool; S2. Construct an object detection model based on YOLOv3, using the following method: S21. The lightweight network MobileNetv3 is used as the feature extraction network, and the AE module is added to the prediction layer to build the YOLOv3 network architecture, thereby improving the speed and real-time performance of object detection. The AE module described is a method that assists detection by providing additional information. This module reduces localization errors in the model algorithm. The AE module primarily enhances the network's learning ability by introducing a ground truth mask. This method is only used during training and gradually reduces the activation factor, thus not affecting detection speed. Specifically: 1) Use the feature map of the prediction layer as the input to the AE layer; 2) Perform global average pooling on the input features across the channel to obtain a single-channel average tensor map; 3) Perform a dot product operation between the average tensor image of the single channel and the GT mask image to obtain the excitation tensor image; 4) Multiply the activation tensor graph by the activation factor; 5) Add the value obtained from operation 4) to each channel of the input feature map of the AE layer; S22. Construct a weighted nonmaximum suppression algorithm to enhance the accuracy of target box localization and improve the detection efficiency for highly overlapping images; S23. Introduce a new loss function to improve the imbalance of positive and negative sample ratio in the original YOLOv3 network architecture; S24. Employ a combination of data augmentation strategies to address data sample imbalance; S25. Use the constructed dataset and Python experimental platform to train the network. The constructed dataset is the new training data aggregate constructed in step S12. Select 60% of the dataset as the training set, 30% as the test set, and 10% as the validation set. S26. Output the improved YOLOv3 object detection model; S3. Real-time acquisition of the current image frame of the monitoring video in the computer room, wherein the monitoring video is real-time video recorded by monitoring cameras pre-deployed in the computer room; S4. Input the acquired current data center image frame into the YOLOv3 data center video detection model constructed in step S26 for detection, and output the detection result. The detection result includes: the monitoring camera information of the current image frame, whether it contains the detection target, the classification result of the detection target, and the coordinate position information of the detection target. If the image frame does not contain the detection target, return to step S3, acquire the next frame of data center video image, and then continue to execute step S4 for detection; if the image frame contains the detection target, output the detection result, and then return to step S3 to acquire the next frame of image for detection, and return the detection result of this image frame to the data center management backend.

2. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, In step S11, the target to be detected is a person, door or window, alarm light, flame, or intruder.

3. The YOLOv3-based data center video detection method according to claim 1, characterized in that, The WiderFace dataset contains 62 scenes and a total of 32,203 images, of which 393,703 are labeled with variations in face scale, lighting, pose, expression, and occlusion.

4. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, MobileNetv3 is a lightweight network that includes a bneck module and introduces a new non-linear activation function. h-swish The search model is constructed using NetAdapt and NAS; the bneck module integrates depthwise separable convolution, SE module and inverse residual structure with linear bottleneck.

5. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, The AE module described above uses additional information to assist in detection, and can gradually reduce the activation factor without affecting the detection speed.

6. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, The weighted nonmaximum suppression algorithm described above reduces the overlap of detection boxes in the same cluster center according to a Gaussian weighted rule when the repetition exceeds a set threshold.

7. The YOLOv3-based data center video detection method according to claim 1, characterized in that, The new loss function in step S23 is introduced GIOU The bounding box regression loss function and the Focal loss function are improved, along with the overall loss function of YOLOv3.

8. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, The data augmentation strategy that combines multiple types is a data augmentation strategy that combines color operations, geometric operations, and the addition of image noise.

9. The YOLOv3-based video detection method for computer rooms according to claim 1, characterized in that, In step S25, the Python experimental platform is an experimental platform using Python 3.8 as the scripting language, Caffe as the training framework, and PyTorch, a third-party deep learning tool library for Python, as the learning library. The network training environment parameters are: operating system Ubuntu 18.04, CPU Intel(R) Core(TM) i7-3770 CPU@3.40GHz, and GPU NVIDIA GTX 1080.

10. The data center video detection method based on YOLOv3 according to claim 1, characterized in that, The image annotation tool in step S12 is labelImg.

Images