Deep neural network-based human detection system for surveillance

Abstract

A deep neural network-based human detection system for surveillance according to one embodiment of the present invention includes: a modified Gaussian YOLOv3 model combined with channel attention and feature entanglement modules, which uses data of overhead view images as a training dataset; an original Gaussian YOLOv3 model pre-trained with a plurality of general datasets; a transfer learning-based model training module that combines the original Gaussian YOLOv3 model and the modified Gaussian YOLOv3 model based on transfer learning; and an overhead view human detection model generated by the transfer learning-based model training model and retrained to detect bounding boxes containing a person in overhead view images using data of overhead view images as a training dataset.

Description

Technology Field

[0001] The present invention relates to a deep neural network-based human detection system for surveillance in smart cities and societies. Background Technology

[0002] Deep neural networks have been regarded as the foundation for many advanced artificial intelligence-based applications for sustainable smart cities and societies. These applications include smart healthcare systems, smart security, smart autonomous driving or self-driving vehicles, robotics, smart surveillance, and smart cities. Deep neural networks have significantly improved the performance of these applications beyond modern levels compared to existing machine learning-based technologies. Among these, person detection is receiving increasing attention from researchers due to its wide range of applications, which are not limited to anomaly detection, crowd monitoring, people counting, identification of people and pedestrians, characterization of human gait, gender classification, and fall detection in the elderly.

[0003] These applications can face various challenges due to the characteristics and properties of the human body, which involves diverse shapes and deformable poses. Additionally, complex and crowded environments, lighting variations, various types of occlusion, and diverse camera perspectives can affect the performance of human detection systems. To address these issues, various machine learning and deep neural network-based methods are being proposed and developed. These methods are generally based on existing manual features and various machine learning classifiers. On the other hand, advancements in deep learning are making human detection tasks more robust and effective in terms of efficiency and computational speed.

[0004] The primary advantage of deep neural networks is that they automatically select and extract the most important, higher-level features with a higher probability of classification. Furthermore, these advanced techniques generally provide additional discriminative power for multi-class object classification problems. Most developed human detection systems are typically based on frontal or asymmetrical camera perspectives. Factors that contribute most significantly to the changing visual appearance of a person in a frontal view image include body rotation, various postures, body movements, and torso and joints. In such situations, the impact of perspective changes is generally negligible.

[0005] A previous study adopted overhead view cameras for person detection applications. In overhead perspective, the appearance of the human body is more anomalous than in frontal perspective due to localized rotation, unique body movements, and position relative to the image center. The occlusion problem in overhead perspective is significantly less than in asymmetric and frontal perspectives, where cross-object occlusion can occur when the scene becomes cluttered. As a result of these characteristics, overhead view person detection is preferred in various real-world applications such as people counting, people tracking, behavior recognition, crowd analysis, behavior understanding, and human pose identification. In addition to handling occlusion issues, the overhead camera perspective can overcome computational costs, transmission load, energy consumption, and installation costs. Prior art literature

[0006] (Patent Document 0001) KR 10-2319468 B1 The problem to be solved

[0007] The problem that the present invention aims to solve is to provide a deep neural network-based human detection system for surveillance that can improve the accuracy of human detection in overhead view images. means of solving the problem

[0008] A deep neural network-based human detection system for surveillance according to one embodiment of the present invention is,

[0009] A modified Gaussian YOLOv3 model combining channel attention and feature entanglement modules, using data from overhead view images as a training dataset;

[0010] Original Gaussian YOLOv3 model pre-trained with multiple general datasets;

[0011] A transfer learning-based model training module that combines the original Gaussian YOLOv3 model and the modified Gaussian YOLOv3 model based on transfer learning; and

[0012] It may include an overhead view person detection model generated by the above-mentioned transfer learning-based model training model and retrained to detect bounding boxes containing people in overhead view images using data of overhead view images as a training dataset.

[0013] In a deep neural network-based human detection system for surveillance according to one embodiment of the present invention,

[0014] The channel attention and feature entanglement module included in the modified Gaussian YOLOv3 model above is,

[0015] Channel attention is performed on the input feature map (F) of the above modified Gaussian YOLOv3 model based on Equation 1, and

[0016] [Mathematical Formula 1]

[0017]

[0018] F is the input feature map, Mc(F) is the channel attention map, is an average-pooled feature, the overall average for each channel of the feature map, is a max-pooled feature, where the maximum value in each channel of the feature map, W0 and W1 are the weights of the 3-layer perceptron, and σ represents the sigmoid activation function.

[0019] In addition, in a deep neural network-based human detection system for surveillance according to one embodiment of the present invention, the channel attention and feature entanglement module comprises,

[0020] Additional connections between different channels can be performed within the same feature map layer.

[0021] In addition, in a deep neural network-based human detection system for surveillance according to an embodiment of the present invention, the modified Gaussian YOLOv3 model and the original Gaussian YOLOv3 model, respectively, are

[0022] The uncertainty of the predicted bounding box can be estimated by modeling the coordinates of the predicted bounding box as Gaussian parameters.

[0023] In addition, in a deep neural network-based human detection system for surveillance according to one embodiment of the present invention,

[0024] The center coordinate value of the above bounding box (t x , t y Let ) be denoted as such, and let the width and height of the bounding box be t, respectively. w , t h When saying,

[0025] The uncertainty of the bounding box is the average of the x-axis coordinates among the center coordinates of the bounding box ( ), average of the y-axis coordinates among the center coordinates of the bounding box above ( ), average of the widths of the bounding boxes above ( ), average of the heights of the bounding boxes above ( ), the variance of the x-axis coordinate among the center coordinates of the bounding box above ( ), the variance of the y-axis coordinate among the center coordinates of the above bounding box ( ), dispersion of the width of the bounding box ( ) and dispersion of the height of the bounding box ( It can be predicted by modeling ). Effects of the invention

[0026] According to a deep neural network-based human detection system for surveillance according to one embodiment of the present invention, the accuracy of human detection in overhead view images can be increased. Brief explanation of the drawing

[0027] Figure 1 is a diagram illustrating an overview of a smart deep neural network-based overhead view person detection system in smart cities and societies. FIG. 2 is a diagram illustrating a deep neural network-based human detection system for surveillance according to an embodiment of the present invention, wherein a pre-trained algorithm is first tested on an overhead view human dataset, and a channel attention and feature entanglement module is used to improve the extracted feature map, and the human detection system is also retrained on an overhead view human dataset using transfer learning, and the newly trained layer is interfaced with the original model. FIG. 3 is a diagram illustrating the schematic structure of an overhead view person detection module, wherein the structure is used for the extraction of bounding boxes and applied for person detection in overhead view images, and for feature extraction, a channel attention and feature entanglement module is used together with a convolution layer. Figure 4 is a schematic diagram of a channel attention module using max-pooling and average-pooling outputs with a shared network. Figure 5(a) shows the predicted feature map and bounding box attributes of YOLOv3, and (b) shows the attributes and elements of the bounding box predicted using the Gaussian parameters of the Gaussian YOLOv3 algorithm. Figure 6 illustrates the detection results of a pre-trained algorithm tested on an overhead view dataset, wherein in the test image, the person inside the blue rectangle is a person accurately classified into the person class with a confidence score value, the person inside the rectangle of another color is detected as other class objects, e.g., a dog, a kite, a motorcycle, etc., and the rectangular boxes marked with dashed lines represent bounding box uncertainty. Figure 7 illustrates the detection results of a pre-trained algorithm tested on a different overhead view dataset, wherein in the test image, the person inside the blue rectangle is a person correctly classified as a person class with a confidence score value, and in the rectangles of other colors, the person is detected as other class objects, e.g., a bird, a skateboard, etc. Figure 8 is a graph showing training and test losses. Figure 9 is a graph showing training and test accuracy. FIG. 10 illustrates the results of the Gaussian YOLOv3 detection algorithm after channel attention, feature entanglement, and additional training on an overhead view human dataset, and it can be confirmed that the performance of the detection model has increased, and the test results show that the present invention effectively classifies and detects people at various locations in an overhead view scene. FIG. 11 illustrates the results of an improved algorithm retrained on an overhead view human dataset, showing that the detection model performance increases after transfer learning, and the test results show that the model according to the present invention effectively classifies people in images and detects people from various top-view positions. Figure 12 is a diagram showing the precision, recall, F1-score, and accuracy of a pre-trained and improved trained algorithm. Specific details for implementing the invention

[0028] The objects, specific advantages, and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments in conjunction with the accompanying drawings.

[0029] Prior to this, terms and words used in this specification and claims shall not be interpreted in their ordinary and dictionary meanings, but shall be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention.

[0030] It should be noted that in assigning reference numbers to the components of each drawing in this specification, identical components are given the same number as much as possible, even if they are shown in different drawings.

[0031] In addition, terms such as "first," "second," "one side," and "other side" are used to distinguish one component from another, and the components are not limited by these terms.

[0032] Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the essence of the invention are omitted.

[0033] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0034] In this invention, the Gaussian YOLOv3 algorithm (Choi, Chun, Kim, & Lee, 2019, "Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving") is used for human detection by utilizing deep neural network-based object detection technology. This algorithm determines bounding box uncertainty by modeling coordinates with Gaussian parameters to improve accuracy and reduce false positives. Gaussian YOLOv3 is combined with channel attention and feature entanglement modules to improve specific feature maps. The channel attention module is combined with the feature map to autonomously learn the weights of each channel, improve key features, and enhance the network's ability to distinguish between people and backgrounds. At the same time, other channels of the feature map are entangled to obtain more representative features. Finally, features obtained from the attention and feature entanglement (Intertwin) modules are fused to form an enhanced feature map. Additionally, transfer learning is adopted to further increase the detection accuracy of the human detection algorithm. Experimental results showed that training improved the potential of the Gaussian YOLOv3 algorithm for human detection with an overall detection accuracy of 94%.

[0035] It can be observed that both different camera perspectives, based on asymmetric and frontal camera perspective, exhibit significant variations in the visual characteristics of the human body (size, shape, scale, pose, and body orientation). A deep neural network-based system for smart surveillance applications is provided to detect people in overhead view environments.

[0036] The Gaussian YOLOv3 algorithm is generally based on YOLOv3 (Redmon & Farhadi, 2018, "YOLOv3: An Incremental Improvement") and determines three predicted bounding boxes for each grid cell. Here, the predicted boxes consist of bounding box coordinates, an object score, and a class score. The key difference is that instead of five different attribute elements, the predicted bounding boxes now consist of eight different attribute elements. Each bounding box coordinate is determined by the mean and variance values. Gaussian YOLOv3 is also combined with channel attention and feature entanglement modules. The channel attention module learns the weights of each channel by connecting with the feature map. This improves the main feature map and enhances the network's ability to distinguish between people and backgrounds. At the same time, different channels of the feature map are entangled to obtain more representative features. Finally, the features received from the attention and feature entanglement modules are fused to form an improved feature map.

[0037] In this invention, the algorithm is first tested with a pre-trained architecture. As previously discussed, the visual characteristics of a person viewed from an overhead view differ significantly from those seen from the front. Consequently, the algorithm fails to detect a person or provides incorrect results. Therefore, transfer learning is utilized to retrain the architecture using two different overhead-view human datasets, and the newly trained layers are combined with the existing architecture. This additional training improves the accuracy results of the human detection system. In general, the main contributions of this invention are as follows.

[0038] - Introduce a deep neural network-based person detection system for smart surveillance applications in smart cities and societies. The system uses the Gaussian YOLOv3 algorithm for overhead view person detection.

[0039] - To improve key features and enhance the network's ability to distinguish between people and backgrounds, a channel attention module is combined with the feature map. Other channels of the feature map are also entangled to obtain more representative features. Finally, the features obtained from the attention and feature intertwine modules are fused to form an enhanced feature map.

[0040] - Training an algorithm on two different overhead view datasets using transfer learning to improve human detection results.

[0041] - Comparison of detection results between the pre-trained algorithm and the improved training algorithm.

[0042] Overhead-view-based person detection is recognized by researchers as one of the challenging tasks. Various overhead-view person detection methods have been developed based on diverse background subtraction and feature-based approaches.

[0043] Ahmad, Ahmed, Khan, Qayum, and Aljuaid (2020) adopted a convolutional neural network-based method to track people in overhead view scenes. Deep learning-based segmentation algorithms are also explored for overhead view person detection (Ahmed, Ahmad, Khan, & Asif, 2020). Ahmed, Din, Jeon, Piccialli, and Fortino (2020) introduced a surveillance system for multi-object tracking by detection. In this study, the authors used various tracking algorithms along with deep learning-based object detection techniques. Ahmed, Ahmad, Ahmad, and Jeon (2020) introduced an overhead view person tracking system utilizing a deep SORT algorithm with the YOLOv3 architecture.

[0044] Based on the literature review above, it can be seen that most studies presented on overhead view person detection are primarily based on feature-based methods.

[0045] In the present invention, a newly developed Gaussian YOLOv3 algorithm is adopted for person detection using two different overhead view person record datasets. The algorithm is also combined with channel attention and feature intertwine modules to improve detection performance and is retrained on overhead view datasets.

[0046] The present invention proposes a deep neural network-based system for human detection in smart surveillance applications, namely smart cities and societies. Figure 1 illustrates the general structure of a smart surveillance system. The system is primarily composed of smart camera devices or sensors installed inside a building. These smart camera devices record video and images that are transmitted to a data center via the cloud using wireless technology or an internet connection. The data center performs pre-processing and transmits the data to a deep neural network-based surveillance system for further analysis and training. After pre-processing and model training, the analyzed data results are transmitted back to a monitoring and surveillance center for various real-time smart surveillance applications.

[0047] As previously discussed, this invention focuses primarily on person detection, which is considered one of the important tasks in various surveillance applications. Although there are various deep neural network-based object detection algorithms, most fail to anticipate incorrect localization, provide deterministic results for bounding boxes, and do not consider the uncertainty of bounding boxes. Therefore, this invention utilizes Gaussian YOLOv3 (Choi et al., 2019), which estimates the uncertainty of bounding boxes by modeling the coordinates of bounding boxes using Gaussian parameters.

[0048] The algorithm was previously trained on the KITTI and Berkeley Deep Drive (BDD) datasets and developed for autonomous driving applications. The present invention extends the model for overhead view person detection. FIG. 2 is a diagram illustrating a deep neural network-based person detection system for surveillance according to an embodiment of the present invention.

[0049] Referring to FIG. 2, a deep neural network-based human detection system for surveillance according to one embodiment of the present invention may include: a modified Gaussian YOLOv3 model (201) that uses data of overhead view images acquired from an overhead view camera (200) as a training data set and includes a channel attention and feature entanglement module (204); an original Gaussian YOLOv3 model (206) that is pre-trained with a plurality of general data sets; a transfer learning-based model training module (208) that combines the original Gaussian YOLOv3 model (206) and the modified Gaussian YOLOv3 model (201) based on transfer learning; and an overhead view human detection model (212) that is generated by the transfer learning-based model training model (208) and trained to detect bounding boxes containing a person in overhead view images using data of overhead view images as a training data set.

[0050] Reference number 202 represents the Gaussian YOLOv3 model, and reference number 210 represents the MS-COCO weight file.

[0051] It can be observed that the dataset of images acquired by the overhead view camera (200) is divided into training samples and test samples. First, a pre-trained Gaussian YOLOv3 model (206) was trained using the MS-COCO (Common Object in Context) dataset weight file (210). Additionally, the feature map can be improved by combining the channel attention and feature intertwine module (204) with the newly trained Gaussian YOLOv3 model (202).

[0052] First, the channel attention module is used for channel-based attention in convolutional neural networks, which can improve principal feature maps and enhance the network's ability to recognize foreground objects (people) and the background.

[0053] Second, different channels of the feature map are simultaneously intertwined to obtain more descriptive features. Finally, features obtained by the attention and feature entanglement modules can be fused to form an improved feature map.

[0054] As with previously trained architectures, people / human bodies are mostly considered from a frontal or lateral perspective. Therefore, additional training is performed to improve the accuracy results of the detection algorithm for human detection from an overhead view perspective. In the present invention, an overhead view human detection model (212) was retrained based on an overhead view human dataset. The overhead view human detection model (212) learned new features of the dataset and performed human detection using a test dataset.

[0055] In the transfer learning-based model training module (208), the newly trained model (202) through transfer learning is combined with the pre-trained model (206) to create an overhead view person detection model (212), and the overhead view person detection model (212) is tested on the same test data set as in FIG. 2. This algorithm improves the results of person detection for overhead view images. The results of the trained and pre-trained algorithms are evaluated and further processed for monitoring applications. Below, the method used for overhead view person detection will be described in detail.

[0056] Gaussian YOLOv3 is generally based on YOLOv3. It is an extended version of the YOLO (You Only Look Once) object detection algorithm. In this version, the accuracy, particularly for small objects, has been significantly improved compared to previous versions of the algorithm. Major changes in YOLOv3 include backbone network tuning, the use of multi-scale features for object detection, and the replacement of Softmax with a Logistic layer for object classification.

[0057] The network architecture of a deep neural network-based human detection system for surveillance according to one embodiment of the present invention is shown in FIG. 3. Darknet-53 is utilized as a backbone network applying the idea of ​​a ResNet architecture in which all residual modules consist of shortcut connections and two convolutional layers. YOLOv3 resolves the vanishing gradient problem of the deep network by utilizing residual skip connections. YOLOv3 maintains fine features for small-sized objects by applying an upsampling layer and a cascading system. All convolutional layers are supported by ReLu activation and batch normalization layers.

[0058] The architecture illustrated in Fig. 3 is divided into two main parts: a feature extractor and a detector. The feature extractor extracts feature maps from the input image at three different scales. Next, predictions are made on these maps by applying a 1×1 detection kernel. Every cell in the predicted feature map predicts three bounding boxes. Each bounding box contains attribute information including center coordinates, height, width, object score, and class confidence. The input image is divided into S×S grid cells to determine the feature map cells responsible for prediction, identical to the final feature map. For each object detected in the image, the grid cell consists of the center of the ground truth bounding box of the target object / person responsible for prediction. Here, the ground truth refers to the original or actual value of the data to be learned.

[0059] All cells in the feature map are utilized for object prediction through one of the detected bounding box information. Each grid cell of the feature map predicts three bounding boxes, as shown in Fig. 5a. Instead of predicting the width and height of the bounding boxes, YOLOv3 predicts an offset for a predefined box, also known as an anchor. An anchor box is a predefined box / shape determined to map the ground truth bounding box, as in the training dataset, and most objects have an average width-to-height ratio. Therefore, this offset can be used in the anchor box to obtain predictions for the object.

[0060] Consequently, the bounding box responsible for object detection (in the present invention, a person) has the highest intersection rate with the anchor containing the correct bounding box. In addition, in one embodiment of the present invention, the channel attention module shown in FIG. 4 is combined with the architecture using the inter-channel relationship of features. Since the feature map of each channel is recognized as a feature detector, channel attention can extract meaningful features for a given input image.

[0061] In Fig. 4 is the input feature map, and represents channel attention. In the feature map, W, H, and C represent width, height, and number of channels, and the circle with a + sign (oplus) represents element-wise multiplication. The calculation for the channel attention module is completed using the following mathematical formula.

[0062]

[0063] In the mathematical formula above indicates that an F-channel attention map is generated. Here, is the overall average for each channel of the feature map as an average-pooled feature, and is a max-pooled feature, representing the maximum value in each channel of the feature map. Each of the above feature values ​​contains feedback that allows for the effective learning of image background information. Max pooling can collect information on prominent feature values.

[0064] A Multilayer Perceptron (MLP) is used, here is the weight of the 3-layer perceptron, and σ is the sigmoid activation function. W0 and W1 are the weights of the multilayer perceptron (MLP), and to improve the nonlinearity of the network, a ReLu (Rectified Linear Unit) activation function may follow W0.

[0065] The method primarily used in convolutional neural networks for feature extraction is based on a layered structure, where each channel of each layer is extracted in the same manner. The aforementioned network may use multi-scale convolution kernels for each layer to extract features, or it may use a fusion-based method.

[0066] In the present invention, the feature entanglement module combines channel attention by establishing connections between different channels in the same feature map layer. The characteristic of this method is that it solves the problem of loss of transmission information and improves the network's feature extraction ability for small objects.

[0067] The predicted feature map of YOLOv3 is shown in Fig. 5a. It can be seen that three prediction boxes are acquired for each grid cell. Each detected predicted bounding box is given by the following coordinate values ​​(e.g., t x , t y , t w , t h It consists of objectity (object score) and class score. The object score determines whether an object is already within the bounding box. The category or class of each object is determined by the class score, with a value between 0 and 1. This model uses the product of these two values ​​for object detection. Unlike object and class information, the bounding box coordinate values ​​are defined in the output as deterministic coordinate values. Therefore, the confidence level of the detected bounding box cannot be determined.

[0068] In addition, the object score of the bounding box does not indicate the confidence of the detected bounding box and causes the bounding box uncertainty problem, which means that it is not possible to recognize how uncertain the result of the detected bounding box is. Therefore, to solve this problem, Gaussian YOLOv3 was adopted.

[0069] YOLOv3 is as shown in FIG. 5a ( To extract bounding box center information represented by ), bounding box regression is used, and It represents bounding box size information. For each detected boundary, there is one exact answer, which is the measured correct answer. Therefore, complex modeling is not required to predict localization uncertainty.

[0070] The uncertainty of the bounding box is It can be modeled using a single Gaussian model. For a given input image x, the Gaussian model output y is provided as follows (Choi et al., 2019).

[0071]

[0072] In the above mathematical formula, is the average and is used as the variance function. The uncertainty of the bounding box is predicted by modeling the μ (mean) and Σ (variance) of each bounding box coordinate detected in the prediction feature, as shown in Fig. 5b. The center coordinate value of the bounding box (t x , t y Let ) be denoted as such, and let the width and height of the bounding box be t, respectively. w , t h When so, the uncertainty of the bounding box is the average of the x-axis coordinates among the center coordinates of the bounding box ( ), average of the y-axis coordinates among the center coordinates of the bounding box above ( ), average of the widths of the bounding boxes above ( ), average of the heights of the bounding boxes above ( ), the variance of the x-axis coordinate among the center coordinates of the bounding box above ( ), the variance of the y-axis coordinate among the center coordinates of the above bounding box ( ), dispersion of the width of the bounding box ( ) and dispersion of the height of the bounding box ( It can be predicted by modeling ).

[0073] The output of the bounding box is , , , , , , and It is indicated as. In the detection layer, the average value of all coordinates is the predicted coordinate of the bounding box, and each variance value represents the uncertainty of each coordinate. The center coordinates of the bounding box are inside the grid cell. and It is indicated by ,. Each mean and variance value is processed between 0 and 1, and the sigmoid function given in Equation 3 is applied.

[0074]

[0075] In YOLOv3, the height and width information of the bounding box is prior to the detected bounding box and exponential function and It is processed using . This is in Fig. 5 and YOLOv3's and It represents, and since both can have positive and negative values, it means that it is not processed by the sigmoid function. Gaussian modeling is applied to predict the uncertainty of bounding box coordinates obtained from the YOLOv3 detection layer without structural changes or an increase in overall computational complexity.

[0076] During training, YOLOv3 uses the binary cross-entropy loss function for class and object prediction. For bounding boxes, YOLOv3 applies a combination of squared error losses. The coordinate information of the bounding boxes is output as Gaussian parameters through Gaussian modeling. Therefore, the loss function for bounding boxes is redefined as the negative log-likelihood (NLL) loss. Here, the functions for class and object losses remain unchanged. The loss function used for bounding box prediction is mathematically given as follows.

[0077]

[0078]

[0079]

[0080]

[0081] In mathematical formulas 4 to 7, the NLL loss is It is indicated as follows. The number of grid cells for each height and width is denoted by H and W, and K represents the number of anchors. Also, represents the uncertainty of the tx coordinates in the output of the detection layer, and the actual ground truth coordinates of tx are It is represented as (for details, refer to Choi et al. (2019)).

[0082] Gaussian YOLOv3 also obtains the bounding box uncertainty for each detection in the image. During object detection, YOLOv3 estimates only the class score and object score and does not estimate the bounding box score because the information regarding the bounding box coordinate score is unknown. Nevertheless, the output of Gaussian YOLOv3 includes localization uncertainty, also known as the bounding box score. Therefore, localization uncertainty is estimated along with the class and object scores during the detection process. Finally, highly confident predictions for the class, object, and bounding box scores are determined. Consequently, Gaussian YOLOv3 improves detection accuracy by reducing false positives and enhancing true positives. Based on the analysis of localization uncertainty, detection criteria are provided as follows.

[0083]

[0084] In the mathematical formula above, Cr reflects the detection criteria for Gaussian YOLOv3, and the object score and class score for the i-th class are denoted as σ(Object) and σ(Class), respectively. Localization uncertainty ( ) represents the average uncertainty of the predicted bounding box coordinates.

[0086] Experimental results

[0087] A detailed description of the dataset and various experimental results are described here. The experiments are run using the Python programming language with an Intel Core i7-875, 64GB RAM, 2.2GHz CPU, 8GB NVIDIA GEFORCE GTX display card, and a 64-bit Windows 10 operating system.

[0088] The overall test results are described below. First, the performance of the pre-trained algorithm is described. Second, the results of the detection algorithm after channel attention and feature entanglement and transfer learning are described. For better comparison, the algorithm is tested using images from the same dataset.

[0089] Algorithm performance evaluation is performed using various quantitative measurements. It is also compared with other deep neural network-based algorithms.

[0091] Data set description

[0092] Two different overhead-view human datasets are utilized for training and testing the algorithm. Both datasets are captured in the same environment but use different camera lenses. In the first dataset, images of people are captured using a standard lens camera, whereas in the second dataset, images are captured using a wide-angle lens.

[0093] Images are captured from an overhead camera perspective in uncontrolled situations with varying lighting conditions. People move freely in the field without restrictions. Consequently, there are differences in the visual characteristics of people in terms of size, scale, orientation, pose, and camera lens. Images were captured at a resolution of 640×480 pixels using a camera positioned at a height of 5 meters. A total of 2,000 sample images were used, with 1,000 used for training and 1,000 for testing.

[0095] Test results of the pre-trained algorithm

[0096] Test results for overhead view human detection using a pre-trained algorithm (trained with the MS-COCO dataset) are shown in Figures 6 and 7. The results are evaluated using different images from the dataset. People in the scene are walking freely. It can be seen in the images that the visual features of a person from the two overhead perspectives are not the same as in the case of the frontal view. The visual features of the human body change significantly from above the head at various positions, as shown in Figures 6 and 7. Since the algorithm is pre-trained on various class objects, in most cases, the human body is detected as a different object.

[0097] For example, in Fig. 6a, for the deep learning algorithm, four people are correctly classified as human because their visual appearance resembles a human, one is classified as a dog, and three are not detected by the algorithm, as indicated by the red cross mark. In Fig. 6b, only one person is detected, and the others are not detected, as indicated by the red cross mark. In Fig. 6c, although the number of people in the scene increases, the pre-trained algorithm can only detect two people. One is classified as a bird, and the others are not detected by the pre-trained algorithm. Similarly, in Fig. 6d, only the person's feet at the bottom of the image are classified as human, while other parts of the body are classified as a motorcycle. This is likely due to his dark-colored clothing. In the sample images, one person is classified as a cow, and three are not detected. As a result, it can be seen that in Fig. 6a, a person wearing white clothes is detected at the top of the sample image, whereas the same person is not detected in the remaining three sample images.

[0098] Additionally, the pre-trained algorithm was tested on another overhead view human dataset where images of people were captured using a wide-angle lens camera. The results are shown in Figure 7. It can be seen that the appearance and pose of the people changed in both overhead views. The pre-trained algorithm provides good results even without training. As can be seen in the visualization results, in most cases, people are correctly classified into the corresponding class.

[0099] In Fig. 7a, four people entering the scene were correctly detected by the Gaussian YOLOv3 algorithm. In one instance, a person's feet were classified as a skateboard, but two people marked with a red cross were not detected. In Fig. 7b, five people in the image were correctly classified by the algorithm, but two people moving in opposite directions were not detected. In Fig. 7c, a person directly below the camera was correctly classified by the deep learning algorithm, but a woman in almost the same position at the same time was not detected by the model. Three people leaving the scene were also not detected by the algorithm, as indicated by the red cross. In Fig. 7d, it was observed that the algorithm provided only a single detection result and did not detect others. One person was classified as a bird, and another was detected as a person.

[0101] Detection results of the improved algorithm

[0102] The algorithm is trained on the overhead view human dataset to improve detection results. After training, the algorithm's detection accuracy improves. The entire experiment is based on the Darknet framework. Gaussian_yolov3_BDD.weights is used as the initialization parameter. The MS-COCO weight file was also used. The batch size is set to 512, the number of epochs during training is 100, and Stochastic Gradient Descent (SGD) is used as the optimizer. The network layers were trained for 45,000 iterations. Training accuracy and loss are achieved at the end of the 19th epoch. The algorithm's accuracy and loss are shown in Figures 8 and 9, respectively. In Figure 8, loss convergence is easily observed, and the error value is close to 0.2 or 0.3 after the 20th epoch. Similarly, in Figure 9, the training accuracy shows that the algorithm's training accuracy is nearly 95% after the 30th epoch.

[0103] After training, the algorithm is tested again on the same test images previously described. The experimental results show that detection results are significantly improved after additional training and transfer learning, as shown in Figures 10 and 11. The detection algorithm detects people located at different positions in the overhead scene. In the sample image of Figure 10, people at different locations with different visual features are accurately detected. As can be seen in Figure 10a, all people who were not detected by the pre-trained algorithm in Figure 6a are now correctly detected.

[0104] In Fig. 6b, the three people in the center of the image, who were previously undetected and marked with a red cross, were effectively detected by the deep learning algorithm in Fig. 10b. In Fig. 10c, the trained algorithm was tested on a larger number of people compared to the number of people in Figs. 10a and 10b, and the algorithm efficiently detects multiple people. Comparing the results in Fig. 6d and Fig. 10d, it can be seen that the trained model accurately detects people.

[0105] In addition, the Gaussian YOLOv3-trained model was tested on different overhead view human dataset images, as shown in Fig. 11. Referring to Figs. 7 and 11, it can be seen that the results improved significantly after transfer learning. For example, comparing Fig. 7a with Fig. 11a, it can be seen that people in various overhead positions were correctly detected, whereas previously, people in the center were indicated by a red cross as not being detected. Similarly, in Fig. 11b, two people wearing white clothes walking in opposite directions were detected, whereas the pre-trained model failed to detect them. In Fig. 11c, multiple people in the center of the image are recognized by the deep learning algorithm. In contrast, the same result is obtained in Fig. 11d, where the trained algorithm accurately detects other people located directly below the overhead camera. People in the same location in Fig. 11d are not detected by the pre-trained algorithm. Furthermore, people who were not detected or were classified as other objects are also accurately detected.

[0107] Evaluation results

[0108] The performance of the detection algorithm is evaluated using various quantitative parameters such as tp, fp, fn, and tn. tp (True Positive) defines bounding boxes that are correctly detected as people or the person class. fp (False Positive) determines a misclassified bounding box as the person class, and tn (True Negative) represents a bounding box correctly classified as background. fn (False Negative) represents a bounding box that is inappropriately recognized and classified as another object. Precision, Recall, F1-score, and Accuracy are calculated for the algorithm. In Figure 12, the results of the pre-trained and improved algorithms can be examined using the overhead view dataset. These values ​​are mathematically given as follows.

[0109]

[0110]

[0111]

[0112]

[0113] The standard error method is used to calculate precision, recall, F1 score, and accuracy. In Fig. 12, the results of a pre-trained and trained algorithm can be examined using an overhead view dataset. From the results, it can be seen that the accuracy of the detection model increases after applying learning and transfer learning. Fig. 12 shows the average values, and it can be seen that the overall detection accuracy of the deep neural network-based human detection system for surveillance according to one embodiment of the present invention has improved to 94%.

[0114] In addition, the mAP value, representing average precision, was calculated to investigate the performance of the algorithm. The mAP value is used to estimate the accuracy of the detection model using precision and recall values. When precision is used in relation to recall, the mAP average precision for class N is given as follows.

[0115]

[0116] Table 1 shows the results of comparing the detection model with other deep learning algorithms. It can be seen that this model exhibits better performance compared to other detection algorithms. The mAP value of the deep neural network-based human detection system for surveillance according to one embodiment of the present invention is 0.90, which is higher than that of other object detection models.

[0117]

[0118] The present invention presents the application of deep neural networks for human detection in smart cities and societies. A Gaussian YOLOv3 algorithm was applied for human detection in various overhead views, including channel attention and feature entanglement modules. Through this, the feature map learns the weights of each channel and combines these channels to obtain more representative features, thereby improving detection. The algorithm's weights were initialized using a pre-trained algorithm. The algorithm is further retrained on an overhead view human dataset to enhance detection results. Additionally, transfer learning was adopted to further reduce the algorithm's false positive rate and increase the accuracy of human detection in overhead views. A key advantage of the algorithm is that accuracy is improved and false positives are reduced because the bounding box uncertainty is determined by modeling the bounding box coordinates using Gaussian parameters. Experimental results demonstrate that combining channel attention, feature entanglement modules, and additional training improves the overall accuracy of a deep neural network-based human detection system for surveillance according to one embodiment of the present invention, with an mAP of 0.90 for overhead images.

[0119] Although the present invention has been described in detail through specific embodiments, this is for the purpose of specifically explaining the invention and is not limited thereto. It will be apparent that modifications or improvements can be made by those skilled in the art within the technical scope of the present invention.

[0120] All simple variations or modifications of the present invention fall within the scope of the present invention, and the specific scope of protection of the present invention will be clarified by the appended claims. Explanation of the symbols

[0121] 200: Overhead view camera 201: Modified Gaussian YOLOv3 Model 202 : Gaussian YOLOv3 Model 204: Channel Attention and Feature Intertwine Module 206 : Original Gaussian YOLOv3 model 208: Transfer Learning-Based Model Training Module 210 : MS-COCO weight file 212: Overhead View Person Detection Model

Claims

Claim 1 A modified Gaussian YOLOv3 model that uses data of overhead view images as a training dataset and combines channel attention and feature entanglement modules; an original Gaussian YOLOv3 model pre-trained with multiple general datasets; and a transfer learning-based model training module that combines the original Gaussian YOLOv3 model and the modified Gaussian YOLOv3 model based on transfer learning. and includes an overhead view person detection model generated by the transfer learning-based model training model and retrained to detect bounding boxes containing people in overhead view images using data of overhead view images as a training dataset; the channel attention and feature entanglement module includes a channel attention module and a feature entanglement module; the channel attention module generates a channel attention map for an input feature map; the feature entanglement module combines channel attention by establishing connections between different channels in the same feature map layer; the overhead view person detection model extracts the feature map of the input image at three different scales, predicts three bounding boxes for all cells of the feature map, and detects an object through one of the three bounding boxes; and each of the modified Gaussian YOLOv3 model and the original Gaussian YOLOv3 model models the coordinates of the predicted bounding box as Gaussian parameters to predict the bounding Estimate the uncertainty of the box, and the center coordinate value of the bounding box (t x , t y Let ) be denoted as such, and let the width and height of the bounding box be t, respectively. w , t h When so, the uncertainty of the bounding box is the average of the x-axis coordinates among the center coordinates of the bounding box ( ), average of the y-axis coordinates among the center coordinates of the bounding box above ( ), average of the widths of the bounding boxes above ( ), average of the heights of the bounding boxes above ( ), the variance of the x-axis coordinate among the center coordinates of the bounding box above ( ), the variance of the y-axis coordinate among the center coordinates of the above bounding box ( ), dispersion of the width of the bounding box ( ) and dispersion of the height of the bounding box ( A deep neural network-based human detection system for surveillance, predicted by modeling ). Claim 2 In claim 1, the channel attention and feature entanglement module performs channel attention on the input feature map (F) of the modified Gaussian YOLOv3 model based on Equation 1, and [Equation 1] F is the input feature map, M c (F) is a channel attention map, is the overall average for each channel of the feature map as an average-pooled feature, A deep neural network-based human detection system for surveillance, wherein is a max-pooled feature with the maximum value in each channel of the feature map, W0 and W1 are weights of a 3-layer perceptron, and σ represents a sigmoid activation function. Claim 3 delete Claim 4 delete Claim 5 delete

Images