Emergency intelligent monitoring and early warning system based on unmanned aerial vehicle

Abstract

The invention discloses an emergency intelligent monitoring and early warning system based on an unmanned aerial vehicle. The emergency intelligent monitoring and early warning system comprises a data acquisition module, an image processing and analysis module, an abnormal behavior detection module, an early warning module and an unmanned aerial vehicle embedded platform. According to the invention, an optimized deep learning algorithm is combined with a domestic i. MX 8M Plus embedded platform, so that an intelligent monitoring system carried by the unmanned aerial vehicle is realized; the system monitors dense crowds in real time by utilizing the maneuverability and the wide view field of the unmanned aerial vehicle, and on the basis of original monitoring, treading, fighting and other behaviors are recognized by analyzing images, so that an early warning system is realized; by autonomously synthesizing a targeted data set, an algorithm model is optimized, the detection accuracy is improved, and technical support is provided for public safety.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring and early warning, and specifically to an intelligent monitoring and early warning system for emergencies based on unmanned aerial vehicles (UAVs). Background Art

[0002] In recent years, although the public security and public management capabilities in China have been continuously improved, incidents endangering people's property and lives still occur from time to time, especially in crowded places such as public security cases, stampedes, fires, etc. Traditional security monitoring methods mainly rely on surveillance cameras and security personnel for manual monitoring, which have many drawbacks. The large population mobility leads to high monitoring difficulty. During large-scale events, the sudden increase in the flow of people makes it difficult for traditional monitoring methods to cope. The utilization rate of video resources is low, data retrieval is slow, and it is unable to quickly provide effective support for security. The police mainly collect evidence after the event, and it is difficult to detect problems in a timely manner and take measures.

[0003] Currently, intelligent monitoring technology is gradually developing. Abroad, systems equipped with intelligent security solutions have begun to be tested, and domestic researchers have also made progress in anomaly detection algorithms. However, domestic monitoring technology still mainly relies on manual processing of camera information, and the popularity of intelligent cameras is not high, making it difficult to cover temporary gathering places. Therefore, developing an efficient and intelligent monitoring and early warning system for abnormal behaviors of crowds has important practical significance. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the present invention combines an optimized deep learning algorithm with a domestic i.MX 8M Plus embedded platform to implement an intelligent monitoring system carried by a UAV. The system uses the mobility and wide field of view of the UAV to monitor crowded people in real time. On the basis of the original monitoring, by analyzing images to identify behaviors such as stampedes and fights, an early warning system is realized. By independently synthesizing a targeted data set and optimizing the algorithm model, the detection accuracy is improved, providing technical support for public safety.

[0005] To achieve the above object, the present invention provides the following technical solution: An intelligent monitoring and early warning system for emergencies based on a UAV, comprising: A data acquisition module for collecting crowd video image data, carried on the UAV and including a high-definition camera; An image processing and analysis module for processing the collected video images, including an image super-resolution processing unit for enhancing the image resolution based on the FSRCNN algorithm; a moving area division unit for identifying the moving area using grayscale and optical flow fields; An abnormal behavior detection module for capturing the dynamic characteristics of the crowd by combining a spatial domain network and a convolutional neural network, and clustering the moving area through the K-means++ and DBSCAN algorithms to detect and classify abnormal behaviors; The warning module, when detecting abnormal behavior, sends a warning signal containing location information to the management personnel through LoRa communication technology; The UAV embedded platform is used to integrate the above modules to achieve the hardware support and operation of the system.

[0006] Preferably, in the image super-resolution processing unit, the FSRCNN algorithm directly maps the low resolution to the high resolution, and uses convolution operations and a multi-layer structure to reduce the amount of calculation, including the following steps: S101: Extract sub-block information from the low-resolution image; S102: Compress the number of image channels; S103: Perform non-linear mapping to achieve super-resolution operation; S104: Expand and restore the image shape; S105: Upsample the image using the learned deconvolution kernel; S106: Use the mean square error as the loss function to optimize the network.

[0007] Preferably, the motion area division unit uses the VIBE algorithm that fuses motion information for background modeling, fuses the optical flow intensity information, and enhances the distinction between the moving target and the background model through the following formula: M(t,i,j) 2 =f x (t,i,j) ２ +f y (t,i,j) ２ , F(t,i,j) = f(t,i,j) + λM(t,i,j), where (t,i,j) is the pixel position of the t-th frame, M(t,i,j) is the optical flow intensity, f(t,i,j) is the current frame image, and F(t,i,j) is the enhanced image.

[0008] Preferably, the abnormal behavior detection module uses a motion feature description method based on streamline flow to capture long-term motion changes, combines time-domain motion information and spatial structure information for abnormal behavior detection, where the streamline flow is obtained by acquiring the motion field of the connection of particle positions within a certain time interval, the particle positions have sub-pixel accuracy, and the velocity vectors of the streamline flow in the horizontal and vertical directions are solved through the triangular interpolation formula and the least squares method.

[0009] Preferably, the abnormal behavior detection module constructs a streamline flow convolutional neural network based on the deep residual network ResNet101, uses ResNet101 as the backbone network architecture in the time domain and spatial domain, expands the network depth and reduces the network complexity through the residual learning module, and the expression of the residual learning module is Y = G(x,{W i}(}) + x, where x and y are the input and output vectors of the layer, and W i is the weight matrix, and G(x, {W i}) is the residual function of the set of multi-layer convolutional layers.

[0010] Preferably, the abnormal behavior detection module clusters the motion regions through the K-means++ and DBSCAN algorithms to detect and classify abnormal behaviors, including the following steps: S201: Initialize the clustering center by K-means++; S202: Perform DBSCAN density clustering; S203: Pruning strategy; S204: Cluster merging; S205: Build a one-class classifier model; S206: Abnormal detection; S207: Model optimization and adjustment.

[0011] Preferably, the warning module sets a motion continuity constraint to improve the accuracy of abnormal detection. By calculating the intersection area of the abnormal region with the abnormal regions in the previous and next frames and comparing it with the area of the abnormal region in the current frame, if the ratio is lower than the set threshold, the abnormal region is regarded as false; otherwise, the abnormal behavior is confirmed and a warning signal is issued.

[0012] Preferably, in terms of hardware configuration, the drone embedded platform is equipped with a LoRa module in addition to the i.MX 8M Plus platform and the high-definition camera for realizing remote low-power data transmission. And through the implementation of PID control and path planning algorithms, semi-physical simulation testing is carried out using GIS+BIM three-dimensional environment modeling to improve the system performance.

[0013] Compared with the prior art, the present invention provides an intelligent monitoring and warning system based on a drone, having the following beneficial effects: The present invention combines an optimized deep learning algorithm with a domestic i.MX 8M Plus embedded platform to realize an intelligent monitoring system carried by the drone; The system uses the mobility and broad vision of the drone to monitor dense crowds in real time. On the basis of the original monitoring, by analyzing images to identify behaviors such as stampedes and fights, an early warning system is realized; By autonomously synthesizing a targeted data set and optimizing the algorithm model, the detection accuracy is improved, providing technical support for public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is the structural diagram of the intelligent monitoring and warning system of the present invention; Figure 2 is the flow chart of the FSRCNN algorithm of the present invention; Figure 3This is the flowchart of the K-means++ DBSCAN algorithm of the present invention; Figure 4 This is the schematic diagram of the FSRCNN algorithm of the present invention. Detailed implementation manners

[0015] To better understand the purpose, structure and function of the present invention, in order to achieve the intelligent monitoring and early warning function, the present invention combines an optimized deep learning algorithm with a domestic i.MX 8M Plus embedded platform to implement an intelligent monitoring system carried by an unmanned aerial vehicle (UAV); the system uses the mobility and broad vision of the UAV to conduct real-time monitoring of dense crowds, and on the basis of the original monitoring, analyzes images to identify behaviors such as stampedes and fights to implement an early warning system; by independently synthesizing a targeted data set and optimizing the algorithm model, the detection accuracy is improved, providing technical support for public safety. A further detailed description is made of an intelligent monitoring and early warning system for emergency situations based on an unmanned aerial vehicle of the present invention.

[0016] Reference Figures 1-4 , the present invention: an intelligent monitoring and early warning system for emergency situations based on an unmanned aerial vehicle, comprising: A data acquisition module, which is used to acquire crowd video image data and is carried on the UAV. It includes a high-definition camera, and the high-definition camera carried thereon continuously acquires crowd video image data at a certain frame rate, for example, 30 frames of images are acquired per second, ensuring comprehensive and continuous on-site information is obtained; Specifically, in the image super-resolution processing unit, the FSRCNN algorithm directly maps low resolution to high resolution and uses convolution operations and a multi-layer structure to reduce the amount of calculation, including the following steps: S101: Extract sub-block information from the low-resolution image. If the input image is grayscale, the kernel depth is 1, and the number of kernels determines the dimension d of the feature map output by this layer of convolution. Since the feature map passing through the feature extraction part will be sent into the real SR process, the dimension d is crucial, so d is the first sensitive variable and needs to be obtained through experimental comparison. Therefore, this layer can be denoted as Conv(5, d, 1); S102: Compress the number of image channels. Consider first reducing the number of channels of the LR image, then performing SR operations on the low-dimensional LR feature map to reduce the operation parameters, and finally performing a dimension-increasing operation on the generated SR image. Use s 1X1 filter kernels to perform dimension reduction processing on the d-dimensional image from the feature extraction layer (from d to s). s is the second sensitive variable, which can reduce the number of parameters for subsequent SR operations. This layer can be denoted as Conv(1, s, d); S103: Perform non - linear mapping to achieve super - resolution operation. The number of channels of the feature map (d, i.e., the number of kernels) and the depth (the number of convolutional layers m, m is the third sensitive variable). The m - depth determines the accuracy and complexity of SR. The size of the convolutional kernel is 3X3, so this part can be denoted as mxConv(3, s, s); S104: Expand to restore the image shape. Use d 1X1 kernels to restore to the image shape before compression. At this time, it can be seen that these three parts of compression, non - linear mapping, and expansion are symmetric. The expansion part can be denoted as Conv(1, d, s); S105: Upsample the image through the learned de - convolutional kernel, considering the information as propagating from HR to LR. To maintain the symmetric structure, a filter size of 9X9 is required, denoted as DeConv(9, 1, d). De - convolution is different from traditional interpolation methods. Traditional methods have a common reconstruction formula for all reconstructed pixels, while the de - convolutional kernel needs to be learned. It can obtain more accurate results for specific tasks. These notations are diverse and meaningful. If we force these kernels to be the same, the parameters will be used inefficiently (equivalent to adding the input feature map to one), and the performance of Set5 will decrease by at least 0.9 dB; S106: Use the mean - square error as the loss function to optimize the network. The mean - square error MSE: , where X is HR, Y is LR, θ is the parameter, and n is the batchsize.

[0017] Furthermore, the motion area division unit uses the VIBE algorithm that fuses motion information for background modeling, fuses optical flow intensity information, randomly selects a pixel sample in the model for replacement, and randomly selects neighboring pixels for replacement; on this basis, the VIBE algorithm that fuses motion information is proposed. Xiao Bo et al. proposed adding optical flow intensity information to the VIBE algorithm to enhance the distinction between moving objects and the background model. The optical flow extraction method proposed by Zach et al. is used to calculate the optical flow images in the x and y directions for the t - th frame image: f x (t) and f y (t). The definition of the optical flow intensity M(t) is: M(t, i, j) 2 =f x (t, i, j) ２ +f y (t, i, j) ２ , where (t, i, j) is the pixel position of the t - th frame; For the current frame image f(t), the enhanced image F(t) is expressed as: F(t,i,j) = f(t,i,j) + λM(t,i,j), where f(t,i,j) is the current frame image and F(t,i,j) is the enhanced image; It can distinguish moving targets similar to the background, obtain a more complete moving area, accurately detect moving targets, and reduce background interference for subsequent anomaly detection.

[0018] The image processing and analysis module is used to process the collected video images, including an image super-resolution processing unit to enhance the image resolution based on the FSRCNN algorithm; a moving area division unit to identify the moving area using grayscale and optical flow fields; Specifically, the anomaly behavior detection module uses a motion feature description method based on streamline flow to capture the motion changes over a long period of time, and combines the time-domain motion information and the spatial structure information for anomaly behavior detection. The streamline flow is obtained by connecting the particle positions within a certain time interval to obtain the motion field, and the particle positions have sub-pixel accuracy. The velocity vectors of the streamline flow in the horizontal and vertical directions are solved through the triangular interpolation formula and the least squares method.

[0019] Furthermore, inspired by the two-stream convolutional neural network, the time-domain motion information and the spatial structure information are combined for anomaly behavior detection. The deep residual network proposed by He et al. is introduced, and a streamline flow convolutional neural network based on the deep residual network ResNet101 is constructed. ResNet101 is used as the backbone network architecture in the time domain and the spatial domain of the original two-stream network to extract the spatial domain information and the time domain information of the scene. The deep residual network learning algorithm does not directly fit the network function, but fits the residual function of multiple convolutional layer sets. The network depth is expanded through the residual learning module, and the network complexity is reduced at the same time; The anomaly behavior detection module constructs a streamline flow convolutional neural network based on the deep residual network ResNet101, uses ResNet101 as the backbone network architecture in the time domain and the spatial domain, expands the network depth and reduces the network complexity through the residual learning module. The expression of the residual learning module is Y = G(x,{W i}) + x, where x and y are the input and output vectors of the layer respectively, and W i is the weight matrix, and G(x,{W i ) is the residual function of the multi-layer convolutional layer set.

[0020] The anomaly behavior detection module combines the spatial domain network and the convolutional neural network to capture the dynamic characteristics of the crowd, and clusters the moving area through the K-means++ and DBSCAN algorithms to detect and classify the anomaly behavior; Specifically, the abnormal behavior detection module clusters the motion areas through the K-means++ and DBSCAN algorithms to detect and classify abnormal behaviors, including the following steps: S201: K-means++ initializes the clustering centers. Randomly select a data point as the clustering center, calculate the distances from other data points to this center, and select the next center according to the probability distribution of the squared distances. Repeat the above steps until k clustering centers are selected. Assign each data point to the nearest clustering center, and update the clustering center as the mean of each cluster. Repeat the assignment and update until the clustering centers are stable; S202: DBSCAN density clustering. Set the neighborhood radius and the minimum number of samples, identify the core sample points, and outline the dense areas based on these points. Re-evaluate the points initially marked as noise to determine whether they are connected to the core sample points. Finally, the unclassified points are marked as noise; S203: Pruning strategy. Prune based on the distance between intervals and remove the clusters with too large distances. If the distance between clusters is less than or equal to the threshold, consider merging; S204: Cluster merging. Analyze whether the clusters in adjacent partitions should be merged, re-execute DBSCAN on the remaining data, and merge the edge clusters. Finally, each cluster represents a dense area, and the noise points represent the pixel points that do not belong to any area. Based on the clustering results, design a one-class classifier for modeling and prediction; S205: One-class classifier modeling. Use the normal behavior data points identified by DBSCAN to train the one-class classifier, and select the features that can best represent normal behavior as the model input; S206: Abnormal detection. Calculate the distance or reconstruction error between the new data points and the trained classifier, set the threshold, and the data points exceeding the threshold are regarded as abnormal; S207: Model optimization and adjustment. Adjust the parameters of K-means++, DBSCAN, and the one-class classifier according to the performance of the validation set, and use the independent test set to verify the accuracy and robustness of the model.

[0021] Warning module. When abnormal behavior is detected, send a warning signal containing location information to the management personnel through the Lora communication technology; Specifically, the warning module sets the motion continuity constraint to improve the accuracy of abnormal detection. By calculating the intersection area between the abnormal area and the abnormal areas in the previous and next frames, and comparing it with the area of the abnormal area in the current frame, if the ratio is lower than the set threshold, the abnormal area is regarded as false, otherwise, the abnormal behavior is confirmed and a warning signal is sent.

[0022] Drone embedded platform, used to integrate the above modules to achieve the hardware support and operation of the system; Specifically, in terms of hardware configuration, the drone embedded platform is equipped with a LoRa module in addition to the i.MX 8M Plus platform and the high-definition camera, which is used to achieve remote low-power data transmission. Moreover, by implementing the PID control and path planning algorithms, and using the GIS+BIM three-dimensional environment modeling for semi-physical simulation testing, the system performance is improved.

[0023] It can be understood that the present invention is described through some embodiments. Those skilled in the art know that, without departing from the spirit and scope of the present invention, various changes or equivalent replacements can be made to these features and embodiments. Additionally, under the teaching of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.

Claims

1. An intelligent emergency monitoring and early warning system based on drones, characterized in that: include: The data acquisition module is used to collect crowd video image data, which is carried on the drone and includes a high-definition camera; The image processing and analysis module is used to process the collected video images, including an image super-resolution processing unit, which improves the image resolution based on the FSRCNN algorithm; a motion area division unit, which uses grayscale and optical flow fields to identify motion areas; The abnormal behavior detection module combines the spatial domain network and convolutional neural network to capture the dynamic characteristics of the crowd, and clusters the motion areas through the K-means++ and DBSCAN algorithms to detect and classify abnormal behaviors; The early warning module sends an early warning signal containing location information to the management personnel through Lora communication technology when abnormal behavior is detected; The UAV embedded platform is used to integrate the above modules to realize the hardware support and operation of the system.

2. The intelligent emergency monitoring and early warning system based on drone according to claim 1 is characterized in that: In the image super-resolution processing unit, the FSRCNN algorithm directly maps low resolution to high resolution and uses convolution operations and multi-layer structures to reduce the amount of calculation, including the following steps: S101: extracting sub-block information from a low-resolution image; S102: compressing the number of image channels; S103: performing nonlinear mapping to achieve super-resolution operation; S104: Expand and restore the image shape; S105: upsampling the image using the learned deconvolution kernel; S106: Optimize the network using mean square error as the loss function.

3. The intelligent emergency monitoring and early warning system based on drone according to claim 2 is characterized in that: The motion region division unit uses the VIBE algorithm that integrates motion information to perform background modeling and integrates optical flow intensity information to enhance the distinction between the moving target and the background model through the following formula: M(t,i,j) 2 =f x (t,i,j) ２ +f y (t,i,j) ２ , F(t,i,j)=f(t,i,j)+λM(t,i,j), where (t,i,j) is the pixel position of the tth frame, M(t,i,j) is the optical flow intensity, f(t,i,j) is the current frame image, and F(t,i,j) is the enhanced image.

4. The intelligent emergency monitoring and early warning system based on drone according to claim 3 is characterized in that: The abnormal behavior detection module adopts a motion feature description method based on pulse line flow to capture motion changes over a long period of time, and combines time domain motion information and spatial structure information to detect abnormal behavior, wherein the pulse line flow is obtained by obtaining the motion field of the particle position line within a certain time interval, and the particle position has sub-pixel level accuracy. The velocity vectors of the pulse line flow in the horizontal and vertical directions are solved by the triangular interpolation formula and the least squares method.

5. The intelligent emergency monitoring and early warning system based on drone according to claim 4 is characterized by: The abnormal behavior detection module builds a pulse flow convolutional neural network based on the deep residual network ResNet101, uses ResNet101 as the backbone network architecture in the time domain and space domain, and expands the network depth and reduces the network complexity through the residual learning module. The expression of the residual learning module is Y = G(x,{W i }) + x, where x and y are the input and output vectors of the layer, respectively, and W i is the weight matrix, G(x,{W i }) is the residual function of a collection of multi-layer convolutional layers.

6. The intelligent emergency monitoring and early warning system based on drone according to claim 5 is characterized by: The abnormal behavior detection module clusters the motion regions by K-means++ and DBSCAN algorithms to detect and classify abnormal behaviors, including the following steps: S201: K-means++ initializes cluster centers; S202: DBSCAN density clustering; S203: pruning strategy; S204: cluster merging; S205: Modeling of one-class classifier; S206: Anomaly detection; S207: Model optimization and adjustment.

7. The intelligent emergency monitoring and early warning system based on drone according to claim 6 is characterized in that: The early warning module sets motion continuity constraints to improve the accuracy of anomaly detection. It calculates the intersection area of ​​the abnormal region and the abnormal regions in the previous and next frames and compares it with the area of ​​the abnormal region in the current frame. If the ratio is lower than the set threshold, the abnormal region is regarded as false. Otherwise, the abnormal behavior is confirmed and an early warning signal is issued.

8. The intelligent emergency monitoring and early warning system based on drone according to claim 7 is characterized in that: In terms of hardware configuration, the UAV embedded platform is equipped with a LoRa module in addition to the i.MX 8M Plus platform and high-definition camera for long-distance low-power data transmission. It also improves system performance by implementing PID control and path planning algorithms and using GIS+BIM three-dimensional environment modeling for semi-physical simulation testing.

Images