Video low-loss compression method and monitoring system for crowd monitoring of unmanned aerial vehicles

Abstract

The application particularly relates to a video low-loss compression method and a monitoring system for crowd monitoring of a UAV. The method comprises the following steps: obtaining a crowd monitoring video to be compressed; in the time domain, cutting redundant video frames according to the inter-frame feature similarity of the crowd monitoring video to generate a plurality of frame target video frames after removing the redundant video frames; in the space domain, extracting a non-background area and a background area of each frame target video frame and performing dynamic coding to generate a plurality of dynamic coding video frames; and constructing a space-time compressed video of the crowd monitoring video based on the plurality of dynamic coding video frames. According to the inter-frame feature similarity of the crowd monitoring video, the redundant video frames are cut in the time domain, and the video frames are dynamically coded in the space domain, so that the video transmission delay can be greatly reduced under the condition of ensuring the accuracy of crowd detection and recognition, thereby improving the effectiveness of the crowd monitoring video compression and meeting the application requirements of the UAV in actual large-scale urban crowd monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video low-loss compression, and particularly relates to a video low-loss compression method and a monitoring system for unmanned aerial vehicle crowd monitoring. BACKGROUND

[0002] City crowd monitoring is very important for city emergency management and maintaining city safety. At present, the main way of crowd monitoring is to use the fixed camera monitoring system deployed in the city. However, the fixed camera has the disadvantages of blind area and poor flexibility of mobile deployment, which seriously restricts city emergency prevention and management. At the same time, the deployment and maintenance of the fixed camera need huge cost.

[0003] On the contrary, the unmanned aerial vehicle provides an important means for large-scale city crowd monitoring due to its low cost, mobile flexibility and wide range of shooting pictures. At present, some work has been done on crowd monitoring system based on unmanned aerial vehicle, but the existing system mainly has the following problems:

[0004] 1) High latency of transmitted video. Due to the limitation of the bandwidth of wireless transmission of the unmanned aerial vehicle, the high-capacity and high-definition crowd monitoring video will result in extremely high transmission latency. It is tested that when the distance between the unmanned aerial vehicle and the ground station is only 30 m, the transmission time of 1280x800 (800p) resolution video to the ground station is more than 30 minutes, which results in serious lag of crowd monitoring information and cannot guarantee the real-time safety of city crowd management.

[0005] 2) Low precision of compressed video. Although reducing the resolution can compress the video capacity to reduce the transmission latency, it will seriously affect the accuracy of crowd detection. It is tested that when the transmission distance is fixed, if the video frame is compressed from 1280x1080 (800p) high resolution to 480x320 (320p) low resolution, the transmission latency is reduced by 65.7%, but the detection accuracy is sharply reduced from 80% to 64%, and the recall rate is also reduced by 4.4%, so that the high accuracy and recall rate requirements of crowd target detection cannot be met.

[0006] In summary, the existing crowd monitoring system based on unmanned aerial vehicle has the problem that it cannot guarantee high-precision crowd detection and identification and low-latency video transmission at the same time, which limits its effectiveness in large-scale city crowd monitoring applications. Therefore, there is an urgent practical need to design a video low-loss compression method that can greatly reduce the video transmission latency while guaranteeing the accuracy of crowd detection and identification. The method can assist the unmanned aerial vehicle in the practical application of large-scale city crowd monitoring. SUMMARY

[0007] In view of the above problems of the prior art, the technical problem to be solved by the present application is to provide a video low-loss compression method for unmanned aerial vehicle crowd monitoring and a monitoring system, which can greatly reduce the video transmission delay under the condition of ensuring the accuracy of crowd detection and recognition by cutting redundant video frames in the time domain according to the inter-frame feature similarity of the crowd monitoring video and dynamically encoding the video frames in the spatial domain, so as to improve the effectiveness of crowd monitoring video compression and meet the application requirements of unmanned aerial vehicles in actual large-scale urban crowd monitoring.

[0008] In order to solve the above technical problems, the present application adopts the following technical solutions:

[0009] The video low-loss compression method for unmanned aerial vehicle crowd monitoring comprises the following steps:

[0010] S1: obtaining a crowd monitoring video to be compressed;

[0011] S2: cutting redundant video frames in the time domain according to the inter-frame feature similarity of the crowd monitoring video to generate a plurality of target video frames after removing redundant video frames;

[0012] S3: extracting non-background regions and background regions of each target video frame and dynamically encoding the same in the spatial domain to generate a plurality of dynamic encoding video frames;

[0013] S4: constructing a space-time compressed video of the crowd monitoring video based on the plurality of dynamic encoding video frames.

[0014] Preferably, the redundant video frame cutting is performed by the following steps:

[0015] S201: converting the crowd monitoring video into corresponding crowd video frames;

[0016] S202: extracting image crowd features of the crowd video frames by a lightweight network;

[0017] S203: extracting image frequency domain information and image structure information of the crowd video frames as corresponding image scene features;

[0018] S204: taking the image crowd features and the image scene features as image total features of the crowd video frames, and performing feature clustering and redundant video frame cutting on the crowd video frames based on the feature similarity of the image total features.

[0019] Preferably, the image crowd features of the crowd video frames are extracted by the following steps:

[0020] S2021: extracting the image crowd features of the kth crowd video frame I kA crowd detection is performed to generate a corresponding crowd detection result; the crowd detection result includes a number of crowd detection proposal boxes in a current crowd video frame and a center coordinate and a confidence c of each crowd detection proposal box, wherein the number of crowd detection proposal boxes represents a crowd number N;

[0021] S2022: performing two-dimensional convolution on the crowd detection result of the kth crowd video frame I k to generate a corresponding crowd density map m k ;

[0022] The formula is described as:

[0023]

[0024] In the formula, m k represents the crowd density map of the kth crowd video frame I k ; x and represent two-dimensional Gaussian kernels; represents an impulse function; represents the confidence of the ith crowd detection proposal box of the kth crowd video frame I k ; represents the center coordinate of the ith crowd detection proposal box of the kth crowd video frame I k ; and N represents the number of crowd detection proposal boxes detected in the kth video frame.

[0025] S2023: performing uniform pooling on the crowd density map m k of the kth crowd video frame I k to obtain a crowd density map P k after image size compression, and further taking the crowd density map P k as an image crowd feature of the kth crowd video frame I k .

[0026] Preferably, the image scene feature of the crowd video frame is extracted by the following steps:

[0027] S2031: performing two-dimensional discrete cosine transform on the kth crowd video frame I k and retaining a low-frequency region of a spectrum energy set as a low-frequency spectrum;

[0028] The formula is described as:

[0029]

[0030] In the formula, the coefficient

[0031]

[0032] In the formula, the coefficient N1 represents the two-dimensional cosine discrete transform value at (k1, k2) in the frequency domain; N1 and N2 represent the height and width of the crowd video frame, respectively; n1 and n2 represent the indicator variables in the summation formula; k1 and k2 represent the coordinates in the two-dimensional frequency domain.

[0033] S2032: Quantize the low-frequency spectrum, that is, quantize each low-frequency cosine discrete transform value to 0 or 1 based on whether the low-frequency cosine discrete transform value of the low-frequency spectrum is higher than its low-frequency cosine discrete transform mean, and use the quantized low-frequency spectrum as the k-th crowd video frame I. k Frequency domain characteristics F k ;

[0034] S2033: Extract the k-th frame of the crowd video frame I k The image structure information is used to generate corresponding image structure features;

[0035] S2034: Transform the k-th frame of the crowd video frame I k Frequency domain characteristics F k Combined with image structural features, it serves as the k-th frame of the crowd video frame I. k Image scene features S k .

[0036] Preferably, the image structural features include brightness features L k Contrast characteristics Y k and color distribution characteristics

[0037] 1) I of the k-th frame of the crowd video k The area is uniformly divided into rectangular blocks, and the local brightness μ of each block is calculated using the following formula. X Local contrast σ X ;

[0038]

[0039]

[0040] In the formula: H and M represent the length and width of the block, respectively; X(i,j) represents the gray value of the pixel at position (i,j) in the image block;

[0041] The k-th frame of the crowd video frame I k The local brightness and contrast values ​​of all blocks are concatenated to form the corresponding brightness feature L. k and contrast feature Y k ;

[0042] 2) For the k-th frame of the crowd video frame I k The R, G, and B channels are used to calculate the pixel distribution ratio in different grayscale ranges. as a corresponding color distribution feature

[0043] Preferably, the redundant frame clipping is performed by the following steps:

[0044] S2041: Cover the consecutive crowd video frames with a sliding window, and according to the image global features of all the crowd video frames within the coverage of the sliding window, take the cosine similarity of the following formula as the inter-frame feature similarity;

[0045]

[0046] In the formula, cos(θ) represents the cosine similarity; X i , Y i represent the components of the image global features of two different crowd video frames in the sliding window; n represents the dimension of the image global features of the crowd video frames;

[0047] S2042: According to the inter-frame feature similarity of the crowd video frames within the sliding window, perform feature clustering on the crowd video frames by using a clustering algorithm, and perform clipping judgment on each class of video frames in the clustering result according to a clipping threshold;

[0048] S2043: Move the sliding window by a set step, and continue to perform feature clustering and clipping judgment until the sliding window traverses all the crowd video frames.

[0049] Preferably, the dynamic encoding is performed by the following steps:

[0050] S301: Perform edge detection on the target video frame, and mark the spatial blocks with edge detection intensity lower than an edge detection threshold as background regions, and mark the spatial blocks with edge detection intensity higher than the threshold as non-background regions;

[0051] S302: Reuse the crowd detection result in step S2021, correct the crowd regions misjudged as background regions to non-background regions; then take all the non-background regions as the encoding interest domain of the target video frame, and take the background regions as the non-encoding interest domain of the target video frame;

[0052] S303: According to the encoding interest domain of the target video frame, dynamically calculate the coding quality map, that is, define the coding quality of each image block in the target video frame, wherein the encoding interest domain is set as lossless coding quality, and the non-encoding interest domain is set as lossy coding quality;

[0053] S304: According to the coding quality map of each frame of target video frame, the video encoder performs corresponding lossless encoding or lossy encoding on each image block, that is, lossless compression of the non-background regions and lossy compression of the background regions are realized.

[0054] The application also discloses a UAV crowd monitoring system, which comprises:

[0055] a UAV end for collecting crowd monitoring videos of a target area;

[0056] a UAV edge processing device for performing spatio-temporal fusion compression on the crowd monitoring videos collected by the UAV end through the video spatio-temporal fusion low-loss compression method of the application, to generate corresponding spatio-temporal compressed crowd videos;

[0057] a ground control end for performing crowd monitoring applications based on the spatio-temporal compressed crowd videos after spatio-temporal fusion compression.

[0058] Preferably, the crowd monitoring applications of the ground control end include crowd detection and counting, crowd target tracking and crowd social distance monitoring.

[0059] Compared with the prior art, the video low-loss compression method for UAV crowd monitoring in the application has the following beneficial effects:

[0060] According to the inter-frame feature similarity of the crowd monitoring videos, the application performs redundant video frame cutting in the time domain, so that the number of redundant video frames can be greatly reduced without affecting the crowd detection accuracy, i.e., the video transmission delay can be greatly reduced under the condition of ensuring the crowd detection and recognition accuracy, thereby improving the effectiveness of crowd monitoring video compression and meeting the application requirements of UAV in actual large-scale urban crowd monitoring.

[0061] On the basis of removing redundant video frames in the time domain to obtain target video frames, the application extracts the non-background area (crowd monitoring target area) and the background area (such as street, sky and other non-crowd monitoring target areas) of the target video frames in the spatial domain, and respectively performs lossless encoding (lossless compression) and lossy encoding (lossy compression) i.e., dynamic encoding, so that the redundant encoding data amount of the background part can be maximized to reduce under the condition of ensuring the picture quality of the crowd area in the video frame, thereby being able to balance the video compression effect and the picture quality of the crowd monitoring area, i.e., the video transmission delay can be greatly reduced under the condition of ensuring the crowd detection and recognition accuracy, thereby further improving the effectiveness of crowd monitoring video compression and better meeting the application requirements of UAV in actual large-scale urban crowd monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below with reference to the drawings, in which:

[0063] Figure 1 the network framework diagram of the video low-loss compression method for UAV crowd monitoring;

[0064] Figure 2Network framework diagram for video temporal sampling based on feature clustering

[0065] Figure 3 Network framework diagram for video spatial sampling based on dynamic coding

[0066] Figure 4 Diagram for the impact of different video resolutions on crowd detection performance

[0067] Figure 5 Diagram for the impact of different crowd density scenarios on crowd detection performance

[0068] Figure 6 Diagram for the impact of different algorithm modules on crowd detection performance

[0069] Figure 7 Diagram for the impact of different image block division bins on crowd detection performance

[0070] Figure 8 Real-time crowd monitoring prototype system based on edge intelligent unmanned aerial vehicle

[0071] Figure 9 Diagram for crowd detection results

[0072] Figure 10 Network framework diagram for unmanned aerial vehicle crowd monitoring system DETAILED DESCRIPTION

[0073] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings of the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0074] It should be noted that similar reference numerals and letters refer to similar items throughout the accompanying drawings, and once an item is defined in one drawing, it is not necessary to further define and explain it in subsequent drawings. In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed during use, and are merely for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third", and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", and the like do not mean that the components must be absolutely horizontal or vertical, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the present application, it should also be noted that unless otherwise specifically specified and limited, the terms "provided", "mounted", "connected", "linked", and the like should be broadly understood, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, or can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0075] The following will be further described in detail through specific embodiments:

[0076] Example 1:

[0077] Currently, there are some works on crowd monitoring systems based on UAVs. For example, the document (Rezaei Mahdi, Azarmi Mohsen. Deep social: Social distancing monitoring and infection risk assessment in covid-19 pandemic) uses a UAV to monitor the social distance of the crowd. However, the system has poor real-time performance because it does not consider the transmission delay of the monitoring video. The document (Xiao Liang, Ding Yuzhen, Huang Jinhao, et al. UAV anti-jamming video transmissions with QoE guarantee: A reinforcement learning-based approach) uses reinforcement learning to select a video compression transmission strategy for the UAV. Although this approach can reduce the transmission delay of the system, it cannot guarantee the accuracy of crowd monitoring. The documents (Yu Tao, Takaku Yoshitaka, Kaieda Yohei, et al. Design and PoC Implementation of Mmwave-Based Offloading-Enabled UAV Surveillance System and Akhter Zubair, Bilal Rana M, Telegenov Kuat, et al. Indigenously Developed HD Video Transmission System for UAVs Employing a 3x3 MIMO Antenna System) apply millimeter wave communication and Multiple-Input and Multiple-Output (MIMO) technology to the UAV monitoring system to transmit high-resolution video in real time. However, the need for additional equipment such as millimeter wave radars and MIMO antennas significantly increases the deployment cost of the system.

[0078] In summary, existing research on crowd monitoring based on UAVs either requires additional equipment and has high costs or cannot guarantee both high-precision crowd monitoring and low video transmission, making it difficult to meet the application requirements of large-scale urban crowd monitoring.

[0079] At the same time, some existing works study UAV crowd monitoring systems and crowd target detection algorithms.

[0080] 1) Unmanned Aerial Vehicle (UAV) Crowd Monitoring Systems. Due to their small size, low cost, and high security, UAVs are often combined with crowd monitoring methods for urban security. For example, the literature (Singh Amarjot, Patil Devendra, Omkar Sn. Eye in the Sky: Real-Time Drone Surveillance System (DSS) for Violent Individuals Identification Using ScatterNet Hybrid Deep Learning Network) uses the ScatterNet deep learning network for human pose estimation, thereby identifying violent behavior in crowds captured by UAVs in real time. The literature (Castellano Giovanna, Castiello Ciro, Mencar Corrado, et al. Crowdcounting from unmanned aerial vehicles with fully-convolutional neural networks) embeds cameras and GPUs on UAVs for real-time crowd density estimation. (The literature...) Marcin, Jakub and Wieczorek Michal's "Deep learning based crowd counting model for drone-assisted systems" utilizes a specially designed drone equipped with a large edge device to directly run a deep neural network for crowd monitoring. However, this approach results in reduced drone system endurance, making it unsuitable for use with conventional drones. As drone technology continues to advance, its applications in various fields are expanding, and new application scenarios are placing higher demands on drone performance. However, due to limitations in endurance and size, drones cannot carry large processing devices, making it difficult for them to handle complex computational tasks. Therefore, addressing the limitation of edge computing capabilities in drones has become a key research focus.

[0081] 2) Crowd target detection algorithm. Convolutional neural network (CNN) occupies a dominant position in the field of target detection, recognition and tracking with its excellent performance. Various CNN network structures have been applied to the field of crowd target detection to improve accuracy. Literature (Zhang Yingying, Zhou Desen, Chen Siqin, et al. Single-image crowd counting via multi-column convolutional neural network) designs a multi-column CNN to fuse crowd density maps of different sizes, and then models crowds of different scales. Literature (Lee Hyungtae, Eum Sungmin, Kwon Heesung. Mer-cnn: Multi-expert r-cnn for object detection) proposes a multi-expert region convolutional neural network for target detection to capture the appearance changes of crowds caused by different crowd poses and angles. Literature (Liu Jiang, Gao Chenqiang, Meng Deyu, et al. DecideNet: Counting varying density crowds through attention guided detection and density estimation) combines the advantages of target detection and density regression-based crowd counting, and designs an adaptive weight crowd counting neural network based on attention mechanism. Literature (Song Qingyu, Wang Changan, Wang Yabiao, et al. To choose or to fuse?Scale selection for crowd counting) proposes a crowd density regression model with multi-scale feature representation, which effectively solves the image scale change problem in crowd density estimation. Although the current use of neural network learning deep features has become an effective way to solve the problem of crowd counting, the crowd detection algorithm based on deep neural network has very high computational performance requirements for processing equipment, making it difficult for current small unmanned aerial vehicle edge devices to run.

[0082] To solve the above problems, the technical scheme is disclosed in the embodiment as follows.

[0083] A video low-loss compression method for unmanned aerial vehicle crowd monitoring is disclosed in the embodiment.

[0084] As shown in Figure 1 The video low-loss compression method for unmanned aerial vehicle crowd monitoring comprises the following steps.

[0085] S1: obtaining a crowd monitoring video to be compressed;

[0086] S2: video time domain sampling based on feature clustering: in the time domain, redundant video frames are cut according to the inter-frame feature similarity of the crowd monitoring video, and a plurality of frame target video frames after removing the redundant video frames are generated;

[0087] S3: video space sampling based on dynamic coding: in the spatial domain, the non-background area and the background area of each frame target video frame are extracted and dynamically coded (i.e. lossless coding and lossy coding are performed respectively), and a plurality of frame dynamic coding video frames are generated;

[0088] S4: constructing a space-time compressed video of the crowd monitoring video based on the plurality of frame dynamic coding video frames.

[0089] In this embodiment, the plurality of frame dynamic coding video frames are spliced together in time sequence to constitute the space-time compressed video.

[0090] Firstly, the present application cuts redundant video frames according to the inter-frame feature similarity of the crowd monitoring video in the time domain to generate target video frames, then extracts the non-background area and the background area of the target video frames and respectively performs dynamic coding (i.e. lossless coding and lossy coding) in the spatial domain, and finally constructs a space-time compressed video based on the dynamic coding video frames. On the one hand, the present application cuts redundant video frames according to the inter-frame feature similarity of the crowd monitoring video in the time domain, so that the number of redundant video frames can be greatly reduced without affecting the accuracy of crowd detection, i.e. the video transmission delay can be greatly reduced under the condition of ensuring the accuracy of crowd detection and recognition, thereby improving the effectiveness of crowd monitoring video compression and meeting the application requirements of unmanned aerial vehicles in actual large-scale urban crowd monitoring. On the other hand, based on the target video frames obtained by removing redundant video frames in the time domain, the present application extracts the non-background area (crowd monitoring target area) and the background area (such as street, sky and other non-crowd monitoring target areas) of the target video frames in the spatial domain, and respectively performs lossless coding (lossless compression) and lossy coding (lossy compression) i.e. dynamic coding, so that the amount of redundant coding data of the background part can be maximized under the condition of ensuring the picture quality of the crowd area in the video frame, thereby being able to balance the video compression effect and the picture quality of the crowd monitoring area, i.e. the video transmission delay can be greatly reduced under the condition of ensuring the accuracy of crowd detection and recognition, thereby further improving the effectiveness of crowd monitoring video compression and better meeting the application requirements of unmanned aerial vehicles in actual large-scale urban crowd monitoring.

[0091] The experimental results show that the technical scheme of the present application can reduce the transmission delay by 31.3% and the crowd counting accuracy is reduced by not more than 2%.

[0092] In order to better explain the technical solutions of the present application, the main mathematical symbols are explained by Table 1.

[0093] Explanation of main mathematical symbols

[0094]

[0095] In the implementation process, the video time domain sampling based on feature clustering refers to cutting the unmanned aerial vehicle captured video in the time domain according to the interframe feature similarity of the unmanned aerial vehicle captured video, so as to greatly reduce the number of redundant video frames while not affecting the crowd detection accuracy. Figure 2 As shown in the figure, first, the crowd features and scene features of the video frames captured by the unmanned aerial vehicle are extracted, then the feature clustering and redundant frame cutting are performed according to the feature similarity of the video frames, and finally the number of redundant frames in the output video frames is significantly reduced compared with the original input video. The redundant video frame cutting is performed by the following steps:

[0096] S201: converting the crowd monitoring video into corresponding crowd video frames;

[0097] S202: extracting image crowd features (such as crowd number and position distribution, etc.) of the crowd video frames by a lightweight network;

[0098] S203: extracting image frequency domain information and image structure information of the crowd video frames as corresponding image scene features;

[0099] S204: taking the image crowd features and image scene features as image total features of the crowd video frames, and performing feature clustering and redundant video frame cutting on the crowd video frames based on the feature similarity of the image total features.

[0100] In this embodiment, the image crowd features and image scene features can be added or weighted summed to form the image total features.

[0101] The present application extracts the image crowd features and image scene features of the crowd video frames as image total features, and then performs feature clustering and redundant video frame cutting on the crowd video frames based on the feature similarity of the image total features, so that the number of redundant video frames can be greatly reduced while not affecting the crowd detection accuracy, thereby greatly reducing the video transmission delay under the condition of ensuring the accuracy of crowd detection and recognition.

[0102] Specifically, the crowd features (such as crowd number and position distribution, etc.) of the image are extracted by using a lightweight network model as the similarity judgment basis of time domain sampling. The image crowd features of the crowd video frames are extracted by the following steps:

[0103] S2021: extracting the image crowd features of the kth crowd video frame I by using the trained lightweight network yolov4-tinyk crowd detection is performed to generate a corresponding crowd detection result; the crowd detection result includes a number of crowd detection proposal boxes in a current crowd video frame and a center coordinate and a confidence c of each crowd detection proposal box, and the number of crowd detection proposal boxes represents a crowd number N;

[0104] S2022: performing two-dimensional convolution on the crowd detection result of the kth crowd video frame I k to generate a corresponding crowd density map m k ;

[0105] The formula is described as:

[0106]

[0107] In the formula, m k represents the crowd density map of the kth crowd video frame I k ; x and represent a two-dimensional Gaussian kernel; represents an impulse function; represents the confidence of the ith crowd detection proposal box of the kth crowd video frame I k ; represents the center coordinate of the ith crowd detection proposal box of the kth crowd video frame I k ; and N represents the number of crowd detection proposal boxes detected in the kth video frame.

[0108] S2023: performing uniform pooling on the crowd density map m k of the kth crowd video frame I k to obtain a crowd density map P k after image size compression, and further taking the crowd density map P k as an image crowd feature of the kth crowd video frame I k .

[0109] The present application extracts the crowd features (such as the number of crowds and the position distribution of crowds) of images by a light network model as the similarity judgment basis of time domain sampling, so that the image crowd feature extraction of crowd video frames can be better realized, and then the feature clustering and redundant video frame cutting of crowd video frames based on the image total feature can be assisted to realize, so that the number of redundant video frames can be greatly reduced while the crowd detection accuracy is not affected.

[0110] Specifically, image frequency domain information and image structure information (brightness, contrast, color distribution, etc.) are used as scene features to determine whether the background of the video captured in the UAV crowd monitoring process has changed. In extracting the scene features, the low-frequency information of the image is mainly retained to depict the scene profile of the video captured by the UAV. The image scene features of the crowd video frame are extracted by the following steps:

[0111] S2031: performing two-dimensional discrete cosine transform on the kth crowd video frame I k and retaining the low-frequency spectrum;

[0112] In this embodiment, the low-frequency spectrum can be set by the user as needed. For example, the low-frequency region with concentrated spectrum energy can be retained as the low-frequency spectrum, or the region with spectrum energy lower than a threshold value can be retained as the low-frequency spectrum.

[0113] The formula is described as:

[0114]

[0115] wherein the coefficient

[0116]

[0117] In the formula: represents the two-dimensional cosine discrete transform value at the frequency domain (k1, k2); N1 and N2 represent the height and width of the crowd video frame, respectively; n1 and n2 represent the index variables in the summation formula; k1 and k2 represent the coordinates of the two-dimensional frequency domain;

[0118] In this embodiment, the meaning of the above formula is to calculate the value of each point in the frequency domain after image transformation, where k1 and k2 range from 0 to N1-1 and 0 to N2-1, respectively, and k1 and k2 are both integers. The left side of the formula returns the matrix value of the frequency domain transformation of the N1×N2 image.

[0119] S2032: quantizing the low-frequency spectrum, i.e., quantizing each low-frequency cosine discrete transform value as 0 or 1 according to whether the low-frequency cosine discrete transform value of the low-frequency spectrum is higher than the low-frequency cosine discrete transform average value, and taking the quantized low-frequency spectrum as the frequency domain feature F k of the kth crowd video frame I k ;

[0120] S2033: extracting the image structure information of the kth crowd video frame I k , and generating the corresponding image structure feature;

[0121] S2034: combining the frequency domain feature F k and the image structure feature of the kth crowd video frame I kand image structure features, as the kth frame of crowd video frame I k image scene features S k .

[0122] The image structure features include luminance features L k , contrast features Y k and color distribution features

[0123] In this embodiment, the luminance features, the contrast features and the color distribution features can be added or weightedly summed to construct the image structure features.

[0124] 1) The kth frame of crowd video frame I k is evenly divided into rectangular blocks, and the local luminance μ X and the local contrast σ X of each block are calculated by the following formula

[0125]

[0126]

[0127] In the formula, H and M represent the length and the width of the block respectively, and X(i,j) represents the gray value of the pixel at the (i,j) position in the image block.

[0128] The local luminance and the local contrast values of all blocks of the kth frame of crowd video frame I k are spliced as the corresponding luminance features L k and the contrast features Y k respectively.

[0129] 2) The pixel distribution proportions of different gray scale intervals of the R, G and B channels of the kth frame of crowd video frame I k are calculated as the corresponding color distribution features

[0130] In this embodiment, the pixel distribution proportions can be added or weightedly summed to construct the color distribution features.

[0131] In this embodiment, the calculation of can be realized by using existing mature means.

[0132] is the histogram of the gray values of the R, G and B channels of the kth frame of crowd video frame I k . The T channel histogram of the kth frame is represented as represents the distribution proportion of the gray scale interval i on the channel T of the kth frame.

[0133] The gray scale interval refers to: dividing the gray scale values 0-255 into n intervals of equal length, for each gray scale interval, counting the number of pixel points falling in each gray scale interval after the image is grayed. For example, for a 20x20 image I k For the red R channel of the image, divide it into 4 gray scale intervals, then the length of each gray scale value interval is 64, i.e. [0, 63], [64, 127], [128, 191], [192, 255], then count the proportion of image pixel gray scale values distributed in each interval, for example, 100 pixel gray scale values are in the [0, 63] interval, 100 pixel gray scale values are in the [64, 127] interval, 100 pixel gray scale values are in the [128, 191] interval, and 100 pixel gray scale values are in the [192, 255] interval.

[0134] Therefore,

[0135] Other G, B channels have the same calculation method as the R channel, and the same reasoning can be used to calculate

[0136] The image frequency domain information and image structure information (brightness, contrast, color distribution, etc.) of the video frame are extracted as scene features to determine whether the background of the video captured in the unmanned aerial vehicle crowd monitoring process changes, so that the image scene feature extraction of the crowd video frame can be better realized, and then the feature clustering and redundant video frame cutting of the crowd video frame based on the image total feature can be assisted to realize, so that the number of redundant video frames can be greatly reduced while the crowd detection accuracy is not affected.

[0137] Specifically, the redundant frame cutting based on feature clustering uses the extracted image crowd features and scene features as image total features, and uses feature similarity for clustering, so as to cut the redundant video frames.

[0138] The redundant frame cutting is performed by the following steps:

[0139] S2041: Use a sliding window to cover the continuous crowd video frames, and calculate the inter-frame feature similarity according to the cosine similarity or Pearson correlation coefficient of the image total features of all crowd video frames in the coverage range according to the following formula;

[0140]

[0141]

[0142] In the formula: cos(θ) represents the cosine similarity; ρ X,Y represents the Pearson correlation coefficient; X i , Yi represents the component of the image total feature of two different crowd video frames in the sliding window; n represents the dimension of the image total feature of the crowd video frame; X, Y represent the image total features of two different crowd video frames in the sliding window.

[0143] S2042: According to the inter-frame feature similarity of the crowd video frames in the sliding window, the crowd video frames are clustered by using a clustering algorithm, and each class of video frames in the clustering result is subjected to a shear judgment according to a shear threshold.

[0144] In this embodiment, the K-Means clustering algorithm, the hierarchical clustering algorithm (divided into agglomerative hierarchical clustering and divisive hierarchical clustering), the DBSCAN clustering algorithm or the support vector machine clustering algorithm can be used to cluster the crowd video frames.

[0145] The feature clustering refers to that, for each frame of crowd video frame, the extracted image crowd feature and scene feature are used as the image total feature, then the similarity between two crowd video frames is calculated according to the total feature of each frame of crowd video frame, and finally the crowd video frames are clustered by using the feature similarity.

[0146] The shear judgment (redundant frame shear) refers to that, according to the shear threshold, each class of video frames in the clustering result is subjected to a shear judgment. That is, in a certain class of video frames, if the average similarity of a frame of crowd video frame with other crowd video frames is higher than the similarity threshold, the video frame is regarded as a redundant frame and is sheared.

[0147] S2043: The sliding window is moved by a set step, and the feature clustering and the shear judgment are continuously performed until the sliding window traverses all the crowd video frames.

[0148] By using the extracted image crowd feature and scene feature as the image total feature and clustering by using the feature similarity, the redundant video frames are sheared, so that the number of redundant video frames can be greatly reduced without affecting the crowd detection accuracy, thereby the effectiveness of crowd monitoring video compression can be further improved, and the application requirements of large-scale urban crowd monitoring in practice can be better met.

[0149] In summary, the specific process of the video time domain sampling based on feature clustering is as follows: inputting the original continuous video frame sequence The first line initializes the parameters, the second to sixth lines extract the image features and calculate the inter-frame feature similarity based on the sliding window, the seventh to The ninth line performs feature clustering and redundant frame shear according to the inter-frame feature similarity, and finally outputs the sampled video frame sequence

[0150] The process of the video time domain sampling based on feature clustering:

[0151] Input: Original N consecutive video frame sequence

[0152] Output: Sampling K-frame video frame sequence

[0153]

[0154] In specific implementation, video spatial sampling based on dynamic coding is used. This involves compressing the background areas (such as streets, sky, and other non-crowd monitoring target areas) within the video frames captured by the drone in the spatial domain, while maintaining the image quality of the crowd monitoring areas. For example... Figure 3 As shown, edge detection is first performed on the drone video to filter out the background. Then, using lightweight crowd detection results, crowd regions that were mistakenly identified as background during background filtering are corrected to obtain a more accurate division of the crowd region (encoded region of interest) and the background region (non-encoded region of interest). Finally, dynamic quality coding is performed on the image based on the encoded region of interest division, that is, lossless coding is performed on the encoded region of interest (Encoding RoI), while lossy coding is performed on the non-encoded region of interest (non-Encoding RoI). Specifically, lossless coding and lossy coding are performed through the following steps:

[0155] S301: Perform edge detection on the target video frame and mark spatial blocks with edge detection intensity below the edge detection threshold as background regions, and spatial blocks with edge detection intensity above the threshold as non-background regions;

[0156] In this embodiment, the edge texture information of the background area in the image is simple, while the texture and contour of the crowd area are relatively complex. Therefore, this invention utilizes the edge texture information of different objects in the image to filter the background area, which can ensure the accuracy of the division between the background area and the non-background area, thereby better balancing video compression effect and image quality of the crowd monitoring area.

[0157] Specifically, first, edge detection is performed on the video frame, and spatial blocks with edge detection intensity lower than an edge detection threshold are marked as background regions (such as most of the road), and spatial blocks higher than the threshold are marked as non-background regions (such as most of the pedestrian sidewalk area). In addition, in order to adapt to the limited processing capacity of the unmanned aerial vehicle node, the present application uses a lightweight edge detector, such as a Canny edge detector (from Gonzalez Rafael C. Digital image processing), to perform edge detection on a video frame with a size of HxW, and the time complexity is only O(HW·log(HW)), which is only comparable to the complexity of a single convolutional layer in a neural network. The running time of a 1280x800 resolution image on an actual small unmanned aerial vehicle edge processing device (such as Jetson Nano) is less than 3ms, so the edge node end computing overhead is extremely small and the processing delay is negligible.

[0158] S302: multiplexing the crowd detection result in step S2021, correcting the crowd region misjudged as a background region to a non-background region; and then taking all non-background regions as an encoding interest domain of the target video frame and taking the background region as a non-encoding interest domain of the target video frame;

[0159] In this embodiment, the crowd region misjudged as a background region in the background filtering is corrected using crowd position information and the like to obtain more accurate region division. Since the background filtering based on edge detection only relies on edge texture intensity to distinguish background regions, the position distribution of the crowd in the image is not considered, resulting in a problem that a region containing a crowd is incorrectly divided into a background region. Specifically, in order to obtain crowd position information in real time on a limited computing power unmanned aerial vehicle node device, the present application re-corrects the crowd region misjudged as a background region to a non-background region by multiplexing the crowd detection result of the time domain sampling part, which can improve the accuracy of the division of the background region and the non-background region, so that the video compression effect and the picture quality of the crowd monitoring region can be better balanced. Finally, all non-background regions are taken as an encoding interest domain, and the background region is taken as a non-encoding interest domain.

[0160] S303: dynamically calculating an encoding quality map according to the encoding interest domain division of the target video frame, that is, defining the encoding quality of each image block in the target video frame, wherein the encoding interest domain is set to lossless encoding quality, and the non-encoding interest domain is set to lossy encoding quality;

[0161] S304: according to the encoding quality map of each frame of the target video frame, the video encoder performs corresponding lossless encoding or lossy encoding on each image block, that is, lossless compression of the non-background region and lossy compression of the background region are realized.

[0162] In this embodiment, the video frame is dynamically quality coded by a video encoder according to the coding region of interest division of the video frame. Specifically, first, a coding quality map is dynamically calculated according to the coding region of interest division of the video frame, which defines the coding quality of each image block in the video frame. The coding region of interest in the video frame is set to lossless coding quality, and the non-coding region of interest is set to lossy coding quality. Then, according to the coding quality map of each frame, the video encoder performs corresponding lossless or lossy coding on each image block. Finally, lossless compression of the crowd monitoring area in the UAV video is realized, and lossy compression of the background area of the non-monitoring target is performed, which ensures that the video transmission volume is further reduced without reducing the crowd monitoring accuracy.

[0163] On the basis of removing redundant video frames to obtain target video frames, the non-background area (i.e. crowd monitoring target area) and the background area (such as street, sky, etc. non-crowd monitoring target area) of the target video frames are extracted in the spatial domain and are respectively lossless coded (lossless compression) and lossy coded (lossy compression) i.e. dynamic coding, so that the redundant coding data amount of the background part can be maximized to reduce under the condition that the picture quality of the crowd area in the video frame is not reduced, i.e. the picture quality of the crowd monitoring area can be better considered in the video compression effect, so as to further improve the effectiveness of the crowd monitoring video compression, and better meet the application requirements of large-scale urban crowd monitoring.

[0164] In summary, the specific process of video spatial sampling based on dynamic coding is as follows: input m×m color video frame I orig , the first to the fourth rows are initialized parameters, edge detection, image blocking and lightweight crowd detection, the fifth to the sixth rows are coding RoI division and video dynamic coding, and finally output the dynamically coded video frame I out and the coding RoI set

[0165] The process of video spatial sampling based on dynamic coding is as follows:

[0166] Input: color video frame I orig with pixel size m×m;

[0167] Output: spatially sampled video frame I out , coding RoI set

[0168]

[0169]

[0170] In order to better illustrate the advantages of the technical scheme of the present application, the following experiments are disclosed in this embodiment.

[0171] 1. Introduction of dataset and experimental method

[0172] 1) Experimental dataset and scenario setting

[0173] The UAV crowd surveillance dataset used in this experiment includes The Oxford Town Centre (OTC) (from

[0174] Harvey Adam, Laplace Jules. Megapixels: origins, ethics, and privacy implications of publicly available face recognition image datasets), Group-detection (GD) (from Solera Francesco, Calderara Simone, Cucchiara Rita. Socially constrained structural learning for groups detection in crowd), and self-labeled dense crowd dataset Multi-scenario Crowd dataset (MSCD) shot by UAV, and UAV vision challenge dataset VisDrone2019 (from Du Dawei, Zhu Pengfei, Wen Longyin, et al. VisDrone-DET2019: The vision meets drone object detection in image challenge results). The four datasets contain more than 20,000 video frames of crowd gathering shot by UAV, including crowd riots, large-scale rallies, urban streets, and other multi-scenarios, with a resolution of 1280x720 or above, matching the crowd detection task.

[0175] 2) Introduction of experimental method

[0176] To evaluate the performance of the edge processing system, the overall latency, detection accuracy, and recall rate of the system are evaluated. In the experimental evaluation, the wireless transmission distance between the UAV and the ground station is set to 30 m, and the influence of crowd scene density and image resolution variation on the processing performance (including system processing latency and detection accuracy) of the edge processing system is considered. Secondly, in order to simulate the scene switching of real UAV video, video frames of different scenes are mixed and sampled.

[0177] System actual experiment environment setting. In order to verify the feasibility of the system, Jetson Nano is used as the small edge processing device of the unmanned aerial vehicle, a notebook computer with GPU is used as the ground base station, wireless wifi is used for data communication, Pytorch and TenorRT are used for deep network inference, and OpenCV and FFmpeg are used for video processing.

[0178] Deep network model training. Yolov5 is used as the crowd detection deep network, and VisDrone2019 dataset is used for training, which contains more than 7000 pictures and their labels. The server used for training is equipped with NVIDIA 3090GPU and Ubuntu18 system, the initial training epoch is 250, the training batch size is 16, and the SGD optimizer is used.

[0179] Similar to most current edge computing system works, in order to comprehensively evaluate the performance of edge computing system, four typical comparison methods with different performance are used in this experiment: 1) Un-CVP (Uncompressed Video Processing): directly transmit the video file shot by the unmanned aerial vehicle to the ground end without compression processing; 2) CVP (Compressed Video Processing): all video frames collected by the unmanned aerial vehicle are compressed by lossy compression and then transmitted to the ground end; 3) CFRC (Clipped Fixed-ROI Compressing): based on the traditional fixed monitoring video processing method (from Huang Kaiqi, Chen Xiaotang, Kang Yunfeng, et al. Intelligent visual surveillance: a review), that is, first cut the video collected by the unmanned aerial vehicle using structural similarity coefficient, and then compress the image in the fixed ROI region; 4) REMIX (from Jiang Shiqi, Lin Zhiqi, Li Yuanchun, et al. Flexible high-resolution object detection on edge devices with tunable latency): based on adaptive image region division video image processing, that is, according to the different crowd density regions in the unmanned aerial vehicle video image, different compression coefficients are used for image compression.

[0180] In order to evaluate the performance of each module of the video space-time fusion low-loss compression algorithm of the invention, the following two comparison modules are used:

[0181] 1) TSO (Temporal Sampling Only): Temporal sampling algorithm, i.e. the video file collected by the UAV is processed according to the temporal sampling method of the application;

[0182] 2) SSO (Spatial Sampling Only): Spatial sampling algorithm, i.e. the video file collected by the UAV is processed according to the spatial sampling method of the application.

[0183] In the experimental evaluation, three performance evaluation indexes are used in the experiment: 1) Average Processing Time (APT), i.e. the sum of the average processing time of the video collected by the UAV at the edge of the UAV and the average transmission time of the compressed video transmitted to the ground end; 2) Precision (P), which refers to the proportion of relevant categories in the total returned results in the detection results, as shown in formula (7); 3) Recall (R), which refers to the proportion of relevant categories in the total relevant categories in the detection results, as shown in formula (8).

[0184]

[0185]

[0186] wherein, and respectively represent the correct detection result set, the detection result set and the Ground-truth set.

[0187] 2. Experimental results

[0188] 1) Algorithm performance evaluation

[0189] Firstly, the influence of different video resolutions on the performance of crowd detection and system processing time is evaluated.

[0190] As shown in Figure 4 (a), the video frame resolution is gradually reduced from 800p to 240p, and the experimental results show that the technical solution of the application can achieve a crowd detection precision of 98.6% and a recall rate of 95.7% for the Un-CVP method, and the detection precision of the REMIX algorithm, CFRC algorithm and CVP algorithm is reduced by an average of 2.2%, 17.2% and 23.2% respectively, and the recall rate is reduced by an average of 1.2%, 8.1% and 9.7% respectively, compared with the technical solution of the application. The main reason is that the CVP algorithm and the CFRC algorithm do not use crowd information for video spatial domain processing, resulting in lossy compression of the image crowd area, so the detection precision is greatly reduced, while the technical solution of the application and the REMIX algorithm consider the crowd information in video processing, so the crowd detection precision and recall rate are guaranteed. As shown in Figure 4As shown in (b), experimental results on system processing latency at different resolutions show that the technical solution of this invention reduces the average edge system processing latency by 31.3% and 33.25% compared to the Un-CVP method and the REMIX algorithm, respectively, while increasing the average processing latency by 8.7% and 29.1% compared to the CFRC and CVP algorithms. Meanwhile, as the video resolution gradually decreases, the processing latency reduction of the technical solution of this invention and the REMIX algorithm is relatively slow. This is because the technical solution of this invention and the REMIX algorithm use neural networks, which introduce a certain basic latency.

[0191] Next, we evaluated the impact of different crowd density scenarios on crowd detection performance and system processing latency.

[0192] like Figure 5 As shown in (a), the resolution was set to 800p for different crowd density scenarios. Experimental results show that, compared with the CVP and CFRC algorithms, the technical solution of this invention improves accuracy by an average of 23.6% and 32.6%, respectively, and recall by an average of 21.6% and 20.5%, respectively. Furthermore, the technical solution of this invention and the REMIX algorithm can achieve crowd detection accuracy and recall similar to the Un-CVP method. Meanwhile, as... Figure 5 As shown in (b), the system processing latency experiment results indicate that, in various crowd density scenarios, the technical solution of this invention reduces the average processing latency by 55.2% and 22.3% compared to the Un-CVP method and the REMIX algorithm, respectively, with average reductions of 77.6 ms and 18.0 ms. Compared to low-density scenarios, in medium-density and high-density scenarios, the system processing latency of the technical solution of this invention and the REMIX algorithm increases by an average of 5.1% and 4.6%, respectively. This is because increased crowd density leads to increased latency in processing crowd features; while the average changes in system processing latency for the Un-CVP method, CVP, and CFRC algorithm are all less than 1.0 ms.

[0193] 2) Analysis of the impact of algorithm modules and parameters

[0194] Ablation experiments were conducted to evaluate the effectiveness of each module in the video spatiotemporal fusion low-loss compression algorithm of this patent application. For example... Figure 6 As shown in (a), the video spatiotemporal compression algorithm, temporal sampling-only (TSO) algorithm, and spatial sampling-only (SSO) algorithm of this patent application achieve 96.7%, 98.6%, and 94.3% accuracy, respectively, and 96.7%, 96.6%, and 94.6% recall, respectively, compared to the Un-CVP method. Meanwhile, the system processing latency experimental results are as follows... Figure 6(b) shows, the average processing time delay of the spatio-temporal compression algorithm of the present patent application is reduced by 38.2% compared with the Un-CVP method; the average processing time delay of the TSO algorithm and the SSO algorithm is reduced by 34.3% and 33.5% respectively compared with the Un-CVP method at higher resolutions (800p and 480p), while the average processing time delay is increased by 36.3% and 47.2% respectively compared with the Un-CVP method at lower resolutions (320p and 240p). The main reason is that the TSO algorithm and the SSO algorithm both use neural networks, resulting in a basic processing time delay, so the time delay cannot be effectively reduced at low resolution. The spatio-temporal compression algorithm of the present patent application is subjected to time domain sampling and spatial sampling, so the basic processing time delay of the neural network is shared, and the average processing time delay is effectively reduced.

[0195] Finally, the influence of the change of the dynamic encoding image block division parameter bins on the crowd detection performance and processing time delay in the spatio-temporal compression algorithm of the present patent application is evaluated. As shown in Figure 7 (a) and (b), the number of image block divisions bins is increased from 5 to 25, and the experimental results show that as the image block division parameter bins increases, the detection accuracy and recall rate at each resolution are increased by an average of 2.6% and 2.1% respectively, and the processing time delay is increased by an average of 3.4ms. This shows that increasing the image block division parameter bins can improve the detection effect to a certain extent, but will increase the processing time delay.

[0196] 3. Prototype system deployment

[0197] In order to verify the practicality and feasibility of the unmanned aerial vehicle edge capability limited video spatio-temporal fusion low-loss compression algorithm proposed in the present patent application, an unmanned aerial vehicle edge intelligent computing prototype system for real-time monitoring of urban crowds is constructed. As shown in Figure 8 (a), a S500 unmanned aerial vehicle is used to carry a Jetson Nano small processing device as an edge node, a notebook computer with a GPU is used as a ground base station, and the unmanned aerial vehicle and the ground base station communicate in real time through wireless wifi.

[0198] The system performs the crowd monitoring task mainly in four steps: 1) the ground base station is deployed in a place where urban crowds gather, and the patrol area of the unmanned aerial vehicle is set; 2) the unmanned aerial vehicle moves to shoot the ground crowd gathering video, and at the same time the edge device of the unmanned aerial vehicle performs video spatio-temporal fusion low-loss compression according to the method proposed in the present patent application, and transmits the video to the ground end with low time delay; 3) the ground end receives the unmanned aerial vehicle video in real time and performs crowd monitoring applications such as crowd counting and social distance detection; 4) the ground base station uploads the crowd monitoring analysis results to the cloud server and performs visual display. As shown in Figure 8(b) as shown, the system can display real-time crowd monitoring video taken by the UAV, detection results, UAV flight trajectory and state, specific character recognition results, and crowd density information.

[0199] Finally, in order to verify the feasibility and accuracy of the system, crowd detection is performed on different monitoring scene video frames taken by the UAV. As shown in Figure 9 the first column to the fourth column of the scene are large-scale crowd gathering scenes of urban streets, election rallies, square riot, and beach marathons, respectively. The detection results show that in the crowd gathering scenes of urban streets and election rallies, the crowd detection performance of the image processed by the CVP algorithm is significantly reduced, while the crowd detection performance of the video frame processed by the compression algorithm of the present application is similar to the detection effect of the original video frame (Un-CVP) in all scenes.

[0200] 5. Summary

[0201] The present application proposes a UAV edge intelligent computing system for real-time urban crowd monitoring to solve the problems of high transmission delay of video and low precision of compressed video in UAV crowd monitoring with limited edge capability. Specifically, a video space-time fusion low-loss compression algorithm is proposed for UAVs with limited edge capability. By fusing video time domain sampling based on feature clustering and video space sampling based on dynamic coding, the UAV collected video is processed by low-loss compression to minimize the video transmission delay under the condition of a certain target detection information loss rate. Then, the compressed video is transmitted to the ground end for related crowd monitoring analysis, realizing real-time and accurate UAV crowd monitoring. Finally, based on the actual UAV, ground base station and cloud server, a UAV edge intelligent computing prototype system is constructed, and comprehensive and in-depth experimental evaluation is performed. The experimental results show that the proposed method can effectively reduce the transmission delay and meet the high precision requirement.

[0202] Embodiment Two:

[0203] A UAV crowd monitoring system is disclosed in this embodiment.

[0204] As shown in Figure 10 , the UAV crowd monitoring system comprises:

[0205] The UAV end is used for collecting crowd monitoring video of a target area.

[0206] In this embodiment, the UAV end is composed of a UAV and a video collection device, mainly for mobile monitoring and video data collection of the gathered crowd.

[0207] An unmanned aerial vehicle edge processing device for spatio-temporal fusion compression of crowd monitoring videos collected by the unmanned aerial vehicle end by implementing the video spatio-temporal fusion low-loss compression method described in Embodiment I, to generate corresponding spatio-temporal compressed crowd videos;

[0208] In this embodiment, the unmanned aerial vehicle edge processing device is a small edge processing device carried on the unmanned aerial vehicle, such as Jetson Nano (from Song Qingyu, Wang Changan, Wang Yabiao, et al. To choose or to fuse?scale selection for crowd counting and Nvidia Corporation. NVIDIA Jetson Nano).

[0209] A ground control end for crowd monitoring applications based on spatio-temporally compressed crowd videos after spatio-temporal fusion compression.

[0210] In this embodiment, the ground control end is composed of a ground control base station, which is responsible for unmanned aerial vehicle flight trajectory control, receives and utilizes the compressed videos from the unmanned aerial vehicle node end for related crowd monitoring applications such as crowd detection, and finally transmits the crowd analysis results to the server for data storage and aggregation.

[0211] The unmanned aerial vehicle crowd monitoring system performs crowd monitoring tasks mainly including the following steps: first, the ground control end is deployed in places where crowds gather, such as commercial plazas, and the unmanned aerial vehicle patrol trajectory is set; second, the unmanned aerial vehicle moves to shoot ground crowd gathering videos, at the same time, the unmanned aerial vehicle end edge device processes the unmanned aerial vehicle videos according to the video spatio-temporal fusion compression method of the present application, and transmits them to the ground end with low latency; finally, the ground end receives the unmanned aerial vehicle videos in real time for related crowd monitoring applications, and uploads the crowd analysis results to the cloud server for visual display.

[0212] The crowd monitoring applications of the ground control end include crowd detection and counting, crowd target tracking, and crowd social distance monitoring.

[0213] 1) Crowd detection and counting. Compared with the lightweight target detection of the unmanned aerial vehicle node end, the crowd detection and counting of the ground control end mainly utilizes large deep neural networks such as Yolov5, SASNet (from Song Qingyu, Wang Changan, Wang Yabiao, et al. To choose or to fuse?scale selection for crowd counting), etc., which have higher accuracy for large-scale crowd detection.

[0214] 2) Crowd target tracking. Using deep learning-based target tracking algorithms such as DeepSORT (from Wojke Nicolai, Bewley Alex, Paulus Dietrich. Simple online and realtime tracking with a deep association metric) and the like, effective multi-crowd target tracking detection can be performed in a crowded scene.

[0215] 3) Crowd social distance monitoring. The UAV edge computing system can monitor the crowd social distance using the ShuffleNet network (from Rezaee Khosro, Mousavirad Seyed Jalaleddin, Khosravi Mohammad R, et al. An Autonomous UAV-Assisted Distance-Aware Crowd Sensing Platform Using Deep ShuffleNet Transfer Learning) to prevent the spread of the epidemic.

[0216] In addition, the ground control terminal transmits the crowd monitoring analysis results, such as crowd detection quantity, proposal box, confidence, tracking trajectory, etc., back to the UAV node terminal through wireless communication, so that the UAV node terminal uses the detection results of the ground terminal to perform more accurate video space-time compression.

[0217] The present application generates corresponding space-time compressed crowd video by spatiotemporal fusion compression of the crowd monitoring video collected by the UAV terminal through the UAV edge processing device, that is, by fusing feature clustering-based video time domain sampling and dynamic coding-based video space sampling to perform low-loss compression processing on the UAV collected video. Not only can the video transmission delay be minimized under the condition of ensuring a certain target detection information loss rate, but also the limitation of UAV edge computing capability can be met, that is, the video transmission delay can be greatly reduced under the condition of ensuring the accuracy of crowd detection and recognition, thereby further improving the effectiveness of crowd monitoring video compression and better meeting the application requirements of actual large-scale urban crowd monitoring. Experimental results show that the technical scheme of the present application can reduce the transmission delay by 31.3%, but the crowd counting accuracy decreases by no more than 2%.

[0218] At the same time, the compressed video is transmitted to the ground terminal with low latency for related crowd monitoring analysis, so that real-time and accurate UAV crowd monitoring can be realized.

[0219] Finally, it needs to be explained that the above examples are only used to illustrate the technical solutions of the present application but not to limit the technical solutions, and those of ordinary skill in the art should understand that the technical solutions of the present application are modified or equivalently replaced without departing from the purpose and scope of the technical solutions, which should be covered in the scope of claims of the present application.

Claims

1. A low-loss video compression method for drone-based crowd monitoring, characterized in that, include: S1: Obtain the crowd monitoring video to be compressed; S2: In the time domain, redundant video frames are cut off based on the inter-frame feature similarity of the crowd monitoring video to generate several target video frames after removing the redundant video frames. In step S2, redundant video frames are cut off through the following steps: S201: Convert crowd monitoring video into corresponding crowd video frames; S202: Extracting crowd features from video frames of crowds using a lightweight network; In step S202, the image crowd features of the crowd video frame are extracted through the following steps: S2021: Using a lightweight network trained on the k-th frame of a crowd video frame I k Perform crowd detection and generate corresponding crowd detection results. The crowd detection results include the number of crowd detection proposal boxes in the current crowd video frame, as well as the center coordinates and confidence level c of each crowd detection proposal box. The number of crowd N is represented by the number of crowd detection proposal boxes. S2022: Using Gaussian kernel to check the k-th frame of the crowd video frame I k The crowd detection results are subjected to two-dimensional convolution to generate the corresponding crowd density map m. k ; The formula is described as follows: Where: m k Represents the k-th frame of the crowd video frame I. k Crowd density map; x and Represents a two-dimensional Gaussian kernel; Represents the impulse function; Represents the k-th frame of the crowd video frame I. k The confidence level of the i-th population detection proposal box; Represents the k-th frame of the crowd video frame I. k The center coordinates of the i-th crowd detection proposal box; N represents the number of crowd detection proposal boxes detected in the k-th video frame; S2023: The k-th frame of the crowd video frame I k Population density map m k Uniform pooling is performed to obtain the crowd density map P after image size compression. k This leads to the creation of the population density map P. k As the k-th frame of the crowd video frame I k Image crowd characteristics; S203: Extract the image frequency domain information and image structure information of the crowd video frame as the corresponding image scene features; S204: Use the features of the crowd and the scene in the image as the total features of the crowd video frame, and perform feature clustering and redundant video frame cutting on the crowd video frame based on the feature similarity of the total image features. S3: In the spatial domain, extract the non-background and background regions of each target video frame and perform dynamic encoding to generate several dynamically encoded video frames. In step S3, dynamic encoding is performed through the following steps: S301: Perform edge detection on the target video frame and mark spatial blocks with edge detection intensity below the edge detection threshold as background regions, and spatial blocks with edge detection intensity above the threshold as non-background regions; S302: Reuse the crowd detection results in step S2021 to correct the crowd areas that were misjudged as background areas to non-background areas; then use all non-background areas as the encoded interest domains of the target video frame and the background areas as the non-coded interest domains of the target video frame. S303: Dynamically calculate the coding quality map based on the coding interest region of the target video frame, that is, define the coding quality of each image block in the target video frame, where the coding interest region is set to lossless coding quality and the non-coding interest region is set to lossy coding quality; S304: Based on the coding quality map of each target video frame, the video encoder performs corresponding lossless or lossy coding on each image block, that is, to achieve lossless compression of non-background areas and lossy compression of background areas. S4: A spatiotemporally compressed video of crowd monitoring composed of several dynamically encoded video frames.

2. The low-loss video compression method for drone-based crowd monitoring as described in claim 1, characterized in that, In step S203, the image scene features of the crowd video frames are extracted through the following steps: S2031: For the k-th frame of the crowd video frame I k Perform a two-dimensional discrete cosine transform while preserving the low-frequency spectrum; The formula is described as follows: Among them, coefficient In the formula: N1 represents the two-dimensional cosine discrete transform value at (k1, k2) in the frequency domain; N1 and N2 represent the height and width of the crowd video frame, respectively; n1 and n2 represent the indicator variables in the summation formula; k1 and k2 represent the coordinates in the two-dimensional frequency domain. S2032: Quantize the low-frequency spectrum, that is, quantize each low-frequency cosine discrete transform value to 0 or 1 based on whether the low-frequency cosine discrete transform value of the low-frequency spectrum is higher than its low-frequency cosine discrete transform mean, and use the quantized low-frequency spectrum as the k-th crowd video frame I. k Frequency domain characteristics F k ; S2033: Extract the k-th frame of the crowd video frame I k The image structure information is used to generate corresponding image structure features; S2034: Transform the k-th frame of the crowd video frame I k Frequency domain characteristics F k Combined with image structural features, it serves as the k-th frame of the crowd video frame I. k Image scene features S k .

3. The low-loss video compression method for drone-based crowd monitoring as described in claim 2, characterized in that, In step S2033, the image structural features include brightness features L. k Contrast characteristics Y k and color distribution characteristics 1) I of the k-th frame of the crowd video k The area is uniformly divided into rectangular blocks, and the local brightness μ of each block is calculated using the following formula. X Local contrast σ X ; In the formula: H and M represent the length and width of the block, respectively; X(i,j) represents the gray value of the pixel at position (i,j) in the image block; The k-th frame of the crowd video frame I k The local brightness and contrast values ​​of all blocks are concatenated to form the corresponding brightness feature L. k and contrast feature Y k ; 2) For the k-th frame of the crowd video frame I k The R, G, and B channels are used to calculate the pixel distribution ratio in different grayscale ranges. As the corresponding color distribution characteristics 4. The low-loss video compression method for drone-based crowd monitoring as described in claim 1, characterized in that, In step S204, redundant frames are cut off through the following steps: S2041: Use a sliding window to cover continuous crowd video frames, and use the cosine similarity of the total image features of all crowd video frames within its coverage area as the inter-frame feature similarity according to the following formula. In the formula: cos(θ) represents the cosine similarity; X i Y i This represents the components of the total image features of two different crowd video frames in the sliding window; n represents the dimension of the total image features of the crowd video frames. S2042: Based on the inter-frame feature similarity of crowd video frames within the sliding window, a clustering algorithm is used to cluster the crowd video frames, and a cutting judgment is made for each class of video frames in the clustering results based on the cutting threshold. S2043: Move the sliding window by the set step size and continue to perform feature clustering and clipping judgment until the sliding window has traversed all the video frames of the crowd.

5. A drone-based crowd monitoring system, characterized in that, include: The drone is used to collect surveillance video of crowds in the target area. The drone edge processing device is used to perform spatiotemporal fusion compression on the crowd monitoring video collected by the drone using the low-loss video compression method for drone crowd monitoring as described in claim 1, to generate a corresponding spatiotemporally compressed crowd video. The ground control terminal is used for crowd monitoring applications based on spatiotemporally compressed crowd videos obtained through spatiotemporal fusion compression.

6. The drone crowd monitoring system as described in claim 5, characterized in that: Ground-based crowd monitoring applications include crowd detection and counting, crowd target tracking, and crowd social distancing monitoring.

Images