A method and system for assessing crowd density in video from a lightweight drone

Abstract

This invention discloses a method and system for assessing crowd density in video from a lightweight drone in the field of image processing technology. The method includes: fusing spatiotemporal information from the video and constructing a deep network structure for crowd density regression; offline training on key frames of the video data to obtain a neural network parameter model for the crowd density heatmap; and online inference on all frames of the video data using the neural network parameter model, counting the number of pixels in the crowd density heatmap of each frame as the number of people to be assessed. This invention assesses crowd density by fusing spatiotemporal information from drone video and adopts a multi-path network architecture to extract temporal information from multiple consecutive frames and spatial information from a single frame, respectively. The two types of information are then fused spatiotemporally using tensor-based fusion to obtain the final crowd density heatmap, which features high estimation accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for assessing crowd density in video from a lightweight drone. Background Technology

[0002] Currently, public safety issues in densely populated areas are becoming increasingly prominent. To ensure public safety, it is crucial to conduct headcount statistics in public places, especially densely populated areas. Lightweight drones are becoming increasingly widely used due to their advantages such as low cost, maneuverability, minimal terrain limitations, and wide observation range.

[0003] Existing surveillance cameras based on fixed locations have limited fields of view, while drone platforms can achieve real-time observation of large-scale scenes in key areas of interest. However, images obtained from drone-based ground observations differ from close-up natural photographs. For targets such as people and vehicles, only top-view images are available, lacking detailed images of the entire scene, and the scale is relatively small. This is particularly problematic in densely populated areas, where detecting and counting individual people results in low accuracy in headcount. With the successful application of deep learning in image processing and analysis, deep learning algorithms are increasingly being incorporated into crowd density statistics.

[0004] However, in the process of implementing the inventive technical solution in the embodiments of this application, the inventors of this application discovered that the above-mentioned technology has at least the following technical problems:

[0005] 1. Video surveillance locations are relatively fixed, making it difficult to achieve mobility, flexibility, and adjustment of the field of view during monitoring;

[0006] 2. The method mainly uses image processing to count the number of individuals in a crowd. However, it is greatly affected by the interference of the environment and the target, resulting in low estimation accuracy and poor robustness.

[0007] 3. Based on images taken by drones, the targets are generally small in scale, with only local information such as the top of the target.

[0008] Based on this, the present invention designs a method and system for assessing crowd density in video from a lightweight drone to solve the above problems. Summary of the Invention

[0009] In order to solve the technical problems mentioned in the background art, the purpose of this invention is to provide a method and system for assessing crowd density in video from a lightweight drone.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A method for assessing crowd density in video footage from a lightweight, small unmanned aerial vehicle (UAV) includes:

[0012] Integrate video spatiotemporal information and construct a deep network structure for crowd density regression;

[0013] Offline training was performed on keyframes of video data to obtain a neural network parameter model for crowd density heatmaps;

[0014] Online inference is performed on all frames of video data using a neural network parameter model, and the number of pixels in the crowd density heatmap of each frame is counted as the number of people to be evaluated.

[0015] Furthermore, the fusion of the video spatiotemporal information includes:

[0016] A branch network structure for extracting temporal motion information from multiple consecutive frames of video images using a 3D convolutional network;

[0017] A branch network structure that extracts spatial information of the current frame image using a two-dimensional convolutional network;

[0018] The two branch network structures described above are fused into a network structure through spatiotemporal feature channels.

[0019] Furthermore, the offline training of keyframes in the video data includes:

[0020] Keyframes are extracted from video data and manually annotated, wherein the manual annotation is used to record the center position of the human head in the image;

[0021] Based on manually labeled head center locations, a crowd density heatmap is generated as the training ground truth.

[0022] The labeled training ground data is used to train the network end-to-end to learn the neural network parameter model.

[0023] Furthermore, the online inference of all frames of the video data includes:

[0024] The video is input frame by frame or captured frames into a neural network model to predict crowd density heatmaps.

[0025] The number of pixels in the predicted image is counted, and the number of pixels that meet the predetermined requirements is recorded as the parameter for the number of people in the image.

[0026] Furthermore, the process of recording the required number of pixels includes:

[0027] Determine if the number of pixels in the image exceeds a threshold, and if it does, control the number of pixels to be used as the number of people in the image.

[0028] A lightweight drone video crowd density assessment system includes:

[0029] The spatiotemporal information fusion module is used to fuse the spatiotemporal information of the currently acquired video.

[0030] The network structure building module is used to construct a deep network structure for population density regression based on the fused spatiotemporal information.

[0031] The offline training module is used to train keyframes of video data to obtain a neural network parameter model for crowd density heatmaps.

[0032] The online inference module is used to infer the number of people in each frame of the video data through a neural network parameter model and count the number of pixels in the crowd density heatmap of each frame.

[0033] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0034] By fusing spatiotemporal information from drone video, crowd density assessment is performed. A multi-path network architecture is adopted to extract temporal information from multiple consecutive frames of images and spatial information from a single frame of image. The two types of information are then fused spatiotemporally using tensor-based fusion to obtain the final crowd density heatmap. The method features high estimation accuracy and robustness. Attached Figure Description

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0036] Figure 1 This is a flowchart of the evaluation method according to an embodiment of the present invention;

[0037] Figure 2 To evaluate the network architecture diagram for embodiments of the present invention;

[0038] Figure 3 This is a diagram of the spatiotemporal channel fusion network structure according to an embodiment of the present invention;

[0039] Figure 4 This is a system block diagram of the evaluation system according to an embodiment of the present invention. Detailed Implementation

[0040] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0041] Please see Figures 1 to 3 This invention provides a technical solution: a method for assessing crowd density in video footage from a lightweight drone, comprising:

[0042] S102. Integrate video spatiotemporal information and construct a deep network structure for crowd density regression;

[0043] S104. Offline training is performed on keyframes of video data to obtain the neural network parameter model of the crowd density heatmap;

[0044] S106. Perform online inference on all frames of the video data using a neural network parameter model, and count the number of pixels in the crowd density heatmap of each frame as the number of people to be evaluated.

[0045] Through the above steps, it can be seen that in the process of assessing crowd density in drone videos, a deep network structure for crowd density regression is built by fusing video spatiotemporal information, namely, extracting the temporal network branch of multi-frame temporal information, the spatial network branch of single-frame image information, and the spatiotemporal feature channel fusion branch structure. Offline training is then performed on keyframes in the video data. Specifically, the center position of the human head in the image is marked and recorded to generate the corresponding crowd density heatmap training ground value. The neural network is then learned based on the training data to build a neural network model. Finally, the built neural network model is used for online inference on all frames of the video to obtain the number of people assessed based on the number of pixels in the crowd density heatmap of each frame.

[0046] To better achieve the fusion of video spatiotemporal information, the fusion of video spatiotemporal information includes:

[0047] A branch network structure for extracting temporal motion information from multiple consecutive frames of video images using a 3D convolutional network;

[0048] A branch network structure that extracts spatial information of the current frame image using a two-dimensional convolutional network;

[0049] The two branch network structures described above are fused into a network structure using spatiotemporal feature channels; the calculation formula for the spatiotemporal feature fusion network structure is: Y = ασ(TT) T )F+σ(SS T )S;

[0050] Where Y∈R (hw)×c This is obtained by reshaping the fused feature map matrix of size h×w×c. Similarly, the feature map matrix T∈R is extracted by the temporal branch network. (hw)×c The feature map matrix extracted by the spatial information extraction branch network is S∈R (hw)×c σ(X) represents the softmax function applied to each row of matrix X, and α is the learnable weight coefficient. Finally, Y is reshaped to h×w×c to obtain the final feature map;

[0051] This deep network structure for crowd density regression achieves spatiotemporal information fusion by extracting temporal information from multiple consecutive frames of images and extracting current image spatial information from a single frame of image, ultimately resulting in a crowd density heatmap.

[0052] During offline training of video data, the offline training of keyframes in the video data includes:

[0053] Keyframes are extracted from video data and manually annotated, wherein the manual annotation is used to record the center position of the human head in the image;

[0054] Based on manually labeled head center locations, a crowd density heatmap is generated as the training ground truth.

[0055] The labeled ground truth training data is used for end-to-end training of the network to learn the neural network parameter model. Specifically, by collecting drone video data covering the application scenario, keyframes are extracted from the videos, and these keyframes are manually labeled. The center position of a person's head in the image is marked and recorded. Based on the center position of the person's head in the image, a crowd density heatmap is generated as the ground truth training data. The calculation formula is as follows:

[0056] Where y(x) represents the generated heatmap, and x is the pixel position of the heatmap. i Let be the center position of the head of the i-th person, and δ(·) be the Δ function. Indicates a kernel width of σ i The Gaussian kernel function, where * denotes convolution;

[0057] The labeled training data is then used for end-to-end training of the network to learn the neural network parameters. Network training employs a loss function, calculated using the following formula:

[0058] in, This represents a heatmap of population density predicted by the network, y i The graph is a heatmap generated by manual labeling, where θ represents the network parameters.

[0059] In the process of online inference on video data, the online inference of all frames of the video data includes:

[0060] The video is input frame by frame or captured frames into a neural network model to predict crowd density heatmaps.

[0061] The number of pixels in the predicted image is counted, and the number of pixels that meet the predetermined requirements is recorded as the number of people in the image. That is, by inputting the drone video frame by frame or captured frame into the neural network parameter model, the crowd density heat map is predicted, and the number of people is further confirmed based on the number of pixels in the heat map, thereby realizing the assessment of the number of people.

[0062] To better assess the number of people, the process of recording the required number of pixels includes:

[0063] Determine whether the number of pixels in the image exceeds a threshold, and if it does, control the number of pixels to be used as the number of people in the image; that is, determine whether the number of pixels contained in the heat map exceeds the set threshold γ. If it exceeds the threshold γ, directly use the number of pixels as the number of people in the image.

[0064] Please see Figure 4 The present invention also provides another technical solution: a lightweight drone video crowd density assessment system, comprising:

[0065] The spatiotemporal information fusion module is used to fuse the spatiotemporal information of the currently acquired video.

[0066] The network structure building module is used to construct a deep network structure for population density regression based on the fused spatiotemporal information.

[0067] The offline training module is used to train keyframes of video data to obtain a neural network parameter model for crowd density heatmaps.

[0068] The online inference module is used to infer the number of people in each frame of the video data through a neural network parameter model and count the number of pixels in the crowd density heatmap of each frame.

[0069] As can be seen from the above, by first fusing spatiotemporal information, constructing a deep network structure for crowd density regression, and then training keyframes in video data offline to obtain a neural network parameter model for crowd density heatmaps, and then performing online inference on all frames of video data based on this neural network parameter model to obtain an assessment of the number of people in the heatmap based on the number of pixels, the assessment of crowd density in videos from small and lightweight drones has been achieved.

[0070] It should be added that the specific procedures for conducting video crowd density assessment are as follows:

[0071] 1. Create the training dataset. Collect drone video images from application scenarios, covering as many different lighting conditions, crowd density, and sparse distribution as possible. Extract keyframes for annotation. Data annotation uses annotation software to manually mark the position of the center point of the human head and records it in an XML file. Divide the training set into training, validation, and test sets;

[0072] 2. Model Training. Construct a crowd density estimation network. The temporal branch network can use 3-4 3D convolutional blocks (3DConv-[BN]-ReLU-Pooling, where BN represents Batch Normalization layer), and the spatial branch network can use network structures such as VGG and ResNet. The specific convolutional module selection should ensure that both branch networks obtain feature maps of the same size. The fusion network uses the fusion structure designed in this invention. Finally, a 1×1 algorithm is used to obtain the predicted crowd density heatmap. Model initialization is performed. The weights of the 3D temporal branch network can use common random initialization, while the spatial branch network for a single image can use a pre-trained image classification model. Then, optimizers such as Adam are used to train the neural network model until convergence.

[0073] 3. Model Testing. On the test set, perform performance testing on the model trained in the previous step to verify whether the accuracy meets the application requirements. If it does not meet the requirements, it is necessary to adjust the network hyperparameters or add richer data, and then retrain to ensure it meets the application requirements.

[0074] 4. Model Deployment. Deploy the trained model in the cloud to provide online inference services;

[0075] 5. For video transmitted back by drones, the deployed model is used to estimate the crowd density in the current frame based on key frames, and the data can be displayed on the front end.

[0076] It should be understood that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are merely for illustrative purposes to aid those skilled in the art and are not intended to limit the scope of the invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and are not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0077] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for assessing crowd density in video footage from a lightweight, small unmanned aerial vehicle (UAV), characterized in that, include: Integrate video spatiotemporal information and construct a deep network structure for crowd density regression; Offline training was performed on keyframes of video data to obtain a neural network parameter model for crowd density heatmaps; Online inference is performed on all frames of video data using a neural network parameter model, and the number of pixels in the crowd density heatmap of each frame is counted as the number of people to be evaluated. The fusion of video spatiotemporal information includes: a branch network structure for extracting temporal motion information from multiple consecutive frames of video images using a three-dimensional convolutional network; a branch network structure for extracting image spatial information from the current frame image using a two-dimensional convolutional network; and a tensor-based spatiotemporal feature channel fusion network structure for fusing the two branch network structures. The calculation formula for the spatiotemporal feature channel fusion network structure is as follows: ; The feature map matrix is ​​obtained by reshaping the fused feature map matrix of size h×w×c. The feature map matrix extracted from the branch network structure of temporal motion information is... The feature map matrix extracted from the branch network structure of image spatial information is , This represents the softmax function, applied to each row of matrix X. These are learnable weight coefficients; The offline training of keyframes in the video data includes: extracting keyframes from the video data and manually annotating them, wherein the manual annotation is used to record the center position of a person's head in the image; generating a crowd density heatmap as the training ground value based on the manually annotated head center position point; and training the network end-to-end with the labeled training ground value data to learn the neural network parameter model.

2. The method for assessing crowd density in video footage from a lightweight unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The online inference of all frames of the video data includes: The video is input frame by frame or captured frames into a neural network model to predict crowd density heatmaps. The number of pixels in the predicted image is counted, and the number of pixels that meet the predetermined requirements is recorded as the number of people in the image.

3. The method for assessing crowd density in video footage from a lightweight unmanned aerial vehicle (UAV) according to claim 2, characterized in that, The process of collecting the required number of pixels includes: Determine if the number of pixels in the image exceeds a threshold, and if it does, control the number of pixels to be used as the number of people in the image.

4. A lightweight unmanned aerial vehicle (UAV) video crowd density assessment system, characterized in that, include: A spatiotemporal information fusion module is used to fuse the spatiotemporal information of the currently acquired video. The fusion process includes: a branch network structure for extracting temporal motion information from multiple consecutive frames of the video using a 3D convolutional network; a branch network structure for extracting image spatial information from the current frame image using a 2D convolutional network; and fusing the two branch network structures through a tensor-based spatiotemporal feature channel fusion network structure. The calculation formula for the spatiotemporal feature channel fusion network structure is as follows: ; This is obtained by reshaping the fused feature map matrix of size h×w×c. The feature map matrix is ​​extracted from the branch network structure of the temporal motion information. The feature map matrix extracted from the branch network structure of image spatial information is , This represents the softmax function, applied to each row of matrix X. These are learnable weight coefficients; The network structure building module is used to construct a deep network structure for population density regression based on the fused spatiotemporal information. An offline training module is used to train keyframes of video data to obtain a neural network parameter model for a crowd density heatmap. The offline training of keyframes includes: extracting keyframes from the video data and manually annotating them, wherein the manual annotation is used to record the center position of a person's head in the image; generating a crowd density heatmap as training ground values ​​based on the manually annotated head center position points; and performing end-to-end training of the network with the labeled training ground value data to learn the neural network parameter model. The online inference module is used to infer the number of people in each frame of the video data through a neural network parameter model and count the number of pixels in the crowd density heatmap of each frame.

Images