Outdoor crowd information extraction method based on unmanned aerial vehicle video

Abstract

The application provides an outdoor people flow information extraction method based on a UAV video, and has the following characteristics: step S1, frame extraction is performed on the video to obtain image frames; step S2, the image frames are input into a convolution network to obtain a plurality of boundary boxes; step S3, interest points are marked based on Shi-Tomasi features; step S4, personnel motion vectors and background motion vectors are obtained according to a KLT optical flow algorithm; step S5, personnel motion parameters are obtained from the personnel motion vectors; step S6, different people flows are obtained by clustering the personnel motion parameters; step S7, personnel moving speeds are obtained according to conversion rates, personnel motion vectors and background motion vectors; and step S8, people flow, people flow average speed and people flow density are obtained according to the boundary boxes, the personnel moving speeds and the conversion rates. In summary, the method can accurately identify personnel targets in the UAV video, identify people flows in different directions and extract people flow information.

Description

TECHNICAL FIELD

[0001] The present application relates to an outdoor crowd information extraction method, in particular to an outdoor crowd information extraction method based on UAV video. BACKGROUND

[0002] At present, unmanned aerial vehicles (UAVs) have been successfully applied in many fields such as military and aerial photography, and have the advantages of continuously improving shooting performance and continuously improving endurance.

[0003] Due to the limitations of traditional monitoring devices such as fixed cameras, UAVs have high application value in outdoor personnel identification and crowd flow information collection. Compared with traditional crowd monitoring methods, UAV video monitoring methods have many advantages. Traditional crowd monitoring devices, such as ground cameras, infrared probes, and wireless probes, are usually set in fixed locations and monitor fixed local ranges, so that large-scale outdoor crowd monitoring requires large-scale deployment of such traditional monitoring devices, and the maintenance of traditional monitoring devices is far less convenient than UAVs. However, a single UAV can monitor a larger outdoor area by cruising or hovering at a high altitude, with a monitoring range that is much larger than traditional crowd monitoring methods and easy to maintain without occupying road space during maintenance. In addition, the UAV monitoring camera is at a top-down angle, with no occlusion or overlap problems, and the corresponding target recognition, pixel distance and real distance conversion algorithms also have better effects.

[0004] However, when the UAV is in flight, its motion process has uncertain drift phenomena due to factors such as strong wind, obstacle avoidance, and vibration, which causes the background of the UAV video (other image areas except personnel) to be constantly moving, and the trajectory to have jitter. In addition, during the recording process of the UAV video, due to changes in the flight height and attitude of the UAV, the detected target has scale transformation and plane rotation, making it difficult to accurately identify the detected target. Therefore, existing technologies have not yet achieved accurate identification of personnel targets, identification of different direction crowds, and extraction of parameters such as density and speed of the UAV in the process of uncertain movement. SUMMARY

[0005] The present application is made to solve the above problems, and aims to provide an outdoor crowd information extraction method based on UAV video.

[0006] The present invention provides an outdoor crowd flow information extraction method based on drone video, which is used to obtain the outdoor crowd flow volume, average crowd flow speed and crowd flow density based on the outdoor crowd flow monitoring video taken by the drone. The method has the following characteristics: step S1, according to the frame rate of the outdoor crowd flow monitoring video, extracting frames of the outdoor crowd flow monitoring video at the same time interval to obtain multiple image frames; step S2, inputting the image frames into a convolutional network algorithm model, separating the image frames from the background through the bounding box, placing each person inside the bounding box, and placing the background outside the bounding box; step S3, based on the Shi-Tomasi feature, marking the interest points of the image frame, and taking the interest points inside the bounding box as the person interest points, and taking the interest points outside the bounding box as the background interest points; step S4, tracking the current image frame f according to the KLT optical flow algorithm t The interest point in the image frame and the next adjacent time frame f t+1 The corresponding interest point in the image is calculated to obtain the motion vector of the interest point. t The motion vectors of all points of interest in the current image frame f are calculated t The motion vectors of the personnel and the background motion vectors of the background in the image frame; Step S5, the current image frame f t All personnel motion vectors are put into the Cartesian rectangular coordinate system, and the motion parameters l of each personnel motion vector are calculated. k and θ k As the personnel motion parameter of the personnel motion vector; Step S6, the current image frame f t All personnel motion parameters are put into the 2D displacement space, and clustered by clustering algorithm. Each cluster represents the current image frame f t Different crowd flow directions in the current image frame f t The conversion rate r t , personnel motion vector and background motion vector, calculate the current image frame f t The moving speed of the people and background in the real world; Step S8, according to the current image frame f t The number of bounding boxes, the speed of movement of people in the real world, and the conversion rate r t , calculate the current image frame f t The flow rate, average speed and density of each flow direction, among which the conversion rate r t According to the current image frame f t The pixel size of the landmark reference in l p and the corresponding real-world size l a Get; or according to the current image frame f t The drone's altitude h t , the image frame f at the previous adjacent moment t-1The drone's altitude h t-1 and image frame f t-1 The conversion rate r t-1 Get; or according to the current image frame f t The average length of all bounding boxes in ω t , image frame f t-1 The average length of all bounding boxes in ω t-1 and image frame f t-1 The conversion rate r t-1 get.

[0007] The outdoor crowd flow information extraction method based on drone video provided by the present invention may also have the following features: wherein, in step S2, the convolutional network algorithm model is Fast R-CNN or MASKR-CNN or Haar cascade + custom convolutional network algorithm model or YOLOv5 convolutional network algorithm model or SSD convolutional network algorithm model.

[0008] The method for extracting outdoor crowd flow information based on drone video provided by the present invention may also have the following features: wherein, when the convolutional network algorithm model is FastR-CNN, step S2 includes the following sub-steps: step S2-1, generating multiple candidate regions and corresponding classification scores of the image frame according to the RPN module of FastR-CNN; step S2-2, merging candidate regions with more pixel intersections through a non-maximum suppression algorithm; step S2-3, sorting all the merged candidate regions from high to low according to the classification scores, and retaining the top N candidate regions as the focus regions; step S2-4, inputting the focus region into the RoI pooling layer of FastR-CNN, and outputting them through the softmax layer and the bounding box regression layer to obtain the position and size of each bounding box, and marking each bounding box in the image frame.

[0009] The method for extracting outdoor crowd flow information based on drone video provided by the present invention may also have the following features: wherein step S3 includes the following sub-steps: step S3-1, converting the image with a size of (2ω x +1)×(2ω y +1) slides the detection window ω over the entire image I of the image frame, and calculates the pixel u in the detection window ω = (p x ,p y ) of the second-order matrix G; Step S3-2, the second-order matrix G is a non-singular matrix with the minimum eigenvalue λ min Greater than the threshold λ th The corresponding pixel point is used as the comparison point, and the largest minimum eigenvalue λ in the 3×3 neighborhood is used as the comparison point. min The corresponding pixel points are marked as interest points; Step S3-3, all interest points are marked according to the minimum eigenvalue λ minRanking from high to low, retaining the first n interest points, taking the interest points in the bounding box as personnel interest points, and taking the interest points outside the bounding box as background interest points.

[0010] In the outdoor crowd information extraction method based on UAV video provided by the application, further features can include that in step S3-1, the formula of the second-order matrix G is as follows: In the formula, I x is the corresponding pixel value, p x is the corresponding pixel value. y is the corresponding pixel value. y is the corresponding pixel value.

[0011] In the outdoor crowd information extraction method based on UAV video provided by the application, further features can include that in step S4, the formula for calculating the personnel motion vector and the background motion vector of the current image frame f t is as follows: In the formula, I is the background motion vector, N b is the total number of background interest points in the current image frame f t , I is the motion vector of the jth background interest point, I is the personnel motion vector of the kth personnel, N t is the total number of personnel interest points marked for the kth personnel, I is the motion vector of the ith personnel interest point of the kth personnel.

[0012] In the outdoor crowd information extraction method based on UAV video provided by the application, further features can include that in step S5, the formula for calculating the motion parameters l k and θ k of the kth personnel is as follows: θ k = tan -1 (s y / s x ), in which s x and s y are the horizontal component and the vertical component of the personnel motion vector of the kth personnel in the Cartesian rectangular coordinate system.

[0013] In the outdoor crowd information extraction method based on UAV video provided by the application, further features can include that in step S7, the formula for calculating the moving speed s of the personnel or background in the real world of the current image frame f t is as follows: In the formula, l is the current image frame f t ​The absolute value of the personnel motion vector or the background motion vector of the personnel in the current image frame f f is the processing time interval of the frame extraction process, t p is the processing time of the current image frame f t , n is the total number of image frames before the current image frame f t , t pi is the processing time of the i-th image frame before the current image frame f t , and a is a decay coefficient, which is a non-negative number.

[0014] In the outdoor crowd information extraction method based on the UAV video provided by the application, the following features can also be provided: in step S8, the calculation formula of the crowd density d t of the i-th crowd direction in the current image frame f i is as follows: In the formula, l i,seg is the length of the crowd road in the real world of the i-th crowd direction in the current image frame f t , which is converted by the conversion rate r t to the pixel length of the crowd road, n i is the total number of personnel belonging to the i-th crowd direction in the current image frame f t , and s t is the average speed of the i-th crowd direction in the current image frame f i,avg , and the calculation formula is as follows: In the formula, s j is the moving speed of the j-th personnel in the i-th crowd direction in the real world, and n i is the total number of personnel belonging to the i-th crowd direction in the current image frame f t , and v t is the crowd flow of the i-th crowd direction in the current image frame f i , and the calculation formula is as follows: v i =s i,avg ×d i .

[0015] In the outdoor crowd information extraction method based on the UAV video provided by the application, the following features can also be provided: when there is an identifiable landmark reference in the current image frame f t , the calculation formula of the conversion rate r t is as follows: When there is no representative reference or the representative reference cannot be identified in the current image frame f t , and the UAV can provide real-time flight height data, the calculation formula of the conversion rate r t is as follows: r1 is the conversion rate of the image frame closest to the current image frame f t and before the current image frame f t , h1 is the UAV height of the image frame corresponding to the conversion rate r1, and when a representative reference object does not appear in the current image frame f t or the representative reference object cannot be recognized, and the UAV cannot provide real-time flight height data of itself, the calculation formula of the conversion rate r t is as follows: where whether the difference between the two averages is significant is determined by a two-sample t-test of independent samples with a bilateral test level a = 0.05, r1 is the conversion rate of the image frame closest to the current image frame f t and before the current image frame f t , and ω1 is the average length of all the bounding boxes in the image frame corresponding to the conversion rate r1.

[0016] Effects of the Invention

[0017] According to the outdoor crowd information extraction method based on UAV video, the recognition of different direction crowds is realized by clustering the personnel motion vectors in the 2D displacement space, the conversion rate of the image frame pixel size and the corresponding real world size is calculated according to the representative reference object or the UAV flight height data or the bounding box, and the extraction of the crowd flow, the average speed of the crowd flow and the crowd density is realized, so that the outdoor crowd information extraction method based on UAV video can accurately identify the personnel target, recognize different direction crowds and extract crowd information from the outdoor crowd monitoring video shot by the UAV. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the outdoor crowd information extraction method based on UAV video in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the outdoor crowd information extraction method based on UAV video of the present application is specifically described in the following embodiments in combination with the drawings.

[0020] In this embodiment, the frame rate of the outdoor crowd monitoring video is 25fps.

[0021] Figure 1 is a flowchart of the outdoor crowd information extraction method based on UAV video in the embodiment of the present application.

[0022] As Figure 1As shown, the outdoor crowd information extraction method based on unmanned aerial vehicle video in the embodiment of the application is used to obtain the crowd flow, average speed and density of outdoor crowd flow according to the outdoor crowd monitoring video shot by the unmanned aerial vehicle, and includes the following steps:

[0023] Step S1, according to the frame rate of the outdoor crowd monitoring video, frame extraction is performed on the outdoor crowd monitoring video at the same time interval to obtain a plurality of image frames. If the time interval is too small, the motion vectors between personnel in different directions will not be obvious, which will affect the performance of the subsequent clustering algorithm in distinguishing the crowd flow. If the time interval is too large, the resolution of the finally extracted crowd flow parameters, i.e., the crowd flow, average speed and density of the crowd flow, will be insufficient. In this embodiment, according to the frame rate 25fps of the input outdoor crowd monitoring video, the frame extraction time interval is selected as 0.2 seconds, i.e., one frame is extracted every 5 frames.

[0024] Step S2, the image frame is input into a convolution network algorithm model, the personnel and the background are separated from the image frame through a bounding box, each personnel is placed in the bounding box, and the background is placed outside the bounding box.

[0025] The convolution network algorithm model is FastR-CNN or MASKR-CNN or Haar cascade + self-defined convolution network algorithm model or YOLOv5 convolution network algorithm model or SSD convolution network algorithm model.

[0026] When the convolution network algorithm model is FastR-CNN, step S2 includes the following sub-steps:

[0027] Step S2-1, according to the RPN module of FastR-CNN, a plurality of candidate regions of the image frame and corresponding classification scores are generated.

[0028] Step S2-2, the candidate regions with more pixel intersections are merged through a non-maximum suppression algorithm.

[0029] The plurality of candidate regions with more pixel intersections can be determined by setting a threshold, for example, a certain candidate region is taken as a reference, and a threshold of 2 / 3 is set. If the pixel overlap of another candidate region with the candidate region exceeds 2 / 3 of the total pixels of the candidate region, the other candidate region is determined as a candidate region with more pixel intersections.

[0030] Step S2-3, all the merged candidate regions are sorted according to the classification scores from high to low, and the first N candidate regions are reserved as the attention regions.

[0031] Step S2-4, the attention regions are input into the RoI pooling layer of Fast R-CNN, and the position and size of each bounding box are output through the softmax layer and the bounding box regression layer, so as to mark each bounding box in the image frame.

[0032] In this embodiment, the convolutional layer output is mapped to low-dimensional features by sliding a plurality of n*n windows over the feature map of the image frame output by the last shared convolutional layer, and then the low-dimensional features are input to two fully connected layers: a Box-Regression Layer (REG) and a Box-Classification Layer (BCL). For each sliding window position, k possible rectangular candidate regions are generated in the BCL layer, and the BCL layer also outputs 2k classification scores to represent the likelihood of the candidate regions including or not including a person. The REG layer outputs 4k coordinates to mark the position of the candidate region rectangular frame, i.e. the bounding box, in the image frame.

[0033] Step S3: Based on the Shi-Tomasi features, mark the interest points of the image frame, and mark the interest points within the bounding box as person interest points and the interest points outside the bounding box as background interest points.

[0034] Step S3 includes the following sub-steps:

[0035] Step S3-1: Slide a detection window ω of size (2ω x +1)×(2ω y +1) over the entire image I of the image frame, and calculate the second-order matrix G of the pixel point u=(p x ,p y ) in the detection window ω. The formula of the second-order matrix G is as follows:

[0036]

[0037] In the formula, I x is the pixel value corresponding to p x , and I y is the pixel value corresponding to p y .

[0038] Step S3-2: When the second-order matrix G is a non-singular matrix and the minimum eigenvalue λ min is greater than the threshold λ th , mark the corresponding pixel point as a comparison point, and mark the pixel point corresponding to the maximum minimum eigenvalue λ min in the 3×3 neighborhood as an interest point.

[0039] In the formula, λ min =min(λ1,λ2), λ1 and λ2 are the eigenvalues of the second-order matrix G, and the threshold λ th is used to determine how many interest points there are. The threshold λ th is set according to the pixels of the input image frame, the recognition accuracy requirement, and the machine operation amount.

[0040] Step S3-3: Mark all the interest points according to the minimum eigenvalue λmin Arrange from high to low, retain the first n interest points to balance processing speed and recall rate, and regard the interest points within the bounding box as person interest points, and the interest points outside the bounding box as background interest points.

[0041] Step S4: Track the current image frame f according to the KLT optical flow algorithm t The interest point in the image frame and the next adjacent time frame f t+1 The corresponding interest point in the image is calculated to obtain the motion vector of the interest point. t The motion vectors of all points of interest in the current image frame f are calculated t The person motion vector of each person and the background motion vector of the background.

[0042] Among them, the current image frame f t The calculation formulas for the personnel motion vector and background motion vector are as follows:

[0043]

[0044]

[0045] In the formula is the background motion vector, N b is the current image frame f t The total number of background interest points in is the motion vector of the jth background interest point, is the motion vector of the kth person, N t is the total number of interest points marked by the kth person, is the motion vector of the i-th person's interest point of the k-th person.

[0046] Step S5: the current image frame f t All personnel motion vectors are put into the Cartesian rectangular coordinate system, and the motion parameters l of each personnel motion vector are calculated. k and θ k The personnel motion parameter serving as the personnel motion vector.

[0047] Among them, the motion parameter l of the kth person is k and θ k The calculation formula is as follows:

[0048]

[0049] θ k =tan -1 (s y / s x ),

[0050] Where sx and s y is the personnel motion vector of the kth personnel horizontal and vertical components in Cartesian rectangular coordinate system.

[0051] Step S6, put all personnel motion parameters of the current image frame f t into 2-dimensional displacement space, cluster all personnel motion parameters by clustering algorithm, each cluster represents different flow direction in the current image frame f t , 2-dimensional displacement space takes absolute displacement as horizontal axis and direction angle as vertical axis, in this embodiment, clustering algorithm is classical k-means algorithm.

[0052] Step S7, according to conversion rate r t , personnel motion vector and background motion vector of the current image frame f t , calculate moving speed of personnel and background in real world in the current image frame f t .

[0053] wherein, moving speed s of personnel or background in real world in the current image frame f t is calculated by following formula:

[0054]

[0055] t=t f +t p ,

[0056]

[0057] wherein, l is absolute value of personnel motion vector or background motion vector of the personnel in the current image frame f t , t f is frame processing time interval, t p is processing time of the current image frame f t , n is total number of image frames before the current image frame f t , t pi is processing time of the i-th image frame before the current image frame f t , t p1 is processing time of the image frame of the previous adjacent time on the current image frame f t , and a is attenuation coefficient, which is greater than or equal to 0; when a is +∞, t p is only related to t p1 , i.e. t p =t p1 ; when a is 0, t p is only related to previous n historical processing times, i.e. there is no attenuation of correlation weight, then In this embodiment, tp It is the estimated value of the total time required for the background GPU to process the image plus the communication delay, etc., for single-frame image processing. f The frame extraction processing time interval is set to 0.2 seconds.

[0058] Step S8, according to the current image frame f t The number of bounding boxes, the speed of movement of people in the real world, and the conversion rate r t , calculate the current image frame f t The flow of people, average speed and density of people in each direction of crowd flow.

[0059] Among them, the current image frame f t The flow density d of the i-th flow direction i The calculation formula is as follows:

[0060]

[0061] Where l i,seg is the current image frame f t The length of the pedestrian flow path in the real world in the direction of the i-th pedestrian flow is calculated by the conversion rate r t The pixel length of the pedestrian road is converted to n i is the current image frame f t The total number of people belonging to the i-th flow direction.

[0062] Current image frame f t The average speed of the flow in the direction of the i-th flow s i,avg The calculation formula is as follows:

[0063]

[0064] Where s j is the real-world moving speed of the jth person in the i-th flow direction, n i is the current image frame f t The total number of people belonging to the i-th flow direction.

[0065] Current image frame f t The flow of people v in the direction of the i-th flow i The calculation formula is as follows:

[0066] v i =s i,avg ×d i .

[0067] Conversion rate r t According to the current image frame f t The pixel size of the landmark reference in l pand the corresponding real-world size l a It is obtained that the landmark reference is an object with a standard length, such as a bus, a road sign, etc.; or according to the current image frame f t The height h t of the UAV of the image frame f t-1 of the previous adjacent time t-1 The conversion rate r t-1 of the image frame f t-1 It is obtained; or according to the average length ω t of all the bounding boxes in the current image frame f t The average length ω t-1 of all the bounding boxes in the image frame f t-1 The conversion rate r t-1 of the image frame f t-1 It is obtained.

[0068] When there is an identifiable landmark reference in the current image frame f t , the conversion rate r t is calculated as follows:

[0069]

[0070] When there is no representative reference or the representative reference is not identifiable in the current image frame f t , and the UAV can provide real-time flight height data, the conversion rate r t is calculated as follows:

[0071] ...

[0073]

[0074] In the formula, r1 is the conversion rate of the image frame with the landmark reference closest to the current image frame f t and before the current image frame f t , and h1 is the UAV height of the image frame corresponding to the conversion rate r1.

[0075] When there is no representative reference or the representative reference is not identifiable in the current image frame f t , and the UAV cannot provide real-time flight height data, the conversion rate r t is calculated as follows:

[0076] ...

[0078]

[0079] In the formula, whether the difference between two average numbers is significant is judged by a two-sample independent t test with a bilateral test level of a = 0.05, a first sample set in the two-sample independent t test is all the lengths of the bounding boxes of the persons in the current image frame f t t , a second sample set is the image frame f t-1 of the previous adjacent time, a sample average value is ω t-1 , r1 is a conversion rate of an image frame having a landmark reference and closest to the current image frame f t and before the current image frame f t , and ω1 is an average length of all the bounding boxes in the image frame corresponding to the conversion rate r1.

[0080] Effects of the embodiment

[0081] According to the outdoor crowd information extraction method based on the UAV video, the recognition of the crowd in different directions is realized by clustering the motion vectors of the persons in the 2D displacement space, the conversion rate of the image frame pixel size and the corresponding real world size is calculated according to the landmark reference or the UAV flight height data or the bounding box, and then the extraction of the crowd flow, the average speed of the crowd and the crowd density is realized. In summary, the method can realize the accurate recognition of the personnel target, the recognition of the crowd in different directions and the extraction of the crowd information for the outdoor crowd monitoring video shot by the UAV.

[0082] The above embodiment is a preferred case of the present application and does not limit the protection scope of the present application.​

Claims

1. A method for extracting outdoor crowd flow information based on drone video, which is used to obtain the outdoor crowd flow, average crowd flow speed and crowd flow density based on the outdoor crowd flow monitoring video taken by the drone, characterized in that: The following steps are involved: Step S1, extracting frames from the outdoor crowd monitoring video at the same time interval according to the frame rate of the outdoor crowd monitoring video to obtain multiple image frames; Step S2: inputting the image frame into a convolutional network algorithm model, separating the person and the background of the image frame using a bounding box, placing each person within the bounding box, and placing the background outside the bounding box; Step S3, marking the points of interest of the image frame based on the Shi-Tomasi feature, taking the points of interest within the bounding box as the people's points of interest, and taking the points of interest outside the bounding box as the background points of interest; Step S4: Track the current image frame f according to the KLT optical flow algorithm t The interest point in the image frame f at the next adjacent moment t+1 The corresponding interest point in the image is calculated to obtain the motion vector of the interest point, and the motion vector of the interest point is calculated according to the current image frame f t The motion vectors of all the interest points in the current image frame f are calculated to obtain t Personnel motion vectors of the respective persons and background motion vectors of the background; Step S5: the current image frame f t All the motion vectors of the personnel are put into the Cartesian rectangular coordinate system, and the motion parameters l of each motion vector of the personnel are calculated. k and θ k A person motion parameter as the person motion vector; Step S6: the current image frame f t All the motion parameters of the personnel are put into a 2-dimensional displacement space, and clustered by a clustering algorithm. Each cluster represents the current image frame f t Different directions of people flow; Step S7, according to the current image frame f t The conversion rate r t , the personnel motion vector and the background motion vector, and calculate the current image frame f t The speed at which the person and the background move in the real world; Step S8, according to the current image frame f t The number of the bounding boxes, the speed of the person moving in the real world and the conversion rate r t , calculate and obtain the current image frame f t The crowd flow, the average crowd flow speed and the crowd flow density of each of the crowd flow directions, Wherein, the conversion rate r t According to the current image frame f t The pixel size of the landmark reference in l p and the corresponding real-world size l a get; or According to the current image frame f t The drone's altitude h t , the image frame f at the previous adjacent moment t-1 The drone's altitude h t-1 and the image frame f t-1 The conversion rate r t-1 get; or According to the current image frame f t The average length ω of all the bounding boxes in t , the image frame f t-1 The average length ω of all the bounding boxes in t-1 and the image frame f t-1 The conversion rate r t-1 get.

2. The method for extracting outdoor crowd flow information based on drone video according to claim 1 is characterized in that: in, In step S2, the convolutional network algorithm model is Fast R-CNN or MASK R-CNN or Haar cascade + custom convolutional network algorithm model or YOLOv5 convolutional network algorithm model or SSD convolutional network algorithm model.

3. The outdoor crowd flow information extraction method based on drone video according to claim 2, Its characteristics are: When the convolutional network algorithm model is Fast R-CNN, step S2 includes the following sub-steps: Step S2-1, generating multiple candidate regions and corresponding classification scores of the image frame according to the RPN module of the Fast R-CNN; Step S2-2, merging the candidate regions with more pixel intersections by a non-maximum suppression algorithm; Step S2-3, sorting all the merged candidate regions from high to low according to the classification scores, and retaining the top N candidate regions as focus regions; In step S2-4, the region of interest is input into the RoI pooling layer of the Fast R-CNN, and the softmax layer and the bounding box regression layer jointly output the region of interest to obtain the position and size of each bounding box, and mark each bounding box in the image frame.

4. The method for extracting outdoor crowd flow information based on drone video according to claim 1, Its characteristics are: Wherein, the step S3 includes the following sub-steps: Step S3-1, the size of (2ω x +1)×(2ω y +1) slides the detection window ω over the entire image I of the image frame, and calculates the pixel u in the detection window ω = (p x , p y )’s second-order matrix G; Step S3-2: transform the second-order matrix G into a non-singular matrix with the minimum eigenvalue λ min Greater than the threshold λ th The corresponding pixel point is used as the comparison point, and the largest minimum eigenvalue λ in the 3×3 neighborhood is used as the comparison point. min The corresponding pixel point is marked as the interest point; Step S3-3: all the points of interest are divided into two groups according to the minimum eigenvalue λ. min Arrange from high to low, retain the first n points of interest, use the points of interest within the bounding box as the personnel points of interest, and use the points of interest outside the bounding box as the background points of interest.

5. The method for extracting outdoor crowd flow information based on drone video according to claim 4 is characterized in that: in, In step S3-1, the formula of the second-order matrix G is as follows: Where I x For p x The corresponding pixel value, I y For p y The corresponding pixel value.

6. The method for extracting outdoor crowd flow information based on drone video according to claim 1, characterized in that: in, In the step S4, the current image frame f t The calculation formulas of the personnel motion vector and the background motion vector are as follows: In the formula is the background motion vector, N b is the current image frame f t The total number of background interest points in is the motion vector of the jth background interest point, is the motion vector of the kth person, N t is the total number of interest points marked by the kth person, is the motion vector of the i-th person's interest point of the k-th person.

7. The method for extracting outdoor crowd flow information based on drone video according to claim 1, characterized in that: in, In step S5, the motion parameter l of the kth person k and θ k The calculation formula is as follows: θ k =tan -1 (s y / s x ), Where s x and s y are the motion vectors of the kth person respectively The horizontal and vertical components in a Cartesian coordinate system.

8. The method for extracting outdoor crowd flow information based on drone video according to claim 1, characterized in that: in, In step S7, the current image frame f t The calculation formula of the moving speed s of the person or the background in the real world is as follows: t=t f +t p , Where l is the current image frame f t The absolute value of the person's motion vector or background motion vector in t f is the frame extraction processing time interval, t p is the current image frame f t The processing time of n is the current image frame f t The total number of image frames at the previous time, t pi is the current image frame f t The processing time of the previous i-th image frame, α is the attenuation coefficient, and its value is non-negative.

9. The method for extracting outdoor crowd flow information based on drone video according to claim 1, characterized in that: in, In step S8, the current image frame f t The flow density d of the i-th flow direction i The calculation formula is as follows: Where l i，seg is the current image frame f t The length of the pedestrian flow path in the real world in the direction of the i-th pedestrian flow is calculated by the conversion rate r t The pixel length of the pedestrian road is converted to n i is the current image frame f t The total number of people belonging to the i-th flow direction, The current image frame f t The average speed of the flow in the direction of the i-th flow s i，avg The calculation formula is as follows: Where s j is the real-world moving speed of the jth person in the i-th flow direction, n i is the current image frame f t The total number of people belonging to the i-th flow direction, The current image frame f t The flow of people v in the direction of the i-th flow i The calculation formula is as follows: v i =s i，avg ×d i 。 10. The method for extracting outdoor crowd flow information based on drone video according to claim 1, characterized in that: in, When the current image frame f t When there is a recognizable landmark reference, the conversion rate r t The calculation formula is as follows: When the current image frame f t When the representative reference object does not appear in the image or the representative reference object is unrecognizable, and the UAV is able to provide its own flight altitude data in real time, the conversion rate r t The calculation formula is as follows: ......， Where r1 is a landmark reference object closest to the current image frame f t And in the current image frame f t The conversion rate of the previous image frame, h1 is the drone height of the image frame corresponding to the conversion rate r1, When the current image frame f t When the representative reference object does not appear in the image or the representative reference object is unrecognizable, and the UAV cannot provide its own flight altitude data in real time, the conversion rate r t The calculation formula is as follows: ......， In the formula, a two-sided independent sample t-test with a two-sided test level of α = 0.05 is used to determine whether the difference between the two means is significant, and r1 is a landmark reference closest to the current image frame f t And in the current image frame f t The conversion rate of the previous image frame, ω1 is the average length of all the bounding boxes in the image frame corresponding to the conversion rate r1.

Images