Unmanned aerial vehicle crowd trajectory judgment method and device based on YOLO and semantic segmentation

Abstract

The application relates to a UAV crowd trajectory judgment method and device based on YOLO and semantic segmentation, and relates to the technical field of crowd behavior analysis. The method comprises the following steps: continuously collecting video frame images by a UAV, inputting the continuous video frame images into a pre-trained target detection network and a semantic segmentation network respectively, fusing segmentation mask and detection box information, realizing calibration and smooth estimation of a target detection box, and generating a short-term smooth trajectory. Long-term trajectory prediction is carried out in combination with a long short-term memory network (LSTM), two kinds of trajectories are weighted and fused through a fusion strategy, and finally, a stable trajectory judgment result is output. The application effectively solves the problems of insufficient crowd detection accuracy, poor trajectory continuity and serious shielding interference of the UAV in a complex intersection scene. The application improves the accuracy and stability of crowd detection and trajectory judgment, effectively deals with shielding and dynamic changes, and guarantees the autonomous tracking and control capability of the UAV in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of crowd behavior analysis technology, and in particular to a method and apparatus for determining crowd trajectories of drones based on YOLO and semantic segmentation. Background Technology

[0002] With the increasing demand for urban traffic management and public safety monitoring, drones, as a new technological means, are being widely used for real-time monitoring of intersection environments and analysis of crowd behavior.

[0003] Currently, a combination of target detection and tracking methods is mainly used to determine crowd trajectories. For example, an improved YOLO network is typically used for pedestrian detection, and traditional tracking algorithms such as KCF or T-SORT are combined to correlate the detection results and predict trajectories.

[0004] However, existing technologies suffer from problems such as detection box scale distortion, trajectory breakage, sensitivity to occlusion, and lack of temporal motion modeling when dealing with complex intersection scenes from the perspective of drones. They are unable to effectively cope with the challenges brought by perspective distortion and dynamic occlusion, resulting in insufficient continuity and accuracy of trajectory judgment, which limits the practical application effect of drones in complex traffic environments. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for determining the trajectory of drone crowds based on YOLO and semantic segmentation, aiming to solve at least one of the above-mentioned technical problems.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: Firstly, this application provides a method for determining the trajectory of drone crowds based on YOLO and semantic segmentation, employing the following technical solution: A method for determining drone crowd trajectories based on YOLO and semantic segmentation includes: Acquire image information containing crowds collected by the drone in the target area, the image information including multiple consecutive video frames; The image information is input into a pre-trained target detection network and a semantic segmentation network to obtain a detection feature map and a segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. Based on the segmentation feature map, the individual detection boxes in the detection feature map are calibrated to obtain a calibrated sequence of individual detection boxes; Based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, the position and velocity state of the target individual in the current frame are smoothly estimated to generate a short-term smooth trajectory. An input sequence is constructed based on the short-term smooth trajectory and input into a pre-trained long short-term memory network to generate a long-term predicted trajectory. Based on a preset fusion strategy, the long-term predicted trajectory and the short-term smoothed trajectory are fused to determine the trajectory judgment result of the target population.

[0007] Based on the above technical solution, the present invention can be further improved as follows.

[0008] Furthermore, acquiring image information containing crowds collected by the drone in the target area includes: Acquire initial continuous video frame images containing crowds collected by the drone in the target area; For each frame of the original image, a distortion mapping table is established based on the UAV camera calibration parameters. The pixel coordinates of the image are remapped through bilinear interpolation to obtain the distortion-corrected image. For each frame of the original image, based on the distortion-corrected image, the images adjacent to the distortion-corrected image, and the lightweight deblurring network of the convolutional neural network, the blur kernel of the intermediate frame is estimated, and the clear image is recovered through non-blind deconvolution to obtain the deblurred image; For each frame of the original image, the deblurred image is converted from the RGB color space to the HSV color space, and the luminance component is subjected to adaptive histogram equalization (CLAHE) to obtain the enhanced image. For each frame of the original image, the enhanced image is subjected to a reverse rotation transformation based on a pre-constructed rotation matrix to obtain a geometrically corrected image; For each frame of the original image, the region of interest (ROI) is extracted and its size is normalized in the geometrically corrected image to obtain the preprocessed image information.

[0009] Furthermore, the object detection network is built based on an improved YOLOv8 network. This improved YOLOv8 network includes a head network, which is optimized using a composite loss function. This composite loss function is obtained by weighting the classification loss and the bounding box regression loss. The bounding box regression loss is the generalized intersection-union ratio (GIoU) ​​loss, and the classification loss is the focal loss (FocalLoss). The formula for calculating the Generalized Intersection over Union (GIoU) ​​loss is as follows: L GIoU =1-IoU+(|C-(B p ∪B gt )|) / |C|; Among them, B p For the prediction box, B gtLet C be the ground truth bounding box, and let C be the smallest bounding rectangle that contains both the predicted bounding box and the ground truth bounding box. The formula for calculating the classification loss, or Focal Loss, is as follows: L FL =-α×(1-p t )γ×log(p t ); Where, p t Let α be the model's predicted probability of the true class, γ be the balancing factor, and γ be the focusing parameter.

[0010] Further, the step of calibrating the individual detection boxes in the detection feature map based on the segmentation feature map to obtain a calibrated sequence of individual detection boxes includes: The segmentation feature map is classified at the pixel level, and the pixel regions corresponding to the pedestrian categories are extracted to generate a crowd region mask; The crowd region mask is multiplied pixel by pixel with the detection feature map to generate a filtered detection feature map; Connectivity analysis is performed on the filtered detection feature map, and the bounding rectangles of each connected region are extracted as a preliminary calibration detection box set; Perform connected component analysis on the crowd region mask and extract the bounding rectangles of each connected component as the candidate crowd region set; The preliminary calibration detection box set and the candidate group region set are subjected to intersection-union matching. The areas where there are candidate group regions but are not covered by the preliminary calibration detection boxes are determined as target occluded areas and supplementary detection boxes are generated. The preliminary calibration detection box is merged with the supplementary detection box to generate a calibrated individual detection box sequence.

[0011] Furthermore, the generation of supplementary detection boxes includes: For candidate group regions determined to be occluded by the target, extract the center coordinates and size information of the bounding rectangle of the candidate group region; The trajectory information of the target corresponding to the candidate group region in adjacent historical frames is obtained, and the trajectory information includes the position coordinates and velocity vector of the target in the historical frames. Based on the trajectory information and the preset least squares fitting linear motion model, the average velocity vector of the target in the current frame is determined; Based on the average velocity vector and the time interval between the current frame and the previous frame, the predicted position coordinates of the supplementary detection box are calculated using a motion extrapolation formula. The supplementary detection box is generated based on the predicted location coordinates and the size information.

[0012] Furthermore, the step of smoothly estimating the position and velocity state of the target individual in the current frame based on the calibrated individual detection box sequence and a preset temporal filtering algorithm to generate a short-term smooth trajectory includes: Based on the calibrated individual detection box sequence, the center coordinates of each target individual detection box in each frame are extracted as observations to construct a state vector containing position and velocity. A state transition equation is established based on a pre-defined uniform motion model. The state vector of the target individual in the current frame is predicted a priori using the state vector of the previous time step and the state transition matrix at the current time step, so as to obtain the predicted state vector and the prediction error covariance matrix. Calculate the Kalman gain based on the prediction error covariance matrix and the observation noise covariance matrix; Based on the Kalman gain and the observations of the current frame, the predicted state vector is corrected to obtain the posterior position estimate and posterior velocity estimate of the target individual in the current frame. The posterior position estimates of multiple consecutive frames are sequentially concatenated to generate the short-term smooth trajectory.

[0013] Furthermore, the step of constructing an input sequence based on the short-term smooth trajectory and inputting it into a pre-trained long short-term memory network to generate a long-term predicted trajectory includes: Based on a preset sliding window, a set number of trajectory points are determined and an input sequence is formed, where each trajectory point includes the position coordinates of the target in the image coordinate system; The input sequence is fed step by step into the forget gate, input gate and output gate of the pre-trained long short-term memory network to determine the hidden state at the current time. The hidden state of the last time step is input into the fully connected layer, and the coordinates of the predicted trajectory points are determined by linear transformation to obtain the long-term predicted trajectory.

[0014] Furthermore, the process of fusing the long-term predicted trajectory and the short-term smoothed trajectory based on a preset fusion strategy to determine the trajectory judgment result of the target population includes: Based on the preset occlusion probability calculation formula, calculate the occlusion probability of the target in the current frame. Based on the occlusion probability, the fusion weight coefficients are determined; The short-term smooth trajectory prediction value and the long-term predicted trajectory value are weighted and fused according to the fusion weight coefficient to obtain the fused trajectory point; The fused trajectory points are smoothed, and the smoothed fused trajectory points are used as the trajectory judgment result of the target population.

[0015] Secondly, this application provides a drone crowd trajectory determination device based on YOLO and semantic segmentation, employing the following technical solution: A drone crowd trajectory determination device based on YOLO and semantic segmentation includes: The image acquisition module is used to acquire image information containing crowds collected by the UAV in the target area, and the image information includes multiple consecutive video frame images; The parallel detection and segmentation module is used to input the image information into a pre-trained target detection network and a semantic segmentation network respectively to obtain a detection feature map and a segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. The calibration module is used to calibrate the individual detection boxes in the detection feature map based on the segmentation feature map, so as to obtain a sequence of calibrated individual detection boxes; The short-term trajectory generation module is used to smoothly estimate the position and velocity state of the target individual in the current frame based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, and generate a short-term smooth trajectory. The long-term trajectory prediction module is used to construct an input sequence based on the short-term smooth trajectory and input it into a pre-trained long short-term memory network to generate a long-term predicted trajectory. The trajectory fusion module is used to fuse the long-term predicted trajectory and the short-term smoothed trajectory based on a preset fusion strategy to determine the trajectory judgment result of the target population.

[0016] Thirdly, this application provides an electronic device that adopts the following technical solution: An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any of the first aspects regarding the method for determining the trajectory of a drone crowd based on YOLO and semantic segmentation.

[0017] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium storing a computer program capable of being loaded by a processor and executing the method for determining the trajectory of a drone crowd based on YOLO and semantic segmentation as described in any of the first aspects.

[0018] In summary, the beneficial effects of this invention are: 1. By employing an improved YOLOv8 network with a composite loss function and combining it with a pixel-level crowd region mask generated by a semantic segmentation network, spatial constraints are applied to the detection feature map, effectively filtering out false detections caused by complex backgrounds. Simultaneously, by performing connected component analysis and intersection-union-ratio matching on the filtered detection boxes and the segmentation mask, occlusion regions are accurately identified. Supplementary detection boxes are generated by combining historical trajectories and motion extrapolation, reducing the false detection rate of small targets in densely occluded scenes and achieving a dual enhancement of detection accuracy and robustness. 2. By first using Kalman filtering to smooth the calibrated detection box sequence and generating a short-term smooth trajectory to suppress detection noise, a long short-term memory network is introduced to learn the nonlinear motion pattern of the target over a long time span, generating a long-term predicted trajectory, thus achieving an accurate grasp of the target's motion trend. When the detection is interrupted due to target occlusion or rapid maneuvering, this dual-mode prediction mechanism can still extrapolate the current state based on historical motion patterns, effectively maintaining the continuity of the trajectory and avoiding the trajectory interruption and identity switching problems common in traditional methods. 3. By fusing semantic segmentation confidence and detection continuity information, the target occlusion probability of the current frame is evaluated in real time, and the fusion weights of the Kalman filter short-term trajectory and the LSTM long-term trajectory are dynamically adjusted accordingly. Smoothing the fused trajectory eliminates jitter that may be introduced by weighted fusion, ensuring the smoothness and accuracy of the final output trajectory.

[0019] 4. Deep collaboration and closed-loop optimization among the detection, segmentation, tracking, and prediction modules have been achieved. Semantic segmentation provides spatial constraints for detection, detection results provide a data foundation for tracking, and temporal prediction in turn provides compensation for detection under occlusion conditions. It can easily cope with highly challenging intersection environments such as drastic changes in lighting, different target sizes, and dynamic occlusion interference, and has extremely high practical robustness. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a method for determining the trajectory of a drone crowd based on YOLO and semantic segmentation, provided as an embodiment of the present invention; Figure 2 This is a schematic diagram of a drone crowd trajectory determination device based on YOLO and semantic segmentation, provided in one embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0023] This application provides a method for determining the trajectory of a drone crowd based on YOLO and semantic segmentation. This method can be executed by an electronic device, which can be a server or a mobile terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The mobile terminal device can be a laptop, a desktop computer, etc., but is not limited to these.

[0024] like Figure 1 As shown, a method for determining drone crowd trajectories based on YOLO and semantic segmentation includes: S101, acquire image information containing crowds collected by the drone in the target area, the image information including multiple consecutive video frame images; In this embodiment, the target area refers to a specific geographical area where the UAV performs its monitoring mission, such as densely populated places like urban intersections, pedestrian streets, and squares. Image information refers to frame image data continuously collected by the camera equipment mounted on the UAV in video format, including multiple consecutive video frames arranged in chronological order, with each frame containing the distribution of people within the target area at the current moment.

[0025] When a drone performs patrol or hovering monitoring missions over a target area, it continuously acquires image data containing crowds within the target area in the form of a video stream using its onboard wide-angle camera. Because the raw images acquired by the drone often suffer from lens distortion, motion blur, uneven lighting, and perspective distortion caused by its posture, preprocessing of the initial continuous video frames is necessary to obtain high-quality image information.

[0026] In this embodiment of the application, obtaining image information containing crowds collected by the drone in the target area includes: S11, acquire initial continuous video frame images containing crowds collected by the drone in the target area; S12, For each frame of the original image, a distortion mapping table is established based on the UAV camera calibration parameters, and the pixel coordinates of the image are remapped through bilinear interpolation to obtain the distortion-corrected image; When a wide-angle camera on a drone captures images, radial and tangential distortions inevitably occur, which manifest as straight line bending and stretching deformation at the image edges. Based on the camera intrinsic parameters and distortion coefficients obtained in advance using the Zhang Zhengyou calibration method, for each pixel in the input image, its corresponding position in the original distorted image is calculated by reverse mapping through the distortion model. Since this position is usually a floating-point coordinate, a bilinear interpolation algorithm is used to calculate the gray value of the target pixel by weighted averaging the gray values ​​of its four nearest neighbor pixels.

[0027] S13, for each frame of the original image, based on the distortion-corrected image, the image adjacent to the distortion-corrected image, and the lightweight deblurring network of the convolutional neural network, the blur kernel of the intermediate frame is estimated, and the clear image is recovered by non-blind deconvolution to obtain the deblurred image; This lightweight deblurring network takes three consecutive frames (previous frame, current frame, and next frame) as input and uses inter-frame redundancy information to estimate the blur kernel (point spread function) of the current frame. The network adopts an encoder-decoder architecture. The encoder extracts multi-scale features through multiple convolutional layers and max pooling layers, and the decoder recovers high-resolution features through upsampling and skip connections. After obtaining the blur kernel, a non-blind deconvolution algorithm based on hyper-Laplacian prior is used to recover the clear image in the frequency domain.

[0028] S14. For each frame of the original image, the deblurred image is converted from the RGB color space to the HSV color space, and the luminance component is subjected to adaptive histogram equalization (CLAHE) to obtain the enhanced image. Specifically, the contrast limit threshold is set to 2.0 to suppress noise amplification. The image is divided into an 8×8 grid, and histogram equalization is performed independently within each grid. Then, bilinear interpolation is used to eliminate block artifacts.

[0029] S15, for each frame of the original image, based on the pre-constructed rotation matrix, the enhanced image is subjected to a reverse rotation transformation to obtain a geometrically corrected image; The rotation matrix is ​​constructed using the pitch, roll, and yaw angles recorded by the UAV's inertial measurement unit (IMU). The angles along the three axes are converted into rotation matrices according to the Euler angle rotation order. Each pixel in the image is converted into a three-dimensional direction vector in the camera coordinate system through the camera intrinsic parameter matrix and then rotated to a horizontal attitude using the rotation matrix. Finally, its coordinate position in the horizontal ground plane image is calculated through projection transformation.

[0030] S16, for each frame of the original image, the region of interest (ROI) is extracted and its size is normalized in the geometrically corrected image to obtain the preprocessed image information.

[0031] In this embodiment, the target area is an intersection. Based on prior knowledge of the intersection scene, the bottom 1 / 3 of the image is set as the near-end pedestrian dense area and the middle 1 / 3 of the image is set as the far-end pedestrian area. The image is scaled to a fixed size of 640×640 pixels by bicubic interpolation and the pixel values ​​are mapped from [0,255] to the [0,1] interval.

[0032] S102, the image information is input into the pre-trained target detection network and semantic segmentation network respectively to obtain detection feature map and segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. In this embodiment, the object detection network refers to a deep learning network that, after being trained on a large amount of labeled data, can locate and classify objects in the input image. In this embodiment, it is built based on an improved YOLOv8 network. The semantic segmentation network refers to a deep learning network that can classify each pixel in the input image and output the probability distribution of each pixel belonging to each category. The individual detection box refers to the rectangular bounding box output by the object detection network for each identified pedestrian in the image, used to mark the spatial location and range of the pedestrian in the image. The pixel-level category probability distribution refers to the probability vector of each category (such as pedestrian, vehicle, road, background, etc.) corresponding to each pixel in the segmentation feature map.

[0033] In this embodiment, preprocessed consecutive video frame images are input in parallel into a pre-trained object detection network and a semantic segmentation network for inference. The object detection network is built based on an improved YOLOv8 network. The head network of this improved YOLOv8 network is optimized using a composite loss function, which is obtained by weighting the classification loss and the bounding box regression loss. Specifically, the bounding box regression loss is the generalized intersection-union (GIoU) ​​loss, and the classification loss is the focal loss.

[0034] The formula for calculating the Generalized Intersection over Union (GIoU) ​​loss is as follows: L GIoU =1-IoU+(|C-(B p ∪B gt )|) / |C|; Among them, B p For the prediction box, B gt Let C be the ground truth bounding box, and let C be the smallest bounding rectangle that contains both the predicted bounding box and the ground truth bounding box. This loss function can still drive the predicted box to move closer to the true box by minimizing the circumscribed rectangle C, even when there is no overlap between the predicted box and the true box, thus avoiding gradient vanishing.

[0035] The formula for calculating the classification loss, or Focal Loss, is as follows: L FL =-α×(1-p t )γ×log(p t ); Where, p t α is the model's predicted probability of the true class, α is the balance factor (usually 0.25), and γ is the focusing parameter (usually 2).

[0036] By reducing the weight of a large number of easily classified negative samples during training using a modulation factor (1-pt)γ, the model focuses on difficult-to-classify pedestrian targets. The object detection network outputs a detection feature map containing multiple individual detection boxes for each input image frame. Each detection box carries its location coordinates in the image, its category, and a confidence score. Simultaneously, the semantic segmentation network outputs a segmentation feature map for the same image frame. This segmentation feature map calculates the category probability for each pixel in the image, identifying the pixel distribution belonging to the crowd region.

[0037] S103, calibrate the individual detection boxes in the detection feature map based on the segmentation feature map to obtain a calibrated sequence of individual detection boxes; In this embodiment of the application, the step of calibrating the individual detection boxes in the detection feature map based on the segmentation feature map to obtain a calibrated sequence of individual detection boxes includes: S31, perform pixel-level classification on the segmentation feature map, extract the pixel region corresponding to the pedestrian category, and generate a crowd region mask; By setting a threshold (usually 0.5), the pixel regions corresponding to the pedestrian categories are extracted, and a binary crowd region mask M(x,y)∈{0,1} is generated, where 1 indicates that the pixel belongs to the crowd region and 0 indicates that it belongs to the background region.

[0038] S32, Multiply the crowd area mask and the detection feature map pixel by pixel to generate a filtered detection feature map; Background false detection boxes located outside the mask of the crowd area have a corresponding feature map region pixel value of 0. After multiplication, the feature response is directly suppressed to zero, thereby filtering out background interference.

[0039] S33, Perform connected component analysis on the filtered detection feature map, and extract the bounding rectangles of each connected component as a preliminary calibration detection box set; S34, Perform connected component analysis on the crowd region mask, and extract the bounding rectangles of each connected component as the candidate crowd region set; S35, perform intersection-union matching between the preliminary calibration detection box set and the candidate group region set, and determine the areas where there are candidate group regions but are not covered by the preliminary calibration detection boxes as target occluded areas and generate supplementary detection boxes. The initial calibration detection box set Bi is matched with the candidate group region set Rc by the intersection-union ratio (IoU). For regions that exist in the candidate group region Rc but are not covered by the initial calibration detection box Bi, they are identified as target occluded regions and supplementary detection boxes are generated.

[0040] Specifically, generating supplementary detection boxes includes: For candidate group regions determined to be occluded by the target, extract the center coordinates and size information of the bounding rectangle of the candidate group region; The trajectory information of the target corresponding to the candidate group region in adjacent historical frames is obtained, and the trajectory information includes the position coordinates and velocity vector of the target in the historical frames. Based on the trajectory information and the preset least squares fitting linear motion model, the average velocity vector of the target in the current frame is determined; Based on the average velocity vector and the time interval between the current frame and the previous frame, the predicted position coordinates of the supplementary detection box are calculated using a motion extrapolation formula. The supplementary detection box is generated based on the predicted location coordinates and the size information.

[0041] S36, merge the preliminary calibration detection box with the supplementary detection box to generate a calibrated individual detection box sequence.

[0042] A temporary confidence score confsupp = conflast × exp(-λ × tocc) is assigned to the supplementary detection boxes, where conflast is the detection confidence score of the last frame before occlusion, tocc is the number of consecutive occlusion frames, and λ is a decay coefficient with a value of 0.2. Finally, the preliminary calibration detection boxes and the supplementary detection boxes are merged, and redundant overlapping boxes are removed by non-maximum suppression (NMS) to output the calibrated individual detection box sequence.

[0043] S104, Based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, the position and velocity state of the target individual in the current frame are smoothly estimated to generate a short-term smooth trajectory; In this embodiment, the time-series filtering algorithm refers to an algorithm that uses the statistical characteristics of time-series data to smooth the observed values. In this embodiment, the Kalman filtering algorithm is used.

[0044] Specifically, based on the calibrated individual detection box sequence and a preset temporal filtering algorithm, the position and velocity state of the target individual in the current frame are smoothly estimated to generate a short-term smooth trajectory, including: Based on the calibrated individual detection box sequence, the center coordinates of each target individual detection box in each frame are extracted as the observation z. k =[xdet,ydet] T Construct a state vector x that includes position and velocity.k =[x,y,vx,vy] T ; The state transition equation x is established based on the pre-defined uniform motion model. k∣k-1 =A xk-1∣k-1 The state transition matrix A, under the assumption of uniform motion, contains a linear relationship between position and velocity, meaning the current position equals the previous position plus the product of velocity and time interval Δt. Using the state vector from the previous moment and the state transition matrix, a priori prediction of the target individual's state vector in the current frame is performed, yielding the predicted state vector x. k|k-1 and prediction error covariance matrix P k∣k-1 =AP k-1∣k-1 A T +Q, where Q is the process noise covariance matrix, used to model the uncertainty of the target motion model (such as a pedestrian suddenly turning or accelerating / decelerating). The Kalman gain is calculated based on the prediction error covariance matrix and the observation noise covariance matrix R; K k =P k|k- 1HT(HP k|k-1 H T +R) -1 , where H is the observation matrix used to map the state space to the observation space, and R reflects the uncertainty of the detector output. The Kalman gain essentially determines the weight ratio of the predicted and observed values ​​in the final estimate.

[0045] Based on the Kalman gain and the observations of the current frame, the predicted state vector is corrected, x k|k =x k|k-1 +K k (z k -Hx k|k-1 ), where (z k -Hx k|k-1 The error covariance matrix P is called the observation residual or innovation, representing the difference between the actual observation and the prediction. It is used to obtain the posterior position and posterior velocity estimates of the target individual in the current frame, and to update the error covariance matrix P. k|k =(IK k H)P k|k-1 This prepares for prediction in the next frame.

[0046] The posterior position estimates of multiple consecutive frames are sequentially concatenated to generate the short-term smooth trajectory. This short-term smooth trajectory preserves the true motion trend of the target while suppressing detection noise, providing high-quality input data for subsequent long-term trajectory prediction.

[0047] S105, Construct an input sequence based on the short-term smooth trajectory and input it into a pre-trained long short-term memory network to generate a long-term predicted trajectory; In this embodiment, Long Short-Term Memory (LSTM) is a special recurrent neural network structure that effectively solves the gradient vanishing problem in long sequence modeling through the gating mechanism of forget gate, input gate and output gate, and can learn and memorize the motion pattern of the target over a long time span.

[0048] Specifically, an input sequence is constructed based on the short-term smooth trajectory and input into a pre-trained long short-term memory network to generate a long-term predicted trajectory, including: Based on a preset sliding window, the trajectory points of a set number of frames are determined and constitute the input sequence, where each trajectory point includes the position coordinates of the target in the image coordinate system, and the set number of frames is usually 10-20 frames. The input sequence is fed step by step into the forget gate, input gate and output gate of the pre-trained long short-term memory network to determine the hidden state at the current time. Let the short-term smoothed trajectory point sequence output by the Kalman filter be P={p t-k+1 ,p t-k+2 ,...,p t For the current time t, the trajectory points of the most recent k frames are taken to form the input sequence X. t =[p t-k+1 ,p -k+2 ,...,p t The input sequence is fed into a pre-trained Long Short-Term Memory network step by step. At each time step t, the coordinates x of the current trajectory point are input. t and the hidden state h from the previous time step t-1 After passing through the Gate of Oblivion t =σ(Wf·[h t-1 ,x t ]+b f The cell state c at the previous moment is determined. t-1 Which information should be retained or discarded? Enter gate i t =σ(W i ·[h t-1 ,x t ]+b i and candidate cell status: ; Together, they decide which new information from the current input needs to be stored in the memory unit, and the unit state is updated as follows: ; To achieve selective fusion of historical and current information, output gate o t =σ(W o ·[h t-1 ,x t ]+b oThe hidden state output h at the current moment is determined based on the updated cell state. t =o t ⊙tanh(c t The LSTM determines the hidden state at the current moment. Through this gating structure, the LSTM can adaptively retain historical motion features (such as motion direction, velocity, and acceleration trends) that are useful for predicting future trajectories, while forgetting noise information that is irrelevant to the current motion pattern.

[0049] The hidden state of the last time step is input into the fully connected layer, and the coordinates Y of the predicted trajectory point are determined through a linear transformation. t =[p t+1 ,p t+2 ,...,p t+N The long-term predicted trajectory is obtained. As time progresses, the sliding window moves forward, discarding the oldest trajectory points and adding the latest Kalman filter estimates, forming a continuous predicted output stream.

[0050] S106, Based on a preset fusion strategy, the long-term predicted trajectory and the short-term smoothed trajectory are fused to determine the trajectory judgment result of the target population.

[0051] The fusion strategy refers to an adaptive fusion method that dynamically adjusts the weights of short-term smoothed trajectories and long-term predicted trajectories in the final result based on the degree of occlusion of the target.

[0052] In this embodiment of the application, based on a preset fusion strategy, the long-term predicted trajectory and the short-term smoothed trajectory are fused to determine the trajectory judgment result of the target population, including: Based on the preset occlusion probability calculation formula, calculate the occlusion probability of the target in the current frame. Based on the occlusion probability, the fusion weight coefficients are determined; The calculation formula is p t =1-(α·C seg +β·C det ), where α and β are preset weight coefficients and satisfy α+β=1 (for example, at pedestrian-intensive intersections, α=0.6 and β=0.4 can be set to emphasize the sensitivity of semantic segmentation to occlusion), C seg C represents the average confidence score of the pixels in the segmented region corresponding to the target (if the target is occluded, the confidence score of the corresponding segmented region will decrease significantly as the visible portion of the target is reduced). det Cdet represents the percentage of frames in the most recent K consecutive frames where the detection box was successfully matched (the Cdet value decreases if the target is occluded for multiple consecutive frames and the detector cannot match a valid box). The lower the segmentation confidence and the worse the detection continuity, the higher the occlusion probability.

[0053] Then based on the occlusion probability p tDetermine the fusion weight coefficient α t The calculation formula is α t =σ(γ·(p t -p0)), where σ is the Sigmoid function, γ is the scaling factor (e.g., 10), and p0 is the preset occlusion probability threshold (e.g., 0.5). When p t When p0 is below, α t Approaching 0, it places more trust in the short-term predictions of the Kalman filter, which rely on observation smoothing; when p t When p0 is higher, α t Approaching 1, it places greater trust in the long-term predictions of LSTM, which rely on historical learning patterns. The non-linear nature of the Sigmoid function ensures that the weights can transition smoothly when the occlusion state changes, avoiding trajectory jitter caused by weight jumps.

[0054] The short-term smooth trajectory prediction value PKF(t+τ) and the long-term predicted trajectory value PLSTM(t+τ) are weighted and fused according to the fusion weight coefficient to obtain the fused trajectory point P. fusion(t+τ) =(1-α t )·PKF(t+τ)+α t ·PLSTM(t+τ). This formula embodies the core idea that "the higher the degree of occlusion, the more dependent it is on long-term prediction." When the target is severely occluded (α... t When approaching 1), the fusion result is mainly dominated by the predicted trajectory of LSTM to ensure the continuity of the trajectory. When the target is clearly visible (α), t When the value is close to 0, the fusion result is mainly dominated by the smooth estimation of Kalman filtering to ensure the trajectory responds quickly to the real motion.

[0055] Finally, the fused trajectory points are smoothed using a Savitzky-Golay filter. A sliding window polynomial fit is applied to the fused trajectory points across M consecutive frames to eliminate minor high-frequency fluctuations that might be introduced by weighted fusion. The Savitzky-Golay filter achieves smoothing through local least squares fitting, effectively filtering out high-frequency noise while preserving the trajectory trend. Unlike simple moving averages, it avoids over-smoothing leading to peak attenuation and better maintains the trajectory's turning characteristics when the target suddenly accelerates. The smoothed fused trajectory points are then output as the target crowd trajectory judgment result.

[0056] In summary, this embodiment acquires continuous video frame images containing crowds from UAVs, inputs them in parallel into a target detection network and a semantic segmentation network to obtain detection feature maps and segmentation feature maps, respectively. The segmentation feature maps are used to calibrate the individual detection boxes in the detection feature maps, generating a calibrated sequence of individual detection boxes. A short-term smooth trajectory is generated based on Kalman filtering, and this short-term smooth trajectory is input into an LSTM network to generate a long-term predicted trajectory. Finally, an adaptive fusion strategy driven by occlusion probability is used to fuse the short-term smooth trajectory and the long-term predicted trajectory to determine the final target crowd trajectory. This method effectively solves the trajectory interruption problem caused by target occlusion in complex intersection scenarios through closed-loop collaborative optimization among detection, segmentation, tracking, and prediction modules, balancing response speed and prediction stability, and significantly improving the UAV's ability to determine crowd trajectories in dynamic and complex environments.

[0057] Figure 2 A schematic diagram of a drone crowd trajectory determination device 200 based on YOLO and semantic segmentation is shown.

[0058] like Figure 2 As shown, a drone crowd trajectory determination device 200 based on YOLO and semantic segmentation mainly includes: Image acquisition module 201 is used to acquire image information containing crowds collected by the UAV in the target area, the image information including multiple consecutive video frame images; Parallel detection and segmentation module 202 is used to input the image information into a pre-trained target detection network and a semantic segmentation network respectively to obtain a detection feature map and a segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. The calibration module 203 is used to calibrate the individual detection boxes in the detection feature map based on the segmentation feature map to obtain a calibrated sequence of individual detection boxes; The short-term trajectory generation module 204 is used to smoothly estimate the position and velocity state of the target individual in the current frame based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, and generate a short-term smooth trajectory. The long-term trajectory prediction module 205 is used to construct an input sequence based on the short-term smooth trajectory and input it into a pre-trained long short-term memory network to generate a long-term predicted trajectory. The trajectory fusion module 206 is used to fuse the long-term predicted trajectory and the short-term smoothed trajectory based on a preset fusion strategy to determine the trajectory judgment result of the target population.

[0059] In one example, the module in any of the above devices may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0060] For example, when modules in a device can be implemented via a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0061] In this application, various objects such as messages / information / devices / network elements / systems / apparatus / actions / operations / processes / concepts may be named. It is understood that these specific names do not constitute a limitation on the relevant objects. The names may be changed depending on the scenario, context, or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from their functions and technical effects embodied / performed in the technical solution.

[0062] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0063] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0064] Figure 3 This is a structural block diagram of an electronic device 300 according to an embodiment of this application.

[0065] like Figure 3As shown, the electronic device 300 includes a processor 301 and a memory 302, and may further include one or more of an information input / output (I / O) interface 303, a communication component 304, and a communication bus 305.

[0066] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps in the YOLO-based and semantic segmentation-based drone crowd trajectory determination method described above. The memory 302 stores various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0067] I / O interface 303 provides an interface between processor 301 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 304 is used to test wired or wireless communication between electronic device 300 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 304 may include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0068] The communication bus 305 may include a path for transmitting information between the aforementioned components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0069] The electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the UAV crowd trajectory determination method based on YOLO and semantic segmentation given in the above embodiments.

[0070] The following describes the computer-readable storage medium provided in the embodiments of this application. The computer-readable storage medium described below can be referred to in correspondence with the UAV crowd trajectory judgment method based on YOLO and semantic segmentation described above.

[0071] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for determining the trajectory of drone crowds based on YOLO and semantic segmentation.

[0072] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0074] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A method for determining the trajectory of a drone crowd based on YOLO and semantic segmentation, characterized in that, include: Acquire image information containing crowds collected by the drone in the target area, the image information including multiple consecutive video frames; The image information is input into a pre-trained target detection network and a semantic segmentation network to obtain a detection feature map and a segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. Based on the segmentation feature map, the individual detection boxes in the detection feature map are calibrated to obtain a calibrated sequence of individual detection boxes; Based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, the position and velocity state of the target individual in the current frame are smoothly estimated to generate a short-term smooth trajectory. An input sequence is constructed based on the short-term smooth trajectory and input into a pre-trained long short-term memory network to generate a long-term predicted trajectory. Based on a preset fusion strategy, the long-term predicted trajectory and the short-term smoothed trajectory are fused to determine the trajectory judgment result of the target population.

2. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The acquisition of image information containing crowds collected by the drone in the target area includes: Acquire initial continuous video frame images containing crowds collected by the drone in the target area; For each frame of the original image, a distortion mapping table is established based on the UAV camera calibration parameters. The pixel coordinates of the image are remapped through bilinear interpolation to obtain the distortion-corrected image. For each frame of the original image, based on the distortion-corrected image, the images adjacent to the distortion-corrected image, and the lightweight deblurring network of the convolutional neural network, the blur kernel of the intermediate frame is estimated, and the clear image is recovered through non-blind deconvolution to obtain the deblurred image; For each frame of the original image, the deblurred image is converted from the RGB color space to the HSV color space, and the luminance component is subjected to adaptive histogram equalization (CLAHE) to obtain the enhanced image. For each frame of the original image, the enhanced image is subjected to a reverse rotation transformation based on a pre-constructed rotation matrix to obtain a geometrically corrected image; For each frame of the original image, the region of interest (ROI) is extracted and its size is normalized in the geometrically corrected image to obtain the preprocessed image information.

3. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The object detection network is built based on an improved YOLOv8 network. This improved YOLOv8 network includes a head network, which is optimized using a composite loss function. This composite loss function is obtained by weighting the classification loss and the bounding box regression loss. The bounding box regression loss is the generalized intersection-union (GIoU) ​​loss, and the classification loss is the focal loss. The formula for calculating the Generalized Intersection over Union (GIoU) ​​loss is as follows: L GIoU =1-IoU+(|C-(B p ∪B gt )|) / |C|; Among them, B p For the prediction box, B gt Let C be the ground truth bounding box, and let C be the smallest bounding rectangle that contains both the predicted bounding box and the ground truth bounding box. The formula for calculating the classification loss, or Focal Loss, is as follows: L FL =-α×(1-p t )γ×log(p t ); Where, p t Let α be the model's predicted probability of the true class, γ be the balancing factor, and γ be the focusing parameter.

4. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The step of calibrating the individual detection boxes in the detection feature map based on the segmentation feature map to obtain a sequence of calibrated individual detection boxes includes: The segmentation feature map is classified at the pixel level, and the pixel regions corresponding to the pedestrian categories are extracted to generate a crowd region mask; The crowd region mask is multiplied pixel by pixel with the detection feature map to generate a filtered detection feature map; Connectivity analysis is performed on the filtered detection feature map, and the bounding rectangles of each connected region are extracted as a preliminary calibration detection box set; Perform connected component analysis on the crowd region mask and extract the bounding rectangles of each connected component as the candidate crowd region set; The preliminary calibration detection box set and the candidate group region set are subjected to intersection-union matching. The areas where there are candidate group regions but are not covered by the preliminary calibration detection boxes are determined as target occluded areas and supplementary detection boxes are generated. The preliminary calibration detection box is merged with the supplementary detection box to generate a calibrated individual detection box sequence.

5. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 4, characterized in that, The generation of supplementary detection boxes includes: For candidate group regions determined to be occluded by the target, extract the center coordinates and size information of the bounding rectangle of the candidate group region; The trajectory information of the target corresponding to the candidate group region in adjacent historical frames is obtained, and the trajectory information includes the position coordinates and velocity vector of the target in the historical frames. Based on the trajectory information and the preset least squares fitting linear motion model, the average velocity vector of the target in the current frame is determined; Based on the average velocity vector and the time interval between the current frame and the previous frame, the predicted position coordinates of the supplementary detection box are calculated using a motion extrapolation formula. The supplementary detection box is generated based on the predicted location coordinates and the size information.

6. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The step of smoothly estimating the position and velocity state of the target individual in the current frame based on the calibrated individual detection box sequence and a preset temporal filtering algorithm to generate a short-term smooth trajectory includes: Based on the calibrated individual detection box sequence, the center coordinates of each target individual detection box in each frame are extracted as observations to construct a state vector containing position and velocity. A state transition equation is established based on a pre-defined uniform motion model. The state vector of the target individual in the current frame is predicted a priori using the state vector of the previous time step and the state transition matrix at the current time step, so as to obtain the predicted state vector and the prediction error covariance matrix. Calculate the Kalman gain based on the prediction error covariance matrix and the observation noise covariance matrix; Based on the Kalman gain and the observations of the current frame, the predicted state vector is corrected to obtain the posterior position estimate and posterior velocity estimate of the target individual in the current frame. The posterior position estimates of multiple consecutive frames are sequentially concatenated to generate the short-term smooth trajectory.

7. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The step of constructing an input sequence based on the short-term smooth trajectory and inputting it into a pre-trained long short-term memory network to generate a long-term predicted trajectory includes: Based on a preset sliding window, a set number of trajectory points are determined and an input sequence is formed, where each trajectory point includes the position coordinates of the target in the image coordinate system; The input sequence is fed step by step into the forget gate, input gate and output gate of the pre-trained long short-term memory network to determine the hidden state at the current time. The hidden state of the last time step is input into the fully connected layer, and the coordinates of the predicted trajectory points are determined by linear transformation to obtain the long-term predicted trajectory.

8. The method for determining drone crowd trajectories based on YOLO and semantic segmentation according to claim 1, characterized in that, The method based on a preset fusion strategy fuses the long-term predicted trajectory and the short-term smoothed trajectory to determine the trajectory judgment result of the target population, including: Based on the preset occlusion probability calculation formula, calculate the occlusion probability of the target in the current frame. Based on the occlusion probability, the fusion weight coefficients are determined; The short-term smooth trajectory prediction value and the long-term predicted trajectory value are weighted and fused according to the fusion weight coefficient to obtain the fused trajectory point; The fused trajectory points are smoothed, and the smoothed fused trajectory points are used as the trajectory judgment result of the target population.

9. A drone crowd trajectory determination device based on YOLO and semantic segmentation, characterized in that, include: The image acquisition module is used to acquire image information containing crowds collected by the UAV in the target area, and the image information includes multiple consecutive video frame images; The parallel detection and segmentation module is used to input the image information into a pre-trained target detection network and a semantic segmentation network respectively to obtain a detection feature map and a segmentation feature map. The detection feature map includes the position, category and confidence of multiple individual detection boxes, and the segmentation feature map includes the pixel-level category probability distribution of multiple crowd regions. The calibration module is used to calibrate the individual detection boxes in the detection feature map based on the segmentation feature map, so as to obtain a sequence of calibrated individual detection boxes; The short-term trajectory generation module is used to smoothly estimate the position and velocity state of the target individual in the current frame based on the calibrated individual detection box sequence and the preset temporal filtering algorithm, and generate a short-term smooth trajectory. The long-term trajectory prediction module is used to construct an input sequence based on the short-term smooth trajectory and input it into a pre-trained long short-term memory network to generate a long-term predicted trajectory. The trajectory fusion module is used to fuse the long-term predicted trajectory and the short-term smoothed trajectory based on a preset fusion strategy to determine the trajectory judgment result of the target population.

10. An electronic device, characterized in that, Includes a processor, which is coupled to a memory; The processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-8.

Images