A method for tracking pedestrians in scenic spots using drones based on deep learning

Abstract

The present invention discloses a method for tracking pedestrians in scenic spots by drones based on deep learning, including: performing background subtraction on pedestrian images taken by drone cameras to obtain pedestrian images after background subtraction, inputting them into an improved convolutional neural network, using feature grouping and semantic information extraction to segment pedestrian motion scenes, and obtaining segmented pedestrian images; using a multi-template matching method to detect pedestrian targets by sliding predefined pedestrian templates and calculating the similarity between the templates and image blocks; performing target tracking based on a second metric, using deep sorting technology and minimum cosine distance metrics to match bounding boxes with tracks, and performing continuous tracking. The present invention improves background modeling and pedestrian target tracking through deep learning, improves background subtraction efficiency, pedestrian background prominence and feature enhancement, and the accuracy and robustness of object tracking in real-time video streams, especially for tracking multiple pedestrian objects that are close or blocked.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) pedestrian tracking, and in particular to a method for tracking pedestrians by unmanned aerial vehicles in scenic spots based on deep learning. Background Art

[0002] With the continuous progress of science and technology, especially the rapid development of computer technology and digital signal processing technology, machine vision technology, as an important branch, has been widely concerned and applied. Machine vision technology obtains video image information through image acquisition devices such as cameras, uses the powerful computing power of computers to convert these image information into digital signals, and conducts in-depth analysis and processing. The emergence of this technology has greatly improved people's ability to understand and utilize image information, especially in the field of pedestrian tracking, showing great potential.

[0003] As an important research direction in the field of machine vision, pedestrian tracking aims to accurately determine whether there are pedestrians in the video by automatically analyzing the video images captured by the camera, and to mark and track them. The application of this technology can not only help us better understand and analyze the behavior patterns of pedestrians, but also provide strong support for various practical application scenarios. However, although many experts and scholars have conducted in-depth research on pedestrian tracking technology and achieved a series of important results, it still faces many challenges in practical applications.

[0004] At the same time, the rapid development of drone technology has provided new possibilities for pedestrian tracking. More and more scenic spots have begun to introduce drones, using their unique aerial perspective and high-definition cameras to provide tourists with unprecedented visual experiences. Drones can not only capture scenic spots far away from tourist traffic areas, but also display the planning layout and architectural features of scenic spots from the air, providing tourists with a more comprehensive and in-depth understanding. In addition, drones can also track specific pedestrians according to the needs of tourists, providing tourists with more personalized and considerate services.

[0005] However, despite the huge potential of drones for pedestrian tracking in scenic spots, there are still some technical problems that need to be solved. One of the most prominent problems is the difficulty in tracking obscured pedestrian objects. In scenic spots, pedestrians are often obscured by obstacles such as buildings and trees, which brings great difficulties to drone pedestrian tracking. In addition, when multiple pedestrians are very close, it is difficult for drones to accurately distinguish and track each pedestrian object. These problems limit the actual application effect of drones in pedestrian tracking in scenic spots.

[0006] Therefore, how to combine machine vision technology with drone technology to develop a drone pedestrian tracking method that can accurately and stably track pedestrians in scenic areas has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of this, the present invention proposes a scenic area drone pedestrian tracking method based on deep learning to solve the problem in the prior art that scenic area drones are difficult to effectively handle obscured pedestrians and accurately distinguish multiple objects at close range when tracking pedestrians.

[0008] The specific technical solutions of the present invention are as follows:

[0009] A method for tracking pedestrians in scenic spots by drones based on deep learning, comprising:

[0010] Step 1: Use a deep learning algorithm to perform background subtraction on the pedestrian image taken by the drone camera, and obtain the pedestrian image after background subtraction by constructing an image representation and extracting foreground and background masks;

[0011] Step 2: Input the pedestrian image after background subtraction into the improved convolutional neural network, and use feature grouping and semantic information extraction to segment the pedestrian motion scene and obtain the segmented pedestrian image;

[0012] Step 3: on the segmented pedestrian image, a multi-template matching method is used to detect pedestrian targets by sliding a predefined pedestrian template and calculating the similarity between the template and the image block;

[0013] Step 4: For the detected pedestrian target, target tracking is performed based on the second metric, and the depth sorting technique and the minimum cosine distance metric are used to achieve accurate matching between the bounding box and the track, thereby performing continuous tracking;

[0014] Step 5: Output the pedestrian target tracking results based on deep learning.

[0015] Specifically, step 1 also includes: representing the image taken by the drone camera as the best background model of a single image, subtracting the binary frame that existed before the current frame, and applying a threshold measurement to study the motion between adjacent frames, thereby obtaining a binary image containing the position of the moving pedestrian target.

[0016] Specifically, step 1 also includes: based on a deep learning algorithm, calculating an image representation for each frame in the input image video, and arranging the original image video frame and the calculated image representation together to form a sequence as input to the background modeling module; the background modeling module extracts foreground and background masks, and outputs the final background map by fusing the mask calculated based on the original video with the mask generated by the frame video, and multiplying the mask of the original video with the mask of the image representation; based on the deep learning background subtraction method, a pixel is regarded as the foreground if and only if the output mask calculated from the original video detects a pixel as the foreground.

[0017] Specifically, in step 2, the improved convolutional neural network uses random initialization to assign weights and obtains rich semantic information through feature grouping of different grouping sizes.

[0018] Specifically, in step 2, the improved convolutional neural network combines the RGB and depth features by addition and groups them to make full use of multimodal information; after the feature grouping by addition, each branch is fused through a connection operation to obtain the feature vector of the convolutional neural network, and then subsequent pooling and full connection operations are performed.

[0019] Specifically, in step 2, in the process of pedestrian motion scene segmentation, an attention mechanism channel is introduced to further enhance important features and suppress irrelevant information; the improved convolutional neural network uses shape activation function to perform nonlinear transformation on features to improve the accuracy of segmentation results.

[0020] Specifically, in step 3, a correlation method is used to find the correlation between the original image and the pedestrian template list. When the correlation is greater than a preset threshold, it is determined that the pedestrian template is detected. The correlation calculation of the pedestrian template is based on the average grayscale of the grayscale template image, the grayscale source image, and the average grayscale of the source image. Matching is performed by comparing the grayscale difference between the template image and the source image block.

[0021] Specifically, in step 4, a two-stage matching cascade is adopted to assign detected bounding boxes to tracks. The first stage uses a deep sorting technique based on motion and appearance metrics to match valid tracks, and the second stage connects unpaired tracks with the most recently established temporary tracks.

[0022] Specifically, the second stage matching process of step 4 uses the minimum cosine distance as the second metric to measure the appearance feature distance between each track and each measurement.

[0023] Specifically, step 4 quantifies the similarity between the detected bounding box and the predicted bounding box by calculating the ratio of twice the intersection to the sum of the set sizes to form a linear assignment problem; an improved Hungarian algorithm is used to solve the minimization problem formed by the cost matrix between the detected bounding box and the predicted bounding box to minimize the second metric; a cosine metric formula is used to calculate the appearance feature similarity between the detected bounding box and the predicted bounding box; an overlap metric formula is used to calculate the overlap between the detected bounding box and the predicted bounding box, and the Euclidean distance metric is obtained by calculating the overlap metric; a specific threshold is set to determine the pedestrian target between each two frames of video images according to the Euclidean distance metric for tracking; based on deep learning technology, the limitations of tracking multiple pedestrian objects that are close to or occluded by each other are solved by associating the detection of the same object across frames.

[0024] The beneficial effects of the present invention are:

[0025] (1) This paper innovates the background modeling module and optimizes the background subtraction process through deep learning technology, so that the video captured by the drone can be more accurately converted into image representation, thereby improving the recognition accuracy of foreground and background. In addition, we also use a double confirmation mechanism to determine the foreground attribute of pixels. Pixels are considered as foreground if and only if the output mask calculated from the original video detects them as foreground, which significantly improves the efficiency and accuracy of background subtraction.

[0026] (2) The present invention also introduces a convolutional neural network algorithm to finely process the original features. Through rotation and addition operations, the background of the pedestrian is effectively highlighted in the feature map, and the edge and internal information are also significantly enhanced. This not only improves the recognition and positioning accuracy of pedestrian targets, but also greatly improves the effect of semantic segmentation of images.

[0027] (3) Finally, the present invention uses an advanced deep learning-based pedestrian tracking algorithm to successfully overcome the technical difficulties in tracking multiple pedestrian objects that are close or occluded by correlating detections of the same object across frames. The algorithm is able to accurately track pedestrians that are very close to each other and occluded, and can even re-identify their trajectories after long periods of occlusion. This innovation makes pedestrian tracking more accurate and robust, bringing revolutionary progress to pedestrian tracking technology in scenic drone applications.

[0028] In summary, the present invention improves background modeling and pedestrian target tracking through deep learning, thereby improving the efficiency of background subtraction, pedestrian background highlighting and feature enhancement, as well as the accuracy and robustness of object tracking in real-time video streams, especially for the tracking of multiple pedestrian objects that are close or occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 The figure is a flow chart of the method for tracking pedestrians by drones in scenic spots based on deep learning according to the present invention. DETAILED DESCRIPTION

[0031] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The present invention proposes a method for tracking pedestrians in a scenic area using a drone based on deep learning. The method first performs background subtraction based on deep learning on the video images collected by the camera of the scenic area drone to effectively distinguish foreground and background pedestrian targets. Subsequently, an improved convolutional neural network is used to perform motion scene segmentation on the image after background subtraction to accurately extract the scene area where the pedestrians are located. Next, a multi-template matching method is used to detect pedestrian targets, and accurate recognition of pedestrian targets is achieved by calculating the similarity between the template and the image block. Finally, the detected pedestrian targets are tracked based on the second metric, and continuous and stable pedestrian target tracking is achieved by calculating the minimum cosine distance between the bounding box and the track. The present invention improves the accuracy and robustness of object tracking in real-time video streams, and provides an efficient and accurate solution for tracking pedestrian targets by drones in scenic areas. The method for tracking pedestrians in a scenic area using a drone based on deep learning of the present invention comprises the following steps:

[0033] Step 1: Perform background subtraction based on deep learning on the pedestrian images taken by the scenic area drone camera:

[0034] In this step, the goal is to accurately identify and remove the background in the drone-captured image so that only pedestrian targets are retained. Since drone-captured videos often contain continuously changing scenes, a method that can handle such dynamic changes is needed.

[0035] First, it is necessary to recognize that background modeling based on a single image presents challenges in distinguishing foreground and background pedestrians. To address this problem, this application introduces the concept of motion detection. By analyzing the changes between consecutive frames, the movement of pedestrians is effectively captured. Specifically, by subtracting the current frame from its previous binary frame and applying a threshold measurement to determine which pixel changes are caused by pedestrian motion. This method is able to generate a binary image in which the white area represents the location of the moving pedestrian and the black area represents the background. The specific formula can be expressed as:

[0036]

[0037] Among them, M i is a binary image containing the position of the moving pedestrian target, i is a pixel in the image, is the input image at time t, is the image frame at time tN, and thr is the threshold used to distinguish moving pedestrians from the background.

[0038] Next, deep learning technology is used to further optimize the background subtraction process. This application designs a deep learning model that can receive the original image video frame and the calculated image representation as input, and output the foreground and background masks. These masks are used to identify whether each pixel in the image belongs to the foreground or the background. Through the deep learning model, this application can automatically learn and extract complex background features, thereby separating the foreground and background more accurately.

[0039] Specifically, a background subtraction algorithm based on deep learning is used to model the background of the image representation taken by the drone. First, the image representation M is calculated for each frame in the input image video according to the formula i Then, the original image video frame and the calculated image representation M i Arranged together, this sequence is passed as input to the background modeling module, which extracts foreground and background masks.

[0040] In order to further improve the accuracy of background subtraction, this application fuses the mask calculated based on the original video with the mask generated by the frame video. This fusion operation takes into account the mask information from different sources, enabling a more comprehensive understanding of the foreground and background in the image. i The final background subtraction result is obtained by multiplying the mask.

[0041] Finally, we get pedestrian images after background subtraction. In these images, the background is effectively removed and only the pedestrian targets are retained. This processing provides high-quality input data for subsequent target detection and tracking.

[0042] Compared with traditional background modeling methods, the background subtraction algorithm based on deep learning has higher efficiency and accuracy. By modeling the video captured by the drone as an image representation, it can better handle dynamically changing scenes and correct the errors of the segmentation and drawing modules in the background modeling module. A pixel is only considered as the foreground when the output mask calculated from the original video detects it as the foreground. This strategy greatly improves the efficiency and accuracy of background subtraction, making it better able to cope with complex scenes captured by drone cameras.

[0043] Step 2: Use the improved convolutional neural network to segment pedestrian motion scenes: This step uses the convolutional neural network algorithm to rotate and add the valuable information in the original features, making the background of pedestrians more prominent. This process can enhance the edge and internal information in the feature map, and also enhance the effect of semantic segmentation of the image. The following is a description of the specific steps:

[0044] First, it is known that different channels in the convolutional neural network correspond to different local features. However, when the entire input feature map is processed as a whole, it is easy to cause the loss of pedestrian motion scene information. In order to solve this problem, this application introduces a feature grouping strategy. Through feature grouping, the input feature map can be divided into multiple groups, each of which focuses on extracting a specific type of feature. This can not only reduce the loss of information, but also improve the efficiency of feature extraction.

[0045] In the improved convolutional neural network, the present application pays special attention to the initialization of weights. Since the weights of different connections in the convolutional neural network layer are jointly determined by initialization and back propagation, unlike traditional methods, the present application uses random initialization to assign weights. This initialization method helps to break the symmetry of the network and promote differential learning between different neurons. In addition, the present application constructs a pyramid-structured convolutional neural network in which feature groupings of different grouping sizes can capture rich semantic information. This structure enables the network to more comprehensively understand the complexity and diversity of pedestrian motion scenes.

[0046] For each layer, the RGB and depth features are first merged by addition. The merged features contain both color information and depth information, providing richer clues for subsequent segmentation tasks. Next, the merged features are divided into several groups, and independent convolution operations are performed in each group. In this way, each group can focus on extracting a specific type of feature, thereby enhancing the network's ability to recognize pedestrian targets. That is, for a layer, the RGB and depth features are first merged by addition and then divided into (k+1) groups. The specific formula can be expressed as

[0047]

[0048] After group convolution, feature representations of multiple branches are obtained. In order to merge these features, a concatenation operation is used. Specifically, the features of each branch are concatenated to form a more comprehensive feature vector. This feature vector not only contains the feature information extracted from different groups, but also incorporates the feature information of the original input. In other words, after addition merging, each branch is connected, and the specific formula can be expressed as F i =R i +D i , where R i and D i is the feature vector of the convolutional neural network, F i is the output mapping of the deep modality. Then, the features obtained from the convolutional neural network are finally fed into the pooling layer and the fully connected layer. The specific formula can be expressed as

[0049] P i =Cat(AM(A F，1 ), AM(A V，2 ), ..., AM(A F，j )),

[0050] Among them, Cat represents the connection operation, AM represents the traditional attention mechanism channel, and P i =Cat(AM(A F，1 ), AM(A F，2 ), ..., AM(A F，j )) represents the features obtained by the convolutional neural network, P i =Cat(AM(A F，1 ), AM(A F，2 ), ..., AM(A F，j )) represents the output mapping of the depth modality, F i The pooling layer is responsible for downsampling the features to reduce the amount of calculation and parameters, while the fully connected layer is responsible for further abstraction and classification of the features.

[0051] To further enhance the effect of feature representation, the input features rotated 180 degrees are added to the original features at the pixel level. This rotation operation helps the network learn features from multiple angles and improves the robustness of the features. Next, the result is multiplied by the sigmoid activation function to obtain the image of the pedestrian motion scene segmentation at the pixel level to highlight the edges and internal information of the pedestrians. The choice of the sigmoid activation function is crucial to retaining important features and suppressing irrelevant information.

[0052] The final result is a pedestrian image after motion scene segmentation.

[0053] By using an improved convolutional neural network for pedestrian motion scene segmentation, pedestrian targets can be more accurately identified and segmented. This improvement not only improves the segmentation accuracy, but also enhances the network's adaptability to complex scenes. After obtaining pedestrian images after motion scene segmentation, other visual processing tasks such as target tracking and behavior analysis can be further performed.

[0054] In general, the core idea of ​​this step is to use the improved convolutional neural network algorithm to extract and fuse the feature information in the pedestrian motion scene to achieve a more accurate segmentation effect. Through a series of operations such as feature grouping, weight initialization, feature fusion and enhancement, and semantic segmentation, an efficient and accurate pedestrian motion scene segmentation model was successfully constructed.

[0055] Step 3: Use the multi-template matching method to detect pedestrian targets on the pedestrian images after the motion scene segmentation in step 2:

[0056] After the pedestrian motion scene segmentation is completed, images focusing on the pedestrian area are obtained. In order to accurately detect pedestrian targets from these images, a multi-template matching method is used.

[0057] First, a set of predefined pedestrian templates are prepared. These templates are made based on a large amount of pedestrian image data, which represent the appearance of pedestrians in different postures, different clothes and different perspectives. These templates are stored in the template list for use in the subsequent matching process.

[0058] Next, the segmented pedestrian image is divided into n × n matrix blocks. The size of these matrix blocks usually matches the size of the template to ensure that the template can fully cover each image block and be compared with the image block.

[0059] On the original image, start sliding the predefined pedestrian template. Whenever the template slides to a new position, it is compared with the image block at the current position. That is, as it slides, the template is distinguished from the image block below it, and then matching is performed by calculating the size similar to the template and the original image block. This comparison process is mainly achieved by calculating the similarity between the template and the image block. Similarity is usually measured based on pixel value, grayscale, texture or other relevant features.

[0060] In order to quantify the similarity between the template and the image patch, the present application uses a correlation method. This method calculates the degree of correlation between the template image and the source image. In this process, the present application considers the grayscale distribution of the template image and the grayscale distribution of the source image. The average grayscale value of the template image and the average grayscale value of the source image are also calculated and used in the calculation of similarity.

[0061] Through the calculation of the correlation method, a correlation score is obtained. If this score exceeds the preset threshold, it is considered that a pedestrian target matching the template is found. This threshold is determined based on experiments and experience to ensure the accuracy and reliability of detection.

[0062] If the correlation score is lower than the threshold, it means that no matching pedestrian template is found at the current position. At this point, we continue to slide the template to the next position and repeat the above matching process.

[0063] By sliding the template across the entire image and calculating the correlation score, all pedestrian targets that match the template can be found. The location and size information of these targets are recorded for subsequent target tracking, behavior analysis, or other visual processing tasks.

[0064] The specific correlation formula can be expressed as:

[0065]

[0066] Among them, x i is the grayscale template image, is the average grayscale of the template image, y i is the grayscale source image, represents the average grayscale of the source image, and N represents the size of the template image.

[0067] Finally, pedestrian targets detected by multi-template matching method are obtained. These targets are accurately identified in the image, providing a reliable basis for subsequent processing and analysis.

[0068] Through the multi-template matching method, the predefined pedestrian templates can be fully utilized to detect pedestrian targets. This method not only takes into account the diversity of pedestrian appearance, but also quantifies the similarity between the template and the image patch through the relevant method, thereby achieving accurate target detection. This provides strong support for pedestrian target tracking, behavior analysis and other related applications.

[0069] Step 4: Track the pedestrian target detected in step 3 based on the second metric:

[0070] The core of pedestrian tracking algorithm is to accurately match the detected pedestrian bounding box with the existing track to achieve continuous tracking. To achieve this goal, a two-stage matching cascade strategy based on the second metric is adopted.

[0071] In the first stage, deep sorting technology is used to match tracks based on the motion and appearance metrics of pedestrians. Deep sorting technology extracts feature representations of targets through a deep neural network model. These features not only contain appearance information such as the shape and texture of the target, but also incorporate dynamic information such as the target's motion trajectory and speed. Based on these features, the similarity score between the detected bounding box and the existing track can be calculated to filter out the best matching track.

[0072] Entering the second stage, we focus on those tracks and detected bounding boxes that were not successfully paired in the first stage. In order to effectively connect these unpaired elements, a second metric based on the minimum cosine distance is introduced. Cosine distance is a measure of the cosine value of the angle between two vectors, which can reflect the directional similarity between vectors. Here, the cosine distance of the appearance feature vector between each unpaired track and the detected bounding box is calculated to quantify the similarity between them.

[0073] In order to more accurately quantify the similarity between bounding boxes, the method of calculating the ratio of twice the intersection and the sum of the set sizes is adopted. The specific formula can be expressed as:

[0074]

[0075] Among them, N is the pedestrian target bounding box detected in step 3; M is the predicted bounding box; D and P are calculated by calculating the cost matrix between each detected bounding box and all predicted bounding boxes, which forms a linear assignment problem.

[0076] This method takes into account the overlapping areas between bounding boxes and their own sizes, so as to more comprehensively evaluate the similarity between them. In this way, the similarity relationship between each detected bounding box and all predicted bounding boxes can be represented as a cost matrix.

[0077] Next, this cost matrix is ​​transformed into a linear assignment problem. To solve this minimization task, the modified Hungarian algorithm is used. This algorithm can efficiently find an assignment scheme that minimizes the total matching cost. In this way, the detected bounding boxes can be accurately assigned to the corresponding tracks, achieving continuous tracking of pedestrian targets.

[0078] That is, to formulate this problem as a solvable minimization task, the second metric between the predicted and detected bounding boxes is formulated as

[0079]

[0080] in, Represents the deviation between the predicted bounding box and the detected bounding box; represents the cost matrix of the i-th detected bounding box; Represents the cost matrix for the j-th predicted bounding box.

[0081] In addition, in order to more comprehensively evaluate the similarity between bounding boxes, the cosine metric and overlap metric are also considered. The cosine metric is used to measure the similarity between appearance feature vectors, while the overlap metric focuses on the degree of spatial overlap between bounding boxes. By combining these two metrics, it is possible to more accurately determine whether the bounding boxes belong to the same pedestrian target.

[0082] The cosine metric between the detected bounding box and the predicted bounding box can be formulated as

[0083]

[0084] The overlap measure between the detected bounding box and the predicted bounding box can be expressed as

[0085]

[0086] This allows us to obtain the Euclidean distance metric between the detected and predicted bounding boxes by calculating the overlap metric,

[0087] Finally, a specific threshold is set to determine the pedestrian target between each two video frames based on the calculated Euclidean distance metric. When the Euclidean distance between two bounding boxes is less than this threshold, they are considered to belong to the same pedestrian target and are tracked continuously.

[0088] In this way, pedestrian target tracking results based on deep learning were obtained.

[0089] This algorithm not only improves the accuracy and robustness of object tracking in real-time video streams, but also effectively handles occlusion and intersection between pedestrians. By using deep learning to associate detections of the same object across frames, it solves the limitation of tracking multiple pedestrian objects that are close to each other or occluded, and can accurately track multiple pedestrian objects that are very close to each other and occluded. It allows us to re-identify trajectories after long-term occlusion, achieving more accurate object tracking. These tracking results provide strong technical support for subsequent applications such as pedestrian behavior analysis and scenic area safety monitoring.

[0090] The beneficial effects of the present invention are:

[0091] First, the present invention innovatively improves the traditional background modeling module by introducing deep learning-based background subtraction technology to convert the video captured by the scenic area drone camera into an image representation. This improvement significantly corrects the errors that may be generated by the segmentation and rendering modules in the background modeling module, ensuring that pixels are only considered as foreground when the output mask consistently identifies them as foreground. This optimization greatly improves the efficiency of background subtraction and provides a more accurate basis for subsequent pedestrian tracking.

[0092] Secondly, the present invention uses a convolutional neural network algorithm to process the original features. Through rotation and addition operations, the background of pedestrians is made more prominent in the feature map. This technology not only enhances the edge and internal information in the feature map, but also significantly improves the effect of semantic segmentation of the image. This improvement helps to more accurately identify and separate pedestrian targets in complex scenic environments.

[0093] Finally, the present invention uses a pedestrian target tracking algorithm based on deep learning, which is an advanced object tracking technology. By correlating detections of the same object across frames using deep learning technology, the present invention successfully addresses the limitations of tracking multiple pedestrian objects that are close to each other or occluded. The algorithm is able to accurately track multiple pedestrian objects that are very close to each other and occluded, and can even re-identify the trajectory after long periods of occlusion. This improvement significantly improves the accuracy and robustness of object tracking in real-time video streams, providing strong support for more accurate object tracking.

[0094] In summary, the present invention achieves more efficient and accurate pedestrian tracking in scenic area drone applications by improving the traditional background modeling module, applying the convolutional neural network algorithm, and adopting the pedestrian target tracking algorithm based on deep learning.

[0095] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for tracking pedestrians by drones in scenic spots based on deep learning, characterized in that: include: Step 1: Use a deep learning algorithm to perform background subtraction on the pedestrian image taken by the drone camera, and obtain the pedestrian image after background subtraction by constructing an image representation and extracting foreground and background masks; Step 2: Input the pedestrian image after background subtraction into the improved convolutional neural network, and use feature grouping and semantic information extraction to segment the pedestrian motion scene and obtain the segmented pedestrian image; Step 3: on the segmented pedestrian image, a multi-template matching method is used to detect pedestrian targets by sliding a predefined pedestrian template and calculating the similarity between the template and the image block; Step 4: For the detected pedestrian target, target tracking is performed based on the second metric, and the depth sorting technique and the minimum cosine distance metric are used to achieve accurate matching between the bounding box and the track, thereby performing continuous tracking; Step 5: Output the pedestrian target tracking result based on deep learning; The step 1 also includes: representing the image captured by the drone camera as the best background model of a single image, using the current frame to subtract the existing binary frame before it, applying a threshold measurement to study the motion between adjacent frames, thereby obtaining a binary image containing the position of the moving pedestrian target; The step 1 further comprises: calculating an image representation for each frame in the input image video based on a deep learning algorithm, and arranging the original image video frame and the calculated image representation together to form a sequence as an input to a background modeling module; the background modeling module extracts foreground and background masks, and outputs a final background image by fusing the mask calculated based on the original video with the mask generated by the frame video, and multiplying the mask of the original video with the mask of the image representation; the background subtraction method based on deep learning regards a pixel as a foreground only if and only if the output mask calculated from the original video detects a pixel as a foreground; Step 4 quantifies the similarity between the detected bounding box and the predicted bounding box by calculating the ratio of twice the intersection to the sum of the set sizes to form a linear assignment problem; uses an improved Hungarian algorithm to solve the minimization problem formed by the cost matrix between the detected bounding box and the predicted bounding box to minimize the second metric; the cosine metric formula is used to calculate the appearance feature similarity between the detected bounding box and the predicted bounding box; the overlap metric formula is used to calculate the overlap between the detected bounding box and the predicted bounding box, and the Euclidean distance metric is obtained by calculating the overlap metric; sets a specific threshold and determines the pedestrian target between each two frames of video images according to the Euclidean distance metric for tracking; based on deep learning technology, the limitations of tracking multiple pedestrian objects that are close to or occluded by each other are solved by associating the detection of the same object across frames.

2. The scenic area drone pedestrian tracking method based on deep learning as claimed in claim 1, characterized in that: In step 2, the improved convolutional neural network uses random initialization to allocate weights, and obtains rich semantic information through feature grouping of different grouping sizes.

3. The scenic area drone pedestrian tracking method based on deep learning as claimed in claim 1, characterized in that: In step 2, the improved convolutional neural network combines the RGB and depth features by addition and groups them for processing to fully utilize multimodal information; after the feature grouping by addition, each branch is fused through a connection operation to obtain a feature vector of the convolutional neural network, and then subsequent pooling and full connection operations are performed.

4. The scenic area drone pedestrian tracking method based on deep learning as claimed in claim 1, characterized in that: In step 2, in the process of pedestrian motion scene segmentation, an attention mechanism channel is introduced to further enhance important features and suppress irrelevant information; the improved convolutional neural network uses a shape activation function to perform nonlinear transformation on the features to improve the accuracy of the segmentation results.

5. The scenic area drone pedestrian tracking method based on deep learning as claimed in claim 1, characterized in that: In step 3, a correlation method is used to find the correlation between the original image and the pedestrian template list. When the correlation is greater than a preset threshold, it is determined that the pedestrian template is detected. The correlation calculation of the pedestrian template is based on the average grayscale of the grayscale template image, the grayscale source image, and the average grayscale of the source image. Matching is performed by comparing the grayscale difference between the template image and the source image block.

6. The scenic area drone pedestrian tracking method based on deep learning as claimed in claim 1, characterized in that: In step 4, a two-stage matching cascade is used to assign detected bounding boxes to tracks. The first stage uses a deep sorting technique based on motion and appearance metrics to match valid tracks, and the second stage connects unpaired tracks with the most recently established temporary tracks.

7. The method for tracking pedestrians by drones in scenic spots based on deep learning as claimed in claim 6, characterized in that: The second stage matching process of step 4 uses the minimum cosine distance as the second metric to measure the appearance feature distance between each track and each measurement.

Images