Method, device, computer equipment and storage medium for detecting and tracking moving targets

Abstract

The present invention discloses a detection and tracking method, device, computer equipment and storage medium for a moving target, including: semantically segmenting continuous frames in a video stream to obtain dynamic feature points and non-dynamic feature points; determining dynamic a priori regions and non-dynamic a priori regions based on dynamic feature points and non-dynamic feature points; determining the change of the drone's posture through the change of the non-dynamic a priori regions in continuous frames, and determining motion compensation information based on the change of the drone's posture; using an adaptive corner point cyclic matching method to perform global deviation verification on dynamic feature points, and eliminating dynamic feature points that do not meet the conditions to obtain target feature matching points; based on the target feature matching points and motion compensation information, using a sparse optical flow method to calculate the optical flow value of the target feature matching points to obtain an optical flow vector image; and performing image processing to obtain a moving target region. The present invention improves the detection and tracking accuracy of targets in drone aerial photography scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a moving target detection and tracking method, device, computer equipment and storage medium. Background Art

[0002] The detection of moving targets is to separate the moving targets from the current image. Currently, the commonly used moving target detection algorithms include background subtraction, optical flow and frame difference. The background subtraction method is to perform a differential operation on the current video frame and the background frame to obtain a dynamic prior region, but it is easily affected by noise and illumination changes, and further morphological processing is required to extract the moving target; the traditional optical flow method calculates the instantaneous speed of pixel movement, and represents the unique movement of the image by analyzing the changes in pixel points, thereby achieving target tracking. This method has high detection accuracy, but the two premise assumptions of the optical flow method limit the scope of application of the optical flow method, and its applicability and real-time performance are poor; the inter-frame difference method is to perform a differential operation on two adjacent frames of images to obtain a moving target, which has the advantages of fast inference speed, but is sensitive to environmental changes, such as illumination changes, camera movement, etc., and the threshold is difficult to select.

[0003] Object tracking is to obtain the position information and complete motion trajectory of the same tracking object in a continuous video image sequence. There are many tracking methods, mainly tracking through the correlation between the features and positions of objects. For example, Meanshift is a feature matching algorithm based on kernel density estimation; Kalman filter (Li X, Wang K, Wang W, et al. A multiple object tracking method using Kalman filter [C] / / The 2010 IEEE international conference on information and automation. IEEE, 2010: 1862-1866.) is a target tracking algorithm based on the prediction and estimation of the position of the tracked object; and there are also tracking models such as DeepSRDCF that apply deep learning technology to the tracking framework for feature extraction (Gao Jun; Zhu Honghui; Yi Lu Murphey. Moving target tracking method based on YOLO and RRN [J]. Computer Engineering and Design, 2019, 40 (07): 2019-2025. DOI: 10.16208 / j.issn1000-7024.2019.07.036).

[0004] In recent years, the detection and tracking of multiple moving targets based on drone videos has attracted extensive research interest. Since both the moving targets and the drone platform have fast and irregular movements, it is difficult for the tracker to accurately track the moving targets. Secondly, the small size of the moving targets and the low quality of the captured images also increase the detection difficulty of the moving target detector. The problem of moving target detection in a dynamic environment has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a moving target detection and tracking method, device, computer equipment and storage medium to improve the detection and tracking accuracy of moving targets in unmanned aerial vehicle aerial photography scenarios.

[0006] In order to solve the above technical problems, the present application provides a method for detecting and tracking a moving target, including:

[0007] Perform semantic segmentation on continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points;

[0008] Determine a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points;

[0009] Determine the change of the drone's posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone's posture;

[0010] The dynamic feature points are globally verified by using an adaptive corner point cyclic matching method, and the dynamic feature points that do not meet the conditions are eliminated to obtain target feature matching points;

[0011] Based on the target feature matching points and motion compensation information, a sparse optical flow method is used to calculate the optical flow values ​​of the feature points of the target feature matching points to obtain an optical flow vector image;

[0012] The optical flow vector image is processed to obtain a moving target area.

[0013] Optionally, performing semantic segmentation on continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points includes:

[0014] A pyramid scene parsing network is used to extract data features of an input image, and the image features are input into a pyramid pooling module, and a global average pooling operation is performed inside each sub-region to generate feature maps of different levels, wherein the backbone network of the pyramid scene parsing network is replaced by a SwinTransformer based on a Transformer architecture instead of a ResNet.

[0015] Convolutions of different sizes are used to reduce the dimension of the feature maps at different levels and upsample them, the upsampled feature maps at different levels are connected, and finally a final output result is obtained through a convolutional neural network. The pixel value of each pixel is determined based on the final output result, and dynamic feature points and non-dynamic feature points are obtained according to the pixel values.

[0016] Optionally, determining the change of the drone posture by the change of the non-dynamic priori area in the continuous frames, and determining the motion compensation information according to the change of the drone posture includes:

[0017] The optimal homography matrix H is solved by minimizing the transfer error, wherein the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there will be a matching relationship between a pair of matched feature points;

[0018] The optimal homography matrix H is used to compensate for the motion of the drone itself, and the pixel coordinates in the previous frame image are converted into the coordinates before the camera itself moves, so as to obtain motion compensation information.

[0019] Optionally, the global deviation verification of the dynamic feature points is performed by using an adaptive corner point cyclic matching method, and dynamic feature points that do not meet the conditions are eliminated to obtain target feature matching points, including:

[0020] Count the total corner quality in the frame image;

[0021] Sort the corner points from high to low quality, and extract some corner points from high-quality samples for verification; the following formula is used to verify the global deviation E of the feature point optical flow value based on the similarity of grayscale and the smoothness of pixel displacement:

[0022]

[0023] Among them, β is the weight between two hypotheses, represents the spatial gradient of the image at the pixel point, Ω represents the integration area, u and v represent the speed of the point in the x and y directions respectively, the global deviation E describes the deviation of the matched inner point from the model assumption. The smaller the global deviation E, the better the quality of the matching point. dt is the time interval between adjacent image frames, I(x) is the grayscale value of the pixel position in the image, I(x+dt) is the grayscale value of the pixel position in the next frame, and dx is the x coordinate offset of the pixel position in the image;

[0024] The global deviation E is calculated cyclically and minimized to eliminate mismatched point pairs and obtain the target feature matching points.

[0025] Optionally, after performing image processing on the optical flow vector image to obtain a moving target area, the moving target detection and tracking method further includes:

[0026] Receive the target position output by the moving target detector, mark it with a rectangular frame and record the target information;

[0027] Predict the position of moving targets based on the Kalman filter algorithm;

[0028] The actual moving target is searched with the center point of the predicted moving target position as the center and the detection box three times the size as the search area to achieve the target association between the two frames and assign the target ID in the current frame, where the detection box three times the size refers to the area formed by enlarging the detection box three times;

[0029] Update the filter based on the prediction result and the detection result of the current frame, and match the detection box between the two frames and assign and inherit the target ID;

[0030] If there is a detection frame that fails to match, save the location information, size and ID of the detection frame in a temporary array;

[0031] If there is a detection box without an assigned ID, match the detection box with the detection boxes in the temporary array one by one. If the match is successful, the target ID of the detection box in the temporary array will be inherited. If the match fails, a new target ID will be randomly assigned.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides a moving target detection and tracking device, including:

[0033] A feature point extraction module is used to perform semantic segmentation on continuous frames in a video stream to obtain dynamic feature points and non-dynamic feature points;

[0034] A region division module, used for determining a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points;

[0035] A compensation calculation module, used to determine the change of the drone's posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone's posture;

[0036] The deviation correction module is used to perform global deviation verification on the dynamic feature points by using an adaptive corner point cyclic matching method, and remove the dynamic feature points that do not meet the conditions to obtain target feature matching points;

[0037] An image generation module is used to calculate the optical flow value of the target feature matching point based on the target feature matching point and motion compensation information by using a sparse optical flow method to obtain an optical flow vector image;

[0038] The region determination module is used to perform image processing on the optical flow vector image to obtain a moving target region.

[0039] Optionally, the feature point extraction module includes:

[0040] A feature extraction unit is used to extract data features of an input image using a pyramid scene parsing network, input the image features into a pyramid pooling module, and perform a global average pooling operation inside each sub-region to generate feature maps of different levels, wherein the backbone network of the pyramid scene parsing network is replaced by a Swin Transformer based on a Transformer architecture instead of a ResNet;

[0041] The feature point extraction unit is used to reduce the dimension of the feature maps at different levels and upsample them using convolutions of different sizes, connect the feature maps at different levels after upsampling, and finally obtain the final output result through a convolutional neural network, determine the pixel value of each pixel based on the final output result, and obtain dynamic feature points and non-dynamic feature points according to the pixel values.

[0042] Optionally, the compensation calculation module includes:

[0043] A matrix solving unit, used for solving an optimal homography matrix H by minimizing the transfer error, wherein the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there is a matching relationship between a pair of matched feature points;

[0044] The coordinate compensation unit is used to compensate the movement of the drone itself by using the optimal homography matrix H, convert the pixel coordinates in the previous frame image into the coordinates before the camera itself moves, and obtain motion compensation information.

[0045] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned moving target detection and tracking method when executing the computer program.

[0046] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned moving target detection and tracking method are implemented.

[0047] The detection and tracking method, device, computer equipment and storage medium of moving targets provided by the embodiments of the present invention obtain dynamic feature points and non-dynamic feature points by semantically segmenting continuous frames in a video stream; determine dynamic a priori regions and non-dynamic a priori regions based on dynamic feature points and non-dynamic feature points; determine the change of the drone's posture through the change of the non-dynamic a priori regions in continuous frames, and determine the motion compensation information according to the change of the drone's posture; use an adaptive corner point cyclic matching method to perform global deviation verification on the dynamic feature points, and remove the dynamic feature points that do not meet the conditions to obtain the target feature matching points; based on the target feature matching points and motion compensation information, use the sparse optical flow method to calculate the optical flow values ​​of the target feature matching points to obtain an optical flow vector image; perform image processing on the optical flow vector image to obtain the moving target region. The extraction and detection of the contour of the moving target is achieved by filtering feature points to remove mismatched optical flow points and calculating the motion compensation of the drone, thereby improving the detection and tracking accuracy of the target in the drone aerial photography scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0049] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0050] Figure 2 It is a flow chart of an embodiment of a method for detecting and tracking a moving target based on drone aerial photography of the present application;

[0051] Figure 3 It is a feature point projection relationship diagram in one embodiment of the detection and tracking method of a moving target based on drone aerial photography of the present application;

[0052] Figure 4 This is a schematic diagram of transfer error in one embodiment of a method for detecting and tracking a moving target based on drone aerial photography of the present application;

[0053] Figure 5 This is a flow chart of a moving target tracking algorithm in one embodiment of a moving target detection and tracking method based on drone aerial photography of the present application;

[0054] Figure 6 It is a structural schematic diagram of an embodiment of a detection and tracking device for a moving target based on drone aerial photography according to the present application;

[0055] Figure 7It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application in this embodiment are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Mentioning "embodiment" in this embodiment means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiment described in this embodiment can be combined with other embodiments.

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0061] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0062] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0063] It should be noted that the method for detecting and tracking moving targets based on drone aerial photography provided in the embodiment of the present application is executed by a server, and accordingly, the device for detecting and tracking moving targets based on drone aerial photography is arranged in the server.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0065] See also Figure 2 , Figure 2 A method for detecting and tracking a moving target based on drone aerial photography provided by an embodiment of the present invention is shown. The method is applied in Figure 1 The server in is used as an example to illustrate the details as follows:

[0066] S201: Perform semantic segmentation on continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points.

[0067] Specifically, the detection and tracking problem of moving objects based on video streams requires the first premise to provide all dynamic priors of the target to be tracked in the video stream, and then filter out the correct tracking target from the dynamic prior targets. For an input image I, its corresponding semantic image I′ can be obtained by the pyramid scene parsing network (PSPNet):

[0068] I′=PSPNet(I)

[0069] In outdoor road scenes, cars, pedestrians, etc. are common dynamic objects. Therefore, in this embodiment, cars, pedestrians, etc. on the road are set as dynamic prior areas. In the semantic image I′, the pixel value at the dynamic prior area is 0, and the pixel value at the rest of the image is 1.

[0070] The semantic image I′ is used to filter the N feature points extracted from the input image I. Each feature point p is recorded as n The pixel coordinates are (x n ,y n ), then each feature point p in the input image n The corresponding pixel in the semantic image is p′ n , the pixel value in the semantic image is I′(x n ,y n ). Perform the following operations for each feature point: If I′(x n ,y n )=0, the feature point is recorded as a dynamic feature point, and the flag bit is 0; if I′(x n ,y n )=1, the feature point is recorded as a static feature point, and the flag is 1. In this way, the feature points in the input image are screened, and the dynamic points are used as the input of the LK optical flow method to calculate the optical flow value to obtain the optical flow point set. After morphological processing of the optical flow point set, the moving target can be further extracted and detected; the static point set is used for subsequent motion compensation.

[0071] Among them, the PSPNet algorithm aggregates contextual information of different regions through a pyramid scene analysis network to further understand the global information of the image. The traditional PSPNet network uses a convolutional neural network to extract data features of the input image, inputs the image features into the pyramid pooling module, performs global average pooling operations inside each sub-region to generate feature maps of different levels, i.e., 1*1, 2*2, 3*3, 6*6 feature maps, and then uses convolutions of different sizes to reduce the dimension and upsample the output prediction results to the backbone network, connects the feature maps of different levels, and finally obtains the final output result through the convolutional neural network. The backbone network of the traditional PSPNet is a ResNet network based on CNN, which requires a deeper level to obtain a larger receptive field, which limits the model's understanding of global information and affects the segmentation accuracy. Considering that there are many small targets with similar features in road shooting images, such as vehicles, pedestrians, etc., in this embodiment, the backbone network of PSPNet is replaced by ResNet with Swin Transformer based on Transformer as the infrastructure, so that the model has a larger receptive field, which is conducive to the screening of subsequent feature points. In a specific optional implementation, semantic segmentation is performed on continuous frames in a video stream to obtain dynamic feature points and non-dynamic feature points, including:

[0072] Use the pyramid scene parsing network to extract the data features of the input image, input the image features into the pyramid pooling module, and perform global average pooling operations in each sub-region to generate feature maps of different levels. The backbone network of the pyramid scene parsing network is replaced by the Swin Transformer based on the Transformer architecture instead of the ResNet.

[0073] Convolutions of different sizes are used to reduce the dimension of the feature maps at different levels and upsample them, the upsampled feature maps at different levels are connected, and finally a final output result is obtained through a convolutional neural network. The pixel value of each pixel is determined based on the final output result, and dynamic feature points and non-dynamic feature points are obtained according to the pixel values.

[0074] S202: Determine a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points.

[0075] S203: Determine the change of the drone's posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone's posture.

[0076] In a specific optional implementation, determining the change of the drone posture by the change of the non-dynamic prior area in the continuous frames, and determining the motion compensation information according to the change of the drone posture includes:

[0077] The optimal homography matrix H is solved by minimizing the transfer error, where the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there will be a matching relationship between a pair of matched feature points;

[0078] The optimal homography matrix H is used to compensate for the UAV's own motion, and the pixel coordinates in the previous frame image are converted to the coordinates before the camera's own motion to obtain motion compensation information.

[0079] Specifically, when the sparse optical flow method is used to detect and track moving targets, the camera is likely to rotate, translate, or shake significantly during the shooting process, which causes a large global deviation E in the calculation of the optical flow value of the feature point in the current frame, that is, E = f(P,P w ,ξ)=pp * , where ξ=(θ,Δx,Δy) is the affine transformation parameter vector; θ is the rotation angle; Δx and Δy are the translations in the X and Y directions respectively. At this time, the moving object will be misidentified and tracking will fail.

[0080] This embodiment solves the motion transformation parameters between two frames through the matching relationship between feature points. Feature point matching is to find the same pair of feature points in different images based on the similarity of the descriptors by calculating the Hamming distance between the two feature point descriptors, and the motion transformation parameters between the two frames can be solved through the corresponding relationship of the matching feature point pairs between the two frames. However, when solving the motion transformation parameters between the two frames, if matching point pairs of dynamic points are used, it will affect the accuracy of the UAV motion compensation. Therefore, it is necessary to distinguish between the dynamic points and static points in the matching point pairs involved in the calculation. Usually, the feature points in the dynamic prior are called external points, and the feature points in the background, that is, static points, are called internal points. Points in the dynamic prior area are used as dynamic prior map points, and points in the non-dynamic prior area are used as non-dynamic prior map points. Figure 3 As shown. The existence of external points often causes a large deviation in parameter solution. Therefore, this embodiment proposes an internal point screening method based on the semantic segmentation network on the parallel semantic segmentation thread. Based on the segmentation results of the PSPNet network in continuous frames, the static points in the non-dynamic prior area, that is, the segmented background area, are used to restore the movement of the camera between the two frames, and the points of the current frame are projected to the virtual frame before the movement, thereby realizing the correction of the camera's own movement and preparing for the subsequent detection and tracking of moving targets.

[0081] Minimize the transfer error to solve the optimal H matrix, and use the homography matrix H to realize the projection relationship between the two frames before and after the camera moves. There will be a matching relationship p2 = Hp1 between a pair of matched feature points.

[0082] The matrix H can be represented by a non-singular 3*3 matrix as:

[0083]

[0084] In this way, we get a transformation that directly describes the coordinates of the two images, and define this transformation relationship as the homography matrix H. However, due to the motion of the camera itself, the measurement of the matching points is very inaccurate. If we directly use the H matrix obtained from the four point pairs to calculate the corresponding matching point poses, the global deviation will be very large or even completely incompatible. By minimizing the transfer error of the two images, we get the optimal transformation matrix H.

[0085] Assume that ε is the transfer error of corresponding pixels between the two images. When solving the H matrix, the Levenberg-Marquardt method is used for continuous optimization. The homography matrix obtained when ε is minimized is the optimal homography matrix H, which represents the optimal pose transformation of the camera between two consecutive frames. The optimal homography matrix H is used to compensate for the movement of the drone itself, and the pixel coordinates in the image frame at the previous moment are converted into the coordinates before the camera itself moves. The RGB-D camera used in this embodiment can actively measure the depth of each target point, thereby directly obtaining the specific position of a certain spatial point. There is a conversion relationship between the pixel point and the spatial point position:

[0086]

[0087] Among them, T describes the camera's position transformation from the world coordinate system to the current coordinate system. Estimating the point of the world plane Then, according to the above formula, it is reprojected to the matching corresponding point pair under the current estimation. superior:

[0088] According to the homography matrix estimated from the current data, the virtual frame is obtained Pixel coordinates projected to the current frame like Figure 4 As shown. At this time, there is a distance error between the two points before and after projection and the actual matching point, that is, in Therefore, the Levenberg-Marquardt method is used to continuously optimize and solve the H matrix. The H matrix with the smallest distance error is the best estimate of the transformation between two pixels:

[0089]

[0090] Among them, x i and x i ′ are the pixel coordinates of the previous frame and the current frame respectively, and d() is the distance calculation formula.

[0091] The motion correction module uses the optimal compensation homography matrix to project the feature points in the previous frame into the current frame in the form of error compensation before calculating the optical flow value of the feature points. The original projection points with errors caused by the movement of the camera itself are corrected to the projection points before the camera moves. By minimizing the transfer error, the influence of drone shaking and translation on the position estimation and tracking of moving objects is eliminated, and the video images of consecutive frames are fixed under the same drone perspective.

[0092] S204: global deviation verification is performed on the dynamic feature points by using an adaptive corner point cyclic matching method, and dynamic feature points that do not meet the conditions are eliminated to obtain target feature matching points.

[0093] In a specific optional implementation, the dynamic feature points are globally verified by using an adaptive corner point cyclic matching method, and the dynamic feature points that do not meet the conditions are eliminated, and the target feature matching points are obtained, including:

[0094] Count the total corner quality in the frame image;

[0095] Sort the corner points from high to low quality, and extract some corner points from high-quality samples for verification; the following formula is used to verify the global deviation E of the feature point optical flow value based on the similarity of grayscale and the smoothness of pixel displacement:

[0096]

[0097] Among them, β is the weight between the two assumptions, Ω represents the integration area, u and v represent the speed in the x and y directions of the point respectively. It represents the spatial gradient of the image at the pixel point. The global deviation E describes the deviation of the matched inlier point from the model assumption. The smaller the global deviation E, the better the quality of the matched point. dt is the time interval between adjacent image frames. I(x) is the gray value of the pixel point in the image. I(x+dt) is the gray value of the pixel point in the next frame. dx is the x-coordinate offset of the pixel point in the image.

[0098] The global deviation E is calculated cyclically and minimized to eliminate mismatched point pairs and obtain the target feature matching points.

[0099] Specifically, in the sparse optical flow method, the grayscale invariance assumption is a strong assumption that is difficult to hold in natural scenes. In order to overcome the defect of low detection accuracy of the traditional derivative optical flow method, this embodiment allows a small grayscale value change within the threshold range, proposes a circular matching algorithm to reduce the mismatch rate of corner points, adds a penalty function to constrain the grayscale invariance assumption, and improves the tracking and matching accuracy of optical flow points.

[0100] The cyclic matching algorithm first counts the total quality of corner points, sorts the corner points from high to low quality, extracts some corner points from high-quality samples for verification, and gradually obtains the optimal solution. This embodiment is based on two assumptions of LK optical flow, namely the similarity of grayscale and the smoothness of pixel displacement, and calculates the global deviation E of the feature point optical flow value based on these two assumptions for verification.

[0101]

[0102] Here, β is the weight between two hypotheses. Represents the spatial gradient of the image at the pixel. Ω represents the integration area, u and v represent the speed of the point in the x and y directions respectively. The global deviation E describes the deviation of the matched internal point from the model assumption. The smaller the global deviation E, the better the quality of the matching point. dt is the time interval between adjacent image frames, I(x) is the grayscale value of the pixel position in the image, I(x+dt) is the grayscale value of the pixel position in the next frame, and dx is the x-coordinate offset of the pixel position in the image. The global deviation E is calculated cyclically and minimized to eliminate mismatched point pairs and improve the accuracy of optical flow calculation. The specific process of the algorithm is as follows:

[0103] 1) Set the initial value and upper limit of the number of iterations, the upper limit of the number of inliers, and the deviation threshold to determine the first batch of inliers;

[0104] 2) Calculate the global deviation E of the inliers, sort the inliers in descending order according to the size of the global deviation, and then select n inliers with lower error values ​​as new inliers;

[0105] 3) Model calculation: The filtered inner points are used to calculate the motion information of each point. Assuming that the pixels in a certain window have the same motion, if a matching point is completely inconsistent with the pixel motion of most other matching points when solving the point motion, then it is confirmed that the point is a mismatch, or a noise point. Assuming that a pair of matching points has a (dx, dy) offset, the size and angle of this displacement can be calculated as arctan(dx / dy).

[0106] 4) If it is a correct matching point, the size and angle deviation should not be large. If a matching point has a large deviation, remove it from the inner points and repeat the above steps.

[0107] S205: Based on the target feature matching points and the motion compensation information, the sparse optical flow method is used to calculate the optical flow values ​​of the feature points of the target feature matching points to obtain an optical flow vector image.

[0108] The sparse optical flow method (LK optical flow method) calculates the optical flow value of the feature points of each frame image, matches and tracks the optical flow points, and then tracks the moving objects. The traditional LK optical flow method is based on three assumptions:

[0109] (1) The pixel brightness remains unchanged in consecutive frames of images;

[0110] (2) The interval between adjacent frames is short;

[0111] (3) Adjacent pixels have similar motions.

[0112] According to hypothesis 1, if the gray value is constant, we have

[0113] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0114] Where t and t+dt are the corresponding times of adjacent image frames, and I(x,y,t) and I(x+dx,y+dy,t+dt) are the positions of pixels in the image.

[0115] According to assumption 2, since the time interval between adjacent image frames is small, the right side of (1) can be expanded by Taylor series to obtain:

[0116]

[0117] Combining them together, we get:

[0118] Then, divide both sides of the equation by dt: in, and are the speeds of the feature points on the x-axis and y-axis, denoted as u and v, respectively, and denoted as I x ,I y , the grayscale change of the feature point over time is I t , and get the following matrix-style result: Additional constraints are introduced to calculate the feature point velocities u and v.

[0119] Through the above process, the optical flow values ​​of the feature points that fall within the dynamic prior area after semantic segmentation are calculated, the mean and standard deviation are solved to determine an optical flow value threshold, and the feature points in ordinary road scenes are distinguished into optical flow points and non-optical flow points. Only optical flow points are retained and optical flow vectors are made. The retained optical flow vectors are binarized and reasonably morphologically operated to extract the coverage area of ​​the moving target.

[0120] In this embodiment, based on the traditional sparse optical flow method, the feature matching points are accurately screened, and motion compensation information is performed according to the displacement of the drone to further improve the robustness of target recognition and tracking.

[0121] S206: Perform image processing on the optical flow vector image to obtain a moving target area.

[0122] Specifically, after feature point screening, motion compensation and optical flow calculation, the calculated optical flow vector image is grayed and segmented by adaptive threshold to obtain a binary image; the binary image is morphologically processed, and the non-zero pixel value area in the final binary image is the detected moving target area. The moving target area is taken as the largest rectangular frame in the video frame to obtain the detection result of the moving target.

[0123] In a specific implementation of the first embodiment, after performing image processing on the optical flow vector image to obtain the moving target area, the moving target detection and tracking method based on drone aerial photography further includes:

[0124] Receive the target position output by the moving target detector, mark it with a rectangular frame and record the target information;

[0125] Predict the position of moving targets based on the Kalman filter algorithm;

[0126] The actual moving target is searched with the center point of the predicted moving target position as the center and the detection box three times the size as the search area to achieve the target association between the two frames and assign the target ID in the current frame, where the detection box three times the size refers to the area formed by enlarging the detection box three times;

[0127] Update the filter based on the prediction result and the detection result of the current frame, and match the detection box between the two frames and assign and inherit the target ID;

[0128] If there is a detection frame that fails to match, save the location information, size and ID of the detection frame in a temporary array;

[0129] If there is a detection box without an assigned ID, match the detection box with the detection boxes in the temporary array one by one. If the match is successful, the target ID of the detection box in the temporary array will be inherited. If the match fails, a new target ID will be randomly assigned.

[0130] Specifically, the current mainstream tracking algorithm mainly uses the Kalman filter algorithm to track moving targets, applies the Kalman filter algorithm to establish a target motion model, and predicts the position of the moving target in the next frame. The Kalman filter algorithm can reduce the search range of the moving target between consecutive frames, quickly and accurately locate the target position in a relatively short time, and achieve accurate and rapid tracking of the moving target. However, due to the limitations of the moving target detection algorithm, in some unknown cases, the same moving target may be missed and the frame ID may be switched during the detection and tracking process. For example, a certain moving target being tracked is in a stationary state in the frame, but it resumes its motion state in a relatively short time. Due to the switching of the motion state, a new ID is reallocated to the moving object, and the same moving target ID may be switched or even tracked missed in a relatively short time.

[0131] In the embodiment, a Re-Find tracking module is proposed to improve the accuracy of moving target tracking. The target association between two frames is achieved by searching the actual moving target with the predicted moving target center point as the center and the detection box three times the size as the search area, and the target ID is allocated under the frame.

[0132] The specific implementation of the Re-Find tracking module in this embodiment is as follows: First, when establishing the tracking relationship between two frames of moving targets, if there is a detection frame that fails to match in the previous frame, this unmatched detection frame will not be discarded immediately, but the target ID, detection frame size, and detection frame center point coordinates of the detection frame will be saved in a temporary array. Then, within a very short period of time, if a suitable detection frame appears to match the detection frame, the algorithm will assign the detection frame ID that failed to match to the current suitable detection frame. Detection frame matching must meet two conditions: one is that the current detection frame ID is assigned to be empty; the other is that the size and center point coordinates of the current detection frame correspond to the detection frame to be matched within a certain error, that is, the two detection frames are similar in size and have similar position coordinates. If the detection frame that failed to match has not been successfully assigned within the maximum time, the system will discard the retained detection frame information. Finally, these detection frames with reallocated target IDs are added to the tracking information to complete the recovery of missed moving target detection and the inheritance of target IDs. The specific process of the tracking algorithm part combined with the Re-Find tracking module is as follows Figure 5 As shown:

[0133] 1) The moving target detector outputs the target position, marks it with a rectangular frame and records the target information

[0134] 2) Predict the position of the moving target based on the Kalman filter algorithm

[0135] 3) Update the filter based on the prediction result and the detection result of the frame to achieve the matching of the detection frame between the two frames and the allocation and inheritance of the target ID

[0136] 4) If there is a detection frame that fails to match in the previous frame, save the location information, size and ID of the detection frame in a temporary array

[0137] 5) If there is a detection frame without an assigned ID in the frame, match the detection frame with the detection frames in the temporary array one by one. If the match is successful, the target ID of the detection frame in the temporary array will be inherited. If the match fails, a new target ID will be randomly assigned.

[0138] Furthermore, if there is a detection frame without an assigned ID in the frame, a determination is made as to whether the target ID of the detection frame is empty. If the target ID of the detection frame is empty, the detection frame is matched one by one with the detection frames in the temporary array; otherwise, the detection frame is deleted.

[0139] Furthermore, this embodiment evaluates the effectiveness of the re-identification module and motion compensation module proposed in this embodiment in target tracking through specific examples. IDs is used to evaluate the ID jump of the same target frame, for example, a detection frame disappears from the detector's field of view for a long time, and then reassigns an ID after it appears; IDF1 is used to evaluate the correctness of the association between the detection frame and the target trajectory. '↑' means the higher the value, the better, and '↓' means the lower the value, the better.

[0140] Table 1 Comparison of Re-Find re-identification modules

[0141]

[0142] Table 2 Comparison of motion correction modules

[0143]

[0144] As shown in Table 1, using the re-identification module proposed in this embodiment, compared with the traditional Kalman filter algorithm, the tracking algorithm of this embodiment has significantly improved the tracking accuracy while meeting the real-time requirements (the inference speed only dropped from 27.0 to 25.8). After adding the re-identification module on the basis of the Kalman filter algorithm, the number of ID switching times was reduced by nearly 60% (IDs from 291 to 124), and the IDF1 was also increased by 14 percentage points (38.6 to 52.7). The above quantitative results confirm that the re-identification strategy proposed in this embodiment has a significant improvement in improving the continuity and accuracy of moving target tracking.

[0145] Table 2 shows the results of moving target tracking using different semantic segmentation algorithms in the motion correction module. The FCN semantic segmentation model is prone to lose local feature information in the segmentation of small targets, and the target tracking evaluation effect is also the worst. The motion compensation module proposed in this embodiment uses an improved PSPNet network, and the tracking index has been significantly improved compared with the original PSPNet network and DeeplabV3 network (IDs is 11, IDF1 is 56.3), which proves the effectiveness of the strategy of selecting the PSPNet network for feature point screening in this embodiment.

[0146] The moving target detector of the algorithm of this embodiment is switched to the traditional LK optical flow method, inter-frame difference method, and background subtraction method and compared with the algorithm of this embodiment. A total of 1789 frames of a scene video segment of the shooting data set are selected for comparison of reasoning speed and accuracy. This embodiment manually marks the detection frames of all moving targets in the scene and records the coordinate information for comparison of reasoning speed and accuracy.

[0147] The experimental results are shown in Table 3. The experimental results show that compared with the traditional moving target detection algorithm, under the premise of ensuring the inference speed (FPS is 17.1), the algorithm of this embodiment has obvious advantages in tracking accuracy (MOTA is 41.5, which is much higher than other algorithms). After applying motion compensation + semantic segmentation thread to filter feature points, the three detection error indicators of IDs, FP, and FN have all decreased significantly (FP is 276, FN is 481, and IDs is 84, which is significantly lower than the original algorithm's 3293, 5920, and 497). However, in terms of inference speed, compared with the algorithm of this embodiment, the traditional algorithm still has obvious advantages (FPS is 22.6, 24.2, and 24.6 respectively), but the algorithm of this embodiment can still ensure the real-time performance of the algorithm under the premise of greatly improving the tracking accuracy. It can be seen from the visualization results that the algorithm of this embodiment has a good tracking effect on small moving targets taken by drones, and there is no switching of vehicle IDs (that is, no tracking failure occurs).

[0148] Table 3 Performance comparison of multiple moving target tracking algorithms

[0149]

[0150] In this embodiment, the dynamic feature points and non-dynamic feature points are obtained by semantically segmenting the continuous frames in the video stream; the dynamic prior regions and non-dynamic prior regions are determined based on the dynamic feature points and non-dynamic prior regions; the change of the drone posture is determined by the change of the non-dynamic prior regions in the continuous frames, and the motion compensation information is determined according to the change of the drone posture; the dynamic feature points are globally verified by adaptive corner point cyclic matching, and the dynamic feature points that do not meet the conditions are eliminated to obtain the target feature matching points; based on the target feature matching points and motion compensation information, the sparse optical flow method is used to calculate the optical flow values ​​of the target feature matching points to obtain the optical flow vector image; the optical flow vector image is processed to obtain the moving target area. The extraction and detection of the moving target contour are achieved by filtering the feature points to eliminate the mismatched optical flow points and calculating the motion compensation of the drone, thereby improving the detection and tracking accuracy of the target in the drone aerial photography scene.

[0151] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0152] Figure 6 The principle block diagram of the moving target detection and tracking device based on drone aerial photography is shown in one-to-one correspondence with the moving target detection and tracking method based on drone aerial photography in the above embodiment. Figure 6As shown, the detection and tracking device for moving targets based on drone aerial photography includes a feature point extraction module 31, a region division module 32, a compensation calculation module 33, a deviation correction module 34, an image generation module 35 and a region determination module 36. The functional modules are described in detail as follows:

[0153] A feature point extraction module 31 is used to perform semantic segmentation on continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points;

[0154] A region division module 32, configured to determine a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points;

[0155] The compensation calculation module 33 is used to determine the change of the drone posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone posture;

[0156] The deviation correction module 34 is used to perform global deviation verification on the dynamic feature points by using an adaptive corner point cyclic matching method, and remove the dynamic feature points that do not meet the conditions to obtain target feature matching points;

[0157] An image generation module 35 is used to calculate the optical flow value of the target feature matching point by using a sparse optical flow method based on the target feature matching point and motion compensation information to obtain an optical flow vector image;

[0158] The region determination module 36 is used to perform image processing on the optical flow vector image to obtain a moving target region.

[0159] Optionally, the feature point extraction module 31 includes:

[0160] A feature extraction unit is used to extract data features of an input image using a pyramid scene parsing network, input the image features into a pyramid pooling module, and perform a global average pooling operation inside each sub-region to generate feature maps of different levels, wherein the backbone network of the pyramid scene parsing network is replaced by a Swin Transformer based on a Transformer architecture instead of a ResNet;

[0161] A feature point extraction unit is used to reduce the dimension of the feature maps of different levels and upsample them using convolutions of different sizes, connect the upsampled feature maps of different levels, and finally obtain the final output result through a convolutional neural network, determine the pixel value of each pixel based on the final output result, and obtain dynamic feature points and non-dynamic feature points according to the pixel values. Optionally, the compensation calculation module 33 includes:

[0162] A matrix solving unit, used for solving an optimal homography matrix H by minimizing the transfer error, wherein the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there is a matching relationship between a pair of matched feature points;

[0163] The coordinate compensation unit is used to compensate the movement of the drone itself by using the optimal homography matrix H, convert the pixel coordinates in the previous frame image into the coordinates before the camera itself moves, and obtain motion compensation information.

[0164] Regarding the specific definition of the detection and tracking device for moving targets based on drone aerial photography, please refer to the definition of the detection and tracking method for moving targets based on drone aerial photography above, which will not be repeated here. Each module in the above-mentioned detection and tracking device for moving targets based on drone aerial photography can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0165] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0166] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0167] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0168] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the detection and tracking method of the moving target based on the aerial photography of the unmanned aerial vehicle, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0169] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the detection and tracking method of the moving target based on the drone aerial photography.

[0170] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0171] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned method for detecting and tracking moving targets based on drone aerial photography.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0173] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for detecting and tracking a moving target, characterized in that: include: Perform semantic segmentation on continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points; Determine a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points; Determine the change of the drone's posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone's posture; The dynamic feature points are globally verified by using an adaptive corner point cyclic matching method, and the dynamic feature points that do not meet the conditions are eliminated to obtain target feature matching points; Based on the target feature matching points and motion compensation information, a sparse optical flow method is used to calculate the optical flow values ​​of the feature points of the target feature matching points to obtain an optical flow vector image; Performing image processing on the optical flow vector image to obtain a moving target area; Taking the largest rectangular frame of the moving target area, obtaining the detection result of the moving target, and tracking the detected moving target; The semantic segmentation of the continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points includes: For any input image I in consecutive frames, its corresponding semantic image I′ can be obtained by the pyramid scene parsing network: I′=PSPNet(I) In the semantic image I′, the pixel value in the dynamic prior region is 0, and the pixel value in the rest of the image is 1. The semantic image I′ is used to filter the N feature points extracted from the input image I. Each feature point p is recorded as n The pixel coordinates are (x n ,y n ), then each feature point p in the input image n The corresponding pixel in the semantic image is p n ′, the pixel value in the semantic image is I′(x n ,y n ); Perform the following operations for each feature point: If I′(x n ,y n )=0, the feature point is recorded as a dynamic feature point, and the flag bit is 0; if I′(x n ,y n )=1, the feature point is recorded as a static feature point, and the flag bit is 1.

2. The method for detecting and tracking a moving target according to claim 1, wherein: The semantic segmentation of continuous frames in the video stream to obtain dynamic feature points and non-dynamic feature points includes: A pyramid scene parsing network is used to extract data features of an input image, and the image features are input into a pyramid pooling module, and a global average pooling operation is performed inside each sub-region to generate feature maps of different levels, wherein the backbone network of the pyramid scene parsing network is replaced by a SwinTransformer based on a Transformer architecture instead of a ResNet. Convolutions of different sizes are used to reduce the dimension of the feature maps at different levels and upsample them, the upsampled feature maps at different levels are connected, and finally a final output result is obtained through a convolutional neural network. The pixel value of each pixel is determined based on the final output result, and dynamic feature points and non-dynamic feature points are obtained according to the pixel values.

3. The method for detecting and tracking a moving target according to claim 1, wherein: Determining the change of the drone posture by the change of the non-dynamic priori area in the continuous frames, and determining the motion compensation information according to the change of the drone posture includes: The optimal homography matrix H is solved by minimizing the transfer error, wherein the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there will be a matching relationship between a pair of matched feature points; The optimal homography matrix H is used to compensate for the motion of the drone itself, and the pixel coordinates in the previous frame image are converted into the coordinates before the camera itself moves, so as to obtain motion compensation information.

4. The method for detecting and tracking a moving target according to claim 1, wherein: The method of adopting adaptive corner point cyclic matching to perform global deviation verification on the dynamic feature points and remove the dynamic feature points that do not meet the conditions to obtain the target feature matching points includes: Count the total corner quality in the frame image; Sort the corner points from high to low quality, and extract some corner points from high-quality samples for verification; the following formula is used to verify the global deviation E of the feature point optical flow value based on the similarity of grayscale and the smoothness of pixel displacement: Among them, β is the weight between two hypotheses, represents the spatial gradient of the image at the pixel point, Ω represents the integration area, u and v represent the speed of the point in the x and y directions respectively, the global deviation E describes the deviation of the matched inner point from the model assumption. The smaller the global deviation E, the better the quality of the matching point. dt is the time interval between adjacent image frames, I(x) is the grayscale value of the pixel position in the image, I(x+dt) is the grayscale value of the pixel position in the next frame, and dx is the x coordinate offset of the pixel position in the image; The global deviation E is calculated cyclically and minimized to eliminate mismatched point pairs and obtain the target feature matching points.

5. The method for detecting and tracking a moving target according to claim 1, wherein: After performing image processing on the optical flow vector image to obtain a moving target area, the moving target detection and tracking method further includes: Receive the target position output by the moving target detector, mark it with a rectangular frame and record the target information; Predict the position of moving targets based on the Kalman filter algorithm; The actual moving target is searched with the center point of the predicted moving target position as the center and the detection box three times the size as the search area to achieve the target association between the two frames and assign the target ID in the current frame, where the detection box three times the size refers to the area formed by enlarging the detection box three times; Update the filter based on the prediction result and the detection result of the current frame, and match the detection box between the two frames and assign and inherit the target ID; If there is a detection frame that fails to match, save the location information, size and ID of the detection frame in a temporary array; If there is a detection box without an assigned ID, match the detection box with the detection boxes in the temporary array one by one. If the match is successful, the target ID of the detection box in the temporary array will be inherited. If the match fails, a new target ID will be randomly assigned.

6. A moving target detection and tracking device, characterized in that: include: A feature point extraction module is used to perform semantic segmentation on continuous frames in a video stream to obtain dynamic feature points and non-dynamic feature points; A region division module, used for determining a dynamic a priori region and a non-dynamic a priori region based on the dynamic feature points and the non-dynamic feature points; A compensation calculation module, used to determine the change of the drone's posture through the change of the non-dynamic prior area in the continuous frames, and determine the motion compensation information according to the change of the drone's posture; The deviation correction module is used to perform global deviation verification on the dynamic feature points by using an adaptive corner point cyclic matching method, and remove the dynamic feature points that do not meet the conditions to obtain target feature matching points; An image generation module is used to calculate the optical flow value of the target feature matching point based on the target feature matching point and motion compensation information by using a sparse optical flow method to obtain an optical flow vector image; A region determination module, used for performing image processing on the optical flow vector image to obtain a moving target region; A tracking module, used to take the largest rectangular frame of the moving target area, obtain the detection result of the moving target, and track the detected moving target; Wherein, the feature point extraction module includes: For any input image I in consecutive frames, its corresponding semantic image I′ can be obtained by the pyramid scene parsing network: I′=PSPNet(I) In the semantic image I′, the pixel value in the dynamic prior region is 0, and the pixel value in the rest of the image is 1. The semantic image I′ is used to filter the N feature points extracted from the input image I. Each feature point p is recorded as n The pixel coordinates are (x n ,y n ), then each feature point p in the input image n The corresponding pixel in the semantic image is p n ′, the pixel value in the semantic image is I′(x n ,y n ); Perform the following operations for each feature point: If I′(x n ,y n )=0, the feature point is recorded as a dynamic feature point, and the flag bit is 0; if I′(x n ,y n )=1, the feature point is recorded as a static feature point, and the flag bit is 1.

7. The moving target detection and tracking device according to claim 6, characterized in that: The feature point extraction module comprises: A feature extraction unit is used to extract data features of an input image using a pyramid scene parsing network, input the image features into a pyramid pooling module, and perform a global average pooling operation inside each sub-region to generate feature maps of different levels, wherein the backbone network of the pyramid scene parsing network is replaced by a Swin Transformer based on a Transformer architecture instead of a ResNet; The feature point extraction unit is used to reduce the dimension of the feature maps at different levels and upsample them using convolutions of different sizes, connect the feature maps at different levels after upsampling, and finally obtain the final output result through a convolutional neural network, determine the pixel value of each pixel based on the final output result, and obtain dynamic feature points and non-dynamic feature points according to the pixel values.

8. The moving target detection and tracking device according to claim 6, characterized in that: The compensation calculation module comprises: A matrix solving unit, used for solving an optimal homography matrix H by minimizing the transfer error, wherein the optimal homography matrix H is used to characterize the transformation relationship between the image coordinates of two consecutive frames, and there is a matching relationship between a pair of matched feature points; The coordinate compensation unit is used to compensate the movement of the drone itself by using the optimal homography matrix H, convert the pixel coordinates in the previous frame image into the coordinates before the camera itself moves, and obtain motion compensation information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting and tracking a moving target as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting and tracking a moving target as described in any one of claims 1 to 5 is implemented.

Images