A matching positioning method and device based on unmanned aerial vehicle FMV video, equipment and medium

Abstract

The present application relates to a kind of matching positioning method, device, equipment and medium based on unmanned aerial vehicle FMV video, comprising: S1.to the full dynamic video FMV of unmanned aerial vehicle real-time shooting target detection is carried out, and target is tracked and numbered;S2.all tracked targets in different video frames of full dynamic video FMV are projected on the same key frame, and all targets in the key frame are numbered and fused, form the target set after fusion;S3.the key frame is matched with satellite map;S4.target set is mapped to the satellite map after matching according to number, generates the motion trajectory about target.The present application improves the stability and continuity of target tracking by projecting multiple frames of target to key frame and clustering fusion;Integrate target detection, tracking, matching and map mapping, etc., realize the intelligence flow transfer integration from video detection to geographic trajectory generation, overall processing procedure is highly automated, greatly improves processing efficiency and system practicability.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and computer technology, and specifically relates to a matching and positioning method, device, equipment and medium based on UAV FMV video. Background Technology

[0002] With the rapid development of UAV technology, full-motion video (FMV) intelligence processing technology has become an important means of geographic reconnaissance, situational awareness and target tracking, and is widely used in military reconnaissance, emergency response and other fields to achieve continuous and large-scale monitoring and tracking of dynamic ground targets.

[0003] However, the following major technical challenges still exist in practical applications: 1) Poor consistency in target tracking: In complex scenarios, existing detection and tracking algorithms are prone to frequent switching of target identity IDs due to occlusion, scale changes, target overlap, etc., making it difficult to maintain identity consistency across frames.

[0004] 2) Low efficiency in intelligence generation and circulation: The conversion process from video analysis to geographic intelligence relies heavily on manual intervention and fails to form an automated and integrated processing loop, making it difficult to meet the needs of real-time intelligence distribution.

[0005] 3) Insufficient geolocation accuracy: Traditional image matching methods are sensitive to changes in viewpoint and lighting differences, making it difficult to achieve high-precision and robust matching between video frames and satellite maps, and ensuring the reliability of the geographic coordinates output of image targets. Summary of the Invention

[0006] To overcome the problems of existing technologies, this invention provides a matching and positioning method, apparatus, device, and medium based on UAV FMV video, which is used to solve the current problems.

[0007] A matching and localization method based on UAV FMV video, the method comprising the following steps: S1. Detect targets in the real-time full-motion video (FMV) captured by the UAV, and track and number the targets; S2. The full dynamic video (FMV) is judged to obtain key frames. All tracked targets in different video frames of the full dynamic video (FMV) are projected onto the same key frame, and all targets in the key frame are numbered and fused to form a fused target set. S3. Match the keyframes with the satellite map for scene representation; S4. Map the target set onto the matched satellite map according to the numbering to generate the motion trajectory of the target.

[0008] In addition to the aspects and any possible implementations described above, a further implementation is provided in which, in S1, the YOLOv8s algorithm is used to perform target detection on the full dynamic video (FMV) to obtain target information in each video frame, and the video frames and target information are sent to the DeepSORT algorithm for target tracking, thereby tracking and numbering the target.

[0009] In addition to the aspects described above and any possible implementation, a further implementation is provided, wherein the target information includes target bounding box location information, target category, and target confidence level.

[0010] In addition to the aspects and any possible implementations described above, an implementation is further provided in which S2 includes: S21. Determine whether the current frame is the first video frame from the full dynamic video (FMV). S22. If it is the first video frame, then treat it as a keyframe; otherwise, treat it as the current frame. S23. Match the current frame with the keyframe; S23. If the match is successful, generate a keyframe target set; S24. If the matching fails, generate the merged target set.

[0011] As described above, and in accordance with any possible implementation, an implementation is further provided, wherein S23 specifically includes: S231. The homography matrix between the current frame and the keyframe is estimated using the RANSAC algorithm; S232. Project the target bounding box of the current frame into the coordinate system of the keyframe using the homography matrix; S233. Store the projection results into the keyframe target set.

[0012] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein S24 specifically includes: S241. Determining the relationship between the interval between the key frame and the current frame and the set minimum interval threshold; S242. If the interval is less than the set minimum interval threshold, then directly switch the keyframe to the current frame; S243. If the gap is greater than or equal to the set minimum gap threshold, then perform clustering operations on the keyframe images and target sets, and at the same time, use the current frame as a new keyframe.

[0013] In addition to the aspects and any possible implementations described above, an implementation is further provided in which S3 includes: S31. Preprocess the keyframe images and satellite maps; S32. Extract and register image feature points from the preprocessed keyframe image and satellite map to obtain the mapping matrix between the two; S33. Based on the mapping matrix and the keyframe target set, calculate the corresponding coordinates of the target in the satellite map to achieve the positioning of the target in the satellite map.

[0014] The present invention also provides a matching and positioning device based on UAV FMV video, the device being used to implement the method, specifically including: The detection module is used to detect targets in real-time full-motion video (FMV) captured by the UAV, and to track and number the targets. The fusion module is used to determine the full dynamic video (FMV) and obtain key frames. It projects all tracked targets in different video frames of the full dynamic video FMV onto the same key frame and numbers and fuses all targets in the key frame to form a fused target set. The matching module is used to perform scene matching between the keyframes and satellite maps; The mapping module is used to map the target set onto the matched satellite map according to the numbering, and generate the motion trajectory of the target.

[0015] The present invention also provides a computer storage medium storing a computer program, the computer program being executed by a processor to implement the method described.

[0016] The present invention also provides an electronic device, the electronic device comprising: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method.

[0017] Beneficial effects of the present invention This invention achieves stable maintenance of long-term, multi-target identity IDs by integrating YOLOv8s detection, DeepSORT short-term tracking, multi-frame projection, and clustering association, avoiding tracking loss and ID jumps. By constructing an end-to-end automatic mapping pipeline from pixel coordinates to geographic coordinates, it enables real-time generation and visualization of motion trajectories, significantly improving the generation speed and usability of intelligence products and solving the problem of intelligence flow lag. Furthermore, it introduces a scene matching mechanism between keyframes and satellite maps, improving the matching accuracy between video content and geographic coordinates, ensuring that target location results have actual geographic reference value, and solving the problem of inaccurate geographic mapping. Therefore, this invention has the following advantages compared to existing technologies: A cross-frame identity fusion mechanism is proposed: This invention effectively solves the problem of inconsistent target identities in complex scenes by projecting targets from multiple frames onto key frames and performing clustering and fusion, thereby improving the stability and continuity of target tracking.

[0018] Constructing an automated intelligence generation pipeline: This invention integrates multiple modules such as target detection, tracking, matching, and map mapping, realizing an integrated intelligence flow from video detection to geographic trajectory generation. The overall processing flow is highly automated, significantly improving processing efficiency and system usability.

[0019] Achieving high-precision geocalibration: This invention combines the dual constraints of visual projection and georegistration to significantly improve the mapping accuracy of targets from image space to geographic space, supporting higher confidence in positioning and trajectory analysis. Attached Figure Description

[0020] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 This is a flowchart of the cross-frame fusion process of the present invention; Figure 3 This is a flowchart of the clustering process of the present invention; Figure 4 This is a flowchart of the scene matching and positioning process of the present invention. Detailed Implementation

[0021] To better understand the technical solution of this invention, the content of this invention includes, but is not limited to, the specific embodiments described below. Similar technologies and methods should be considered within the scope of protection of this invention. To make the technical problems to be solved, the technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0022] It should be understood that the embodiments described in this invention are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0023] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0024] like Figures 1-4 As shown, a matching and localization method based on UAV FMV video is described, the method comprising the following steps: S1. Detect targets in the real-time full-motion video (FMV) captured by the UAV, and track and number the targets; S2. The full dynamic video (FMV) is judged to obtain key frames. All tracked targets in different video frames of the full dynamic video (FMV) are projected onto the same key frame, and all targets in the key frame are numbered and fused to form a fused target set. S3. Match the keyframes with the satellite map for scene representation; S4. Map the target set onto the matched satellite map according to the numbering to generate the motion trajectory of the target.

[0025] Further, S2 includes: S21. Determine whether the current frame is the first video frame from the full dynamic video (FMV). S22. If it is the first video frame, then treat it as a keyframe; otherwise, treat it as the current frame. S23. Match the current frame with the keyframe; S23. If the match is successful, generate a keyframe target set; S24. If the matching fails, generate the merged target set.

[0026] Furthermore, S23 specifically includes: S231. The homography matrix between the current frame and the keyframe is estimated using the RANSAC algorithm; S232. Project the target bounding box of the current frame into the coordinate system of the keyframe using the homography matrix; S233. Store the projection results into the keyframe target set.

[0027] Furthermore, S24 specifically includes: S241. Determining the relationship between the interval between the key frame and the current frame and the set minimum interval threshold; S242. If the interval is less than the set minimum interval threshold, then directly switch the keyframe to the current frame; S243. If the gap is greater than or equal to the set minimum gap threshold, then perform clustering operations on the keyframe images and target sets, and at the same time, use the current frame as a new keyframe.

[0028] Further, S3 includes: S31. Preprocess the keyframe images and satellite maps; S32. Extract and register image feature points from the preprocessed keyframe image and satellite map to obtain the mapping matrix between the two; S33. Based on the mapping matrix and the keyframe target set, calculate the corresponding coordinates of the target in the satellite map to achieve the positioning of the target in the satellite map.

[0029] Specifically, the implementation process of this invention is as follows: The present invention realizes the matching and positioning of FMV video and the technology of intelligence information flow, which mainly includes four steps: target detection and tracking, cross-frame fusion, scene matching and trajectory drawing.

[0030] 1. Target Detection and Tracking: First, FMV video frames are acquired. YOLOv8s is used to identify targets in the video frames, and then DeepSORT is used to assign and maintain tracking numbers for the targets. Specifically, the real-time full-motion video (FMV) acquired by the UAV's onboard camera is used as input. This video image contains multiple moving and stationary targets. Since the camera is in motion during UAV flight, stationary targets have relative motion relative to the camera, making target tracking necessary. Target detection is performed on the FMV video frames using the YOLOv8s algorithm to obtain target information, including target bounding box location information, target category, and target confidence score. Since subsequent algorithms mainly correct the target's geographical location, target category and target confidence score are not used. The video frames and target information are then fed into the DeepSORT algorithm for target tracking, assigning and maintaining tracking numbers for the targets. When a target is first detected by the detector and matches successfully for several consecutive frames, the Deepsort tracker assigns a new tracking number. In subsequent frames, the tracker combines motion information and appearance features for matching, and the tracking number remains unchanged after a successful match. If the target leaves the frame or is occluded for a set period of time, the tracker marks it as deleted and releases the tracking number. The YOLOv8s algorithm and the DeepSORT algorithm are commonly used algorithms, and their specific processes will not be detailed in this invention.

[0031] 2. Cross-frame fusion: Project targets from multiple frames onto keyframes and cluster them for unified numbering. Specifically, select a keyframe, assuming the current keyframe is the nth keyframe, denoted as KF(n). Perform correlation analysis on subsequent consecutive images. For images that successfully match the same keyframe KF(n), project the target bounding box information on that image onto the coordinate system of the keyframe KF(n), thus ensuring that the target trajectory is unaffected by viewpoint changes over a period of time. Images that fail to match are selected as the new keyframe KF(n+1). When selecting a new keyframe KF(n+1), for all video frames {F(1), F(2), F(3), ...} between KF(n) and KF(n+1), based on the homography matrices {H(1), H(2), H(3), ...} obtained by matching KF(n) with the video frames {F(1), F(2), F(3), ...} respectively, all targets in each video frame are projected onto the coordinate system of KF(n), thus obtaining a target set of all target information within the video segment between KF(n) and KF(n+1) in the same coordinate system. This target set and the keyframe KF(n) image are then fed into a clustering algorithm. For the target set that is successfully clustered, the target with the earliest discovery time is selected, and its target number assigned by the DeepSort algorithm is used as the target number of all targets in this target set, and the target numbers are further fused. Figure 2 As shown, the specific process of this cross-frame fusion is as follows: (1) Keyframe determination: If the current frame is the first frame of the video, then the current frame is a keyframe; if it is not the first frame of the video, then the current frame enters the matching process: (2) Matching keyframes with the current frame: The ORB algorithm is used to extract image features, namely Oriented FAST keypoints and BRIEF descriptors, from the current frame and keyframes. The BRIEF descriptors in the image features are used to match and judge the current frame with the keyframes.

[0032] (3) Successful Match: The RANSAC algorithm is used to estimate the homography matrix based on the Oriented FAST keypoints in the keyframe and current frame image features. Based on the homography matrix, the target bounding box position information of the target in the current frame can be projected to the keyframe coordinate system through inverse homography transformation. The projected target information is used to generate the keyframe target set, that is, the target set after all targets of all video frames that are successfully matched with the keyframe are projected to the keyframe coordinate system.

[0033] (4) Matching failure: Then determine the relationship between the interval between the key frame and the current frame and the set minimum interval threshold. When the interval is less than the threshold, that is, the interval is too small, it is considered that the drone shooting perspective changes frequently and the current segment has no reference value. The key frame is directly switched to the current frame. If the interval is equal to or exceeds the threshold, the current segment is considered to have reference value. Then the current key frame image and the target set are sent to the clustering algorithm, and the current frame is switched to the new key frame.

[0034] (5) Clustering Algorithm: On the keyframe, the K-means clustering algorithm is performed on the set of all projected target bounding box position information in the target set, dividing them into several classes based on the spatial distance of the targets on the image plane. For each class formed after clustering, the target that appears earliest in time, i.e., the target with the smallest video frame number, is found, and the tracking number of this target is used as the unified number of the entire class, and reassigned to each target in the class, improving the consistency and stability of number allocation in target tracking. Among them, ORB, RANSAC, and K-means algorithms are all commonly used algorithms, and the specific process will not be described in detail in this invention.

[0035] The specific process is as follows: Figure 3 As shown, the process is as follows: First, the center coordinates and dimensions of each target are extracted as features using the target set of the keyframe as input, and the optimal number of clusters K is determined using the elbow rule. Then, the K-means algorithm is used to spatially cluster the targets and detect whether there are any anomalies in the orientation values ​​of each target: if an orientation anomaly is found, the original tracking number of the target is retained; otherwise, all targets within the same cluster are uniformly assigned the earliest target number in that cluster. Finally, the fused target numbers are output, achieving the merging of the identities of neighboring targets on the keyframe and the preservation of the numbers of abnormal targets.

[0036] 3. Scene matching: Match keyframes with satellite maps to achieve georeferencing.

[0037] Satellite map-based scene matching target localization is a method that locates detected targets by matching keyframe images with satellite maps and combining this with geographic information from the maps. The specific process is as follows: Figure 4 As shown.

[0038] The specific implementation of this technology includes: (1) Data preprocessing: The keyframe image is orthorectified according to the aircraft multiplexing data to ensure matching accuracy; the satellite map is cropped from the multiplexing data to narrow down the matching area. (2) Scene matching: Based on the SuperPoint and SuperGlue algorithms, image feature points are extracted and registered on the preprocessed keyframe images and satellite map images to obtain the mapping matrix between the two images, namely the homography matrix.

[0039] (3) Positioning calculation: For the target set of the key frame, based on the homography matrix obtained by scene matching, the target box position information of each target in the key frame coordinate system can be projected to the satellite map through inverse homography transformation, thereby calculating the corresponding coordinates of the target on the satellite map. Then, combined with the map geographic information, the corresponding coordinates are transformed to obtain the latitude and longitude coordinates of the target, thus realizing the target positioning.

[0040] Among them, the SuperPoint algorithm and the SuperGlue algorithm are commonly used algorithms, and the specific process will not be described in detail in this invention.

[0041] 4. Trajectory Plotting: The target's latitude and longitude coordinates, calculated according to scene matching, are plotted on the satellite map. By connecting all targets with the same tracking number in the order of their discovery time, the target's motion trajectory can be visually generated on the satellite map. Because the targets are projected onto keyframes during cross-frame fusion, the motion state caused by the FMV video camera is eliminated. At this point, the trajectories of stationary targets largely overlap, while moving targets display their motion trajectory relative to the ground.

[0042] This invention achieves stable maintenance of long-term, multi-target identity IDs by integrating YOLOv8s detection, DeepSORT short-term tracking, multi-frame projection, and cluster association, thus avoiding tracking loss and ID number jumps.

[0043] By constructing an end-to-end automatic mapping pipeline from pixel coordinates (i.e., the target coordinate box location information generated by target detection based on the current frame) to geographic coordinates, the real-time generation and visualization of motion trajectories are realized, which significantly improves the generation speed and availability of intelligence products (target location information, category, confidence level and target motion trajectory) and solves the problem of delayed intelligence flow.

[0044] This invention introduces a scene matching mechanism between keyframes and satellite maps, improving the matching accuracy between video content and geographic coordinates, ensuring that the target positioning results have actual geographic reference value, and solving the problem of inaccurate geographic mapping.

[0045] As an embodiment of the present invention, the present invention also discloses a matching and positioning device based on UAV FMV video. The device is used to implement the method, specifically including: The detection module is used to detect targets in real-time full-motion video (FMV) captured by the UAV, and to track and number the targets. The fusion module is used to determine the full dynamic video (FMV) and obtain key frames. It projects all tracked targets in different video frames of the full dynamic video FMV onto the same key frame and numbers and fuses all targets in the key frame to form a fused target set. The matching module is used to perform scene matching between the keyframes and satellite maps; The mapping module is used to map the target set onto the matched satellite map according to the numbering, and generate the motion trajectory of the target.

[0046] As an embodiment of the present invention, the present invention also discloses a computer storage medium storing a computer program, the computer program being executed by a processor to implement the method described.

[0047] As an embodiment of the present invention, the present invention also discloses an electronic device, the electronic device comprising: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method.

[0048] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A matching and localization method based on UAV FMV video, characterized in that, The method includes the following steps: S1. Detect targets in the real-time full-motion video (FMV) captured by the UAV, and track and number the targets; S2. The full dynamic video (FMV) is judged to obtain key frames. All tracked targets in different video frames of the full dynamic video (FMV) are projected onto the same key frame, and all targets in the key frame are numbered and fused to form a fused target set. S3. Match the keyframes with the satellite map for scene representation; S4. Map the target set onto the matched satellite map according to the numbering to generate the motion trajectory of the target.

2. The method according to claim 1, characterized in that, In step S1, the YOLOv8s algorithm is used to perform target detection on the full dynamic video (FMV) to obtain target information in each video frame. The video frames and target information are then sent to the DeepSORT algorithm for target tracking, and the targets are tracked and numbered.

3. The method according to claim 1, characterized in that, The target information includes target bounding box location information, target category, and target confidence level.

4. The method according to claim 3, characterized in that, S2 includes: S21. Determine whether the current frame is the first video frame from the full dynamic video (FMV). S22. If it is the first video frame, then treat it as a keyframe; otherwise, treat it as the current frame. S23. Match the current frame with the keyframe; S23. If the match is successful, generate a keyframe target set; S24. If the matching fails, generate the merged target set.

5. The method according to claim 4, characterized in that, S23 specifically includes: S231. The homography matrix between the current frame and the keyframe is estimated using the RANSAC algorithm; S232. Project the target bounding box of the current frame into the coordinate system of the keyframe using the homography matrix; S233. Store the projection results into the keyframe target set.

6. The method according to claim 4, characterized in that, S24 specifically includes: S241. Determining the relationship between the interval between the key frame and the current frame and the set minimum interval threshold; S242. If the interval is less than the set minimum interval threshold, then directly switch the keyframe to the current frame; S243. If the gap is greater than or equal to the set minimum gap threshold, then perform clustering operations on the keyframe images and target sets, and at the same time, use the current frame as a new keyframe.

7. The method according to claim 1, characterized in that, S3 includes: S31. Preprocess the keyframe images and satellite maps; S32. Extract and register image feature points from the preprocessed keyframe image and satellite map to obtain the mapping matrix between the two; S33. Based on the mapping matrix and the keyframe target set, calculate the corresponding coordinates of the target in the satellite map to achieve the positioning of the target in the satellite map.

8. A matching and positioning device based on UAV FMV video, characterized in that, The apparatus is used to implement the method according to any one of claims 1-7, specifically including: The detection module is used to detect targets in real-time full-motion video (FMV) captured by the UAV, and to track and number the targets. The fusion module is used to determine the full dynamic video (FMV) and obtain key frames. It projects all tracked targets in different video frames of the full dynamic video FMV onto the same key frame and numbers and fuses all targets in the key frame to form a fused target set. The matching module is used to perform scene matching between the keyframes and satellite maps; The mapping module is used to map the target set onto the matched satellite map according to the numbering, and generate the motion trajectory of the target.

9. A computer storage medium, characterized in that, The medium stores a computer program, which is executed by a processor to implement the method according to any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method of any one of claims 1-7.

Images