Perception decision integration target detection and tracking method based on high and low perspective cooperation

Abstract

The application discloses a kind of perception decision integration target detection and tracking method based on high-low view angle cooperation, comprising: high-low video stream access and timing multimodal alignment based on network time synchronization protocol;Based on the backbone layer network target detection and depth feature extraction of CSPDarknet;Based on homography correction and geographic coordinate registration, cross-view space mapping;Based on the track association integration of unified geographic spatial distance measurement and bipartite graph matching;Based on the perception decision integration generation of multi-branch spatio-temporal graph convolution network ST-GCN.The application can solve the problem of feature collapse and matching recognition error rate soaring caused by the dependence of existing technology on re-identification depth appearance features in response to high-low pitch angle sharp crossing, realize the effect of uninterrupted and high-precision entity re-anchor in the target appearance distortion state across the scale deformation barrier.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a target detection and tracking method based on high and low viewpoint collaboration that integrates perception and decision-making. Background Technology

[0002] With the rapid development of the low-altitude economy and the upgrading of intelligent security needs, video surveillance systems in complex urban environments (such as core business districts, transportation hubs, and large parks) are evolving from "single-plane monitoring" (purely ground cameras) to a three-dimensional system of "high-point UAV patrol + low-point ground joint monitoring" (i.e., air-ground collaborative control). The "macroscopic unobstructed" view from the air-based perspective and the "microscopic high-definition" view from the ground-based perspective have strong physical complementarity. Achieving collaborative target tracking and status fusion from both air-ground perspectives has become a core technology that urgently needs to be broken through in the security field.

[0003] When achieving target handover and region mapping across cameras and viewpoints, the industry currently mainly adopts the following three conventional mainstream methods. However, under the stringent scenario of "extremely large air-to-ground elevation difference collaboration," all of them have exposed fatal flaws:

[0004] 1) Re-identification (ReID) and feature matching methods based on deep learning visual features;

[0005] The target's identity is represented by high-dimensional visual features such as color, texture, and aspect ratio extracted by a convolutional neural network, and cross-camera handover is performed by comparing feature similarity (such as IoU cross ratio and appearance cosine distance) in the images of different cameras.

[0006] Limitations: Due to the extreme "perspective scaling and depth distortion" caused by the wide-angle top-down distance of high-altitude drones and the close-up level distance of ground cameras, the visual features (even aspect ratio) of the same target (such as a car) are completely different in the two images. Traditional feature comparison algorithms are prone to failure, resulting in the target trajectory being frequently cut off or misclassified as multiple different targets by the system because they "don't look alike".

[0007] 2) Inverse perspective mapping (IPM) technology based on the homography matrix;

[0008] By using a simple 3D downward linear projection matrix, we attempt to directly convert and map a specific 2D defense zone (ROI, region of interest) of a ground camera to the 3D view of a drone from high altitude, so as to achieve synchronous overlay of the region.

[0009] Limitations: When a drone is performing high-frequency, high-dynamic maneuvers in three-dimensional space (hovering, swaying, large-angle pitch and zoom), its effective field of view changes constantly. Once the vertex of the original ROI on the ground falls into the mathematical blind spot of the drone's field of view at this moment (such as above the horizon or in the reverse blind zone of the lens), the traditional linear algorithm will cause irreversible "singular value overflow (Infinity / NaN error)" because the projection denominator approaches zero. This will cause the warning red circle layer of the entire monitoring system to collapse instantly, become stringy, or appear as a full-screen garbled text obstruction.

[0010] 3) A high-concurrency video stream splicing method based on centralized cloud computing power;

[0011] The ultra-high-definition raw video streams from various front-end ground cameras are uniformly collected and transmitted back to a centralized cloud super server cluster. The central brain then merges and splices the streams before issuing business instructions.

[0012] Limitations: It consumes a great deal of valuable network bandwidth and generates high time latency during data packaging, uploading, cross-route queuing, cloud computing, and command feedback. It simply cannot meet the real-world emergency early warning needs in frontline grids, such as "immediately block fleeing vehicles ahead," which require ultra-low latency and millisecond-level response.

[0013] The aforementioned traditional multi-camera collaborative tracking systems have already reached the limits of their algorithmic capabilities when dealing with underlying architectural constraints such as severe perspective distortion, the disconnect between perception and decision-making, and the isolation of the two-dimensional pixel domain. Even more serious is the fact that when this outdated technology stack, heavily reliant on apparent features and centralized open-loop processing, is forcibly applied to next-generation 3D security scenarios with larger spatial spans and unprecedented dynamic complexity, its original underlying vulnerabilities completely evolve into a fatal systemic disaster. Specifically:

[0014] 1) Severe deformation across viewpoints leads to significant failure of feature-matching-based re-identification (Re-ID) algorithms: Existing multi-camera collaborative tracking algorithms (such as FairMOT and DeepSORT multi-camera extensions) mainly rely on deep learning networks to extract the two-dimensional image texture and appearance features of the target for cross-camera comparison. However, there are drastic differences in pitch angle and observation distance between high and low-angle cameras (for example, the same car appears as a clear front and side at a low angle, but only as a rectangular roof at a high angle). This structural deformation across viewpoints causes the matching success rate of Re-ID algorithms based on appearance difference measurement to plummet, making it difficult to achieve stable cross-camera archiving of the same entity.

[0015] 2) The "perception" and "decision-making" processes are isolated and disconnected, resulting in a lack of closed-loop control capabilities: Existing tracking frameworks are typically open-loop and unidirectional, passively outputting the target's bounding box and trajectory ID sequence. Such systems lack the ability to proactively adapt to the current perception situation (e.g., dynamically adjusting the PTZ parameters of the pan-tilt-zoom camera for local reconnaissance) or directly output scene-level control commands (e.g., intersection collision intervention signals). Serializing the perception results before inputting them to an independent decision-maker not only increases the end-to-end latency of the overall system transmission and response but also easily leads to the loss of implicit spatial topological features at the underlying level.

[0016] 3) Coordinate domain isolation prevents multi-source data from being aligned and fused in real physical space: Most current solutions only process tracking logic at the two-dimensional pixel coordinate level. The two-dimensional pixel images under different viewpoints have different representations, making it difficult to converge the entire set of targets under the same intuitive and clear three-dimensional real-world reference frame to calculate distance and motion vectors. This results in a lack of benchmark data support for deep multi-agent collaboration, trajectory prediction, and physical-level collision avoidance decisions.

[0017] Therefore, in order to overcome the numerous barriers such as "trajectory chain breakage caused by severe perspective distortion", "system collapse caused by dead point overflow of reverse perspective mapping" and "high response latency caused by the fragmentation of perception decision-making links", a new architectural paradigm is urgently needed that can abandon pure visual appearance dependence, connect the coordinates of the three-dimensional real physical world, and have ultra-low latency self-consistent fusion capability at the edge. This would break the traditional open-loop multi-lens tracking mindset and improve the ability to prevent distortion in cross-view area mapping, accurately track targets, and fuse decentralized edge states in collaboration between air and ground cameras. Summary of the Invention

[0018] To overcome the shortcomings of existing technologies, this invention provides an integrated perception and decision-making target detection and tracking method based on high and low viewpoint collaboration. This method includes: high and low viewpoint video stream access and temporal multimodal alignment based on a network time synchronization protocol; target detection and depth feature extraction based on the backbone layer of CSPDarknet; cross-viewpoint spatial mapping based on homography correction and geographic coordinate registration; trajectory association integration based on a unified geospatial distance metric and bipartite graph matching; and integrated perception and decision-making generation based on a multi-branch spatiotemporal graph convolutional network (ST-GCN). This invention addresses the problems in existing technologies where the reliance on re-identification of depth appearance features leads to feature collapse and a surge in matching error rates when dealing with rapid changes in elevation angles between high and low viewpoints. It achieves uninterrupted and highly accurate entity re-anchoring even under distorted target appearance conditions, overcoming scale deformation barriers.

[0019] The technical solution adopted by this invention to solve its technical problem is as follows:

[0020] Step 1: Accessing high- and low-bit video streams and aligning them with temporal multimodal modes based on network time synchronization protocols;

[0021] Data input: High-position camera RTSP pull source data Low-angle camera RTSP pull source data The system's internal standard clock signal ;

[0022] When high-position and low-position cameras capture image frames and input them into the network, they are stamped with local universal time standard timestamps. Then, on the central processing server, the Network Time Protocol (NTP) and time window filtering algorithm are used to perform frame-level alignment on videos from different sources, and image pairs with small matching errors and set thresholds are extracted for post-processing.

[0023] Data output: a sequence of high-bit image frame matrices that are synchronized in time. and the corresponding low-bit image frame matrix sequence ;

[0024] Step 2: Target detection and deep feature extraction based on the backbone network of CSPDarknet;

[0025] Data input: The sequence of high-bit image frame matrices output from step 1 With low-bit image frame matrix sequence ;

[0026] High-bit image frame matrix sequence With low-bit image frame matrix sequence After input, the data passes through the CSPDarknet feature extraction module and the feature pyramid multi-scale fusion module in sequence.

[0027] Without updating the weights of the preceding convolutional layer, the prediction decoupling head outputs the physical target bounding box coordinates and the target-level feature representation layer in the current frame.

[0028] The detection and feature extraction process can be abstractly represented as follows:

[0029]

[0030]

[0031] in, This represents the feature parsing and detection head network computation operations based on YOLO;

[0032] Data output: High-order detection box set and high-level detection confidence High-density feature metric Low-level detection box set and low-position detection confidence Low-level dense feature vector measurement ;

[0033] Step 3: Cross-view spatial mapping based on homography correction and geographic coordinate registration;

[0034] Data input: The set of high-order detection boxes obtained in step 2 With low-level detection box set ;

[0035] The dimension reduction transformation is performed by utilizing the geometric relationship of the camera's field of view on the ground plane; based on the preset physical calibration anchor points, the coordinates of the bottom pixel space of the two-dimensional image are transformed to a unified geospatial coordinate system using spatial perspective transformation; the unified geospatial coordinate system adopts GPS latitude and longitude coordinates or preset plane rectangular coordinates so that spatial references from different perspectives have a unified distance measurement basis.

[0036] For the pixel coordinates of the target grounding point, the spatial mapping relationship is expressed as follows:

[0037]

[0038] in:

[0039] : The 3x3 plane homography mapping matrix calculated by pre-extracting more than four calibration points in the field;

[0040] : The calculated longitude or absolute lateral distance coefficient before being divided by the scale factor in the unified geospatial coordinate system after mapping;

[0041] : The calculated latitude or absolute longitudinal distance coefficient value before dividing by the scale factor in a unified geospatial coordinate system after mapping;

[0042] : Scaling homogeneous scale parameter;

[0043] The actual longitude and latitudinal calculation results are as follows: and ;

[0044] Data output: High-level entity location set in a unified geospatial coordinate system and low-level entity location set ;

[0045] Step 4: Trajectory association and integration based on unified geospatial distance metric and bipartite map matching;

[0046] Data input: The set of high-level entity locations in the unified geospatial coordinate system obtained in step 3. and low-level entity location set and asynchronous message time difference ;

[0047] A state motion propagation module is constructed using Kalman filtering to predict historical motion directions. Then, the Hungarian algorithm is used to calculate and allocate the data, and trajectories with the same life cycle are obtained. Targets that are close to each other and are detected by different cameras are identified as the same object as long as their predicted and observed positions in a unified geographic spatial coordinate system meet the distance threshold constraint.

[0048] In the trajectory association process, matching and assigning cost metric matrices Used to indicate the first The existing trajectory and the first The association cost between new input projection points:

[0049]

[0050] in and They represent the first and second generations obtained after derivation using the Kalman wave filter, respectively. The tracking entity's longitudinal and latitudinal physical prediction points are determined; then, a minimum cost allocation scheme is solved using Hungarian matching to eliminate... False positive binding boundary, The physical meter-level deviation threshold;

[0051] Data output: Globally fused tracking trajectory set after correlation and integration ;

[0052] Step 5: Integrated generation of perception and decision-making based on ST-GCN (Spontaneous-Geographed Graph Convolutional Network with Multi-Branch Branches);

[0053] Data input: The global fusion tracking trajectory set obtained in step 4 ;

[0054] extract Sequence data is used to construct a target object graph network; each tracked entity is mapped as a node, and the geographical distance relationship and relative velocity vector between entities are defined as the edges of the graph; the input is fed into a graph feature decision extractor composed of a 3-layer spatial structure aggregation and a temporal series network to extract dynamic topological change patterns, and then the output action probability set is performed by a state machine soft decoding through a fully connected layer;

[0055] Data output: Final tracking-decision integrated machine instruction signal sequence .

[0056] Preferably, step 1 specifically comprises:

[0057] Step 1.1: Timestamp marking and edge node preprocessing;

[0058]

[0059] in This means that when an image frame is fed into the network, the video stream attributes are stripped away, and only the source information is retained; and These represent the timestamps assigned to the high-bit and low-bit image frames when they are accessed at the edge node, respectively. and These represent video frames from the high-order and low-order bits, respectively. This represents the preprocessed high-bit video image frame; This represents the preprocessed low-bit video image frame;

[0060] Step 1.2: Dynamic time window filtering and strict multimodal frame alignment;

[0061]

[0062] in, The maximum allowable time deviation threshold, This represents a dynamic time window filtering matrix, used to pair and align high-bit and low-bit video frames under time deviation constraints.

[0063] Preferably, step 2 specifically comprises:

[0064] Step 2.1: Extraction and fusion of multi-scale feature pyramids;

[0065]

[0066] in, This indicates the feature extraction module, which is used to extract deep semantic features from images; This indicates the feature pyramid multi-scale fusion module, used to complete multi-scale feature fusion; This represents a measure of high-density features. This represents a measure of low-order dense features;

[0067] Step 2.2: Target solution based on the predictive decoupling head;

[0068]

[0069] in and These represent the classification confidence scores from high-level and low-level perspectives, respectively. This represents the prediction decoupling head, used to separate the classification task from the bounding box regression task;

[0070] The final output of this step is: a set of high-order detection boxes. Confidence set and high-density feature vector measurement Low-level detection box set Confidence set and low-level dense feature vector measurement .

[0071] Preferably, step 3 specifically comprises:

[0072] Step 3.1: Extract the pixel coordinates of the target grounding point;

[0073]

[0074] in, This indicates the physical center point of the bottom edge of the target bounding box. This represents the target bounding box to be mapped. and These represent the horizontal and vertical pixel coordinates of the target's grounding point, respectively.

[0075] Step 3.2: Reverse perspective transformation and unified geospatial conversion;

[0076]

[0077]

[0078] in, This represents the lateral coordinate components in the unified geospatial coordinate system after mapping. This represents the mapped vertical coordinate components. This indicates the scaling homogeneous scale parameter. This is the 3×3 planar homography mapping matrix obtained by calculating using at least four pre-defined real-world fixed points. This indicates the target's mapped location in a unified geographic spatial coordinate system;

[0079] Step 3.3: Apply anti-perspective overflow regional anti-outbound geometric bisection approximation;

[0080] During the calculation That is, a certain point falls into the perspective error black hole. The starting point is the legal zone. implement:

[0081]

[0082] in, This represents the intermediate geographic spatial point obtained from the current binary search calculation; , and , These represent the starting point and the ending point respectively. direction and Coordinate components in the direction; Represents the horizontal coordinate components. Represents the vertical coordinate components;

[0083] Next, verify whether the inverse perspective projection back to the 2D field of view exceeds the boundary; if it does not exceed the boundary, then... Update the search range, otherwise backtrack, until... At this time This refers to the safety threshold corresponding to the outer edge of the field of view. and These represent the differences between the horizontal and vertical coordinates, respectively. This represents the convergence threshold.

[0084] Preferably, step 4 specifically comprises:

[0085] Step 4.1: Motion state deduction and physical space verification of the great circle;

[0086]

[0087]

[0088] in, and These represent the Kalman filters respectively. Predicting the first The location coordinates of each tracked entity in a unified geospatial coordinate system. Indicates the first The tracking target at time The geographical location; and These represent the observation points of the target to be correlated from high and low perspectives, respectively; Indicates the allowed spatial distance threshold; This represents the Haversine great circle distance calculation function, which is the actual surface distance between latitude and longitude points;

[0089] Step 4.2: Multi-terminal temporal nonlinear exponential trust fusion and discount calculation;

[0090]

[0091] in, This represents the time-weighted result after fusion. Indicates the number of data items participating in the fusion. Indicates the first Item detection confidence level, Indicates the time decay coefficient. This represents the target state value to be fused. This is a time-dependent nonlinear exponential decay term;

[0092] Step 4.3: Establishing the allocation cost matrix and integrating identity ID cleaning;

[0093]

[0094] in, , Indicates the coordinates of the new input point;

[0095] The Hungarian algorithm is used to find the global minimum cost and distance metric. The stable association result set, in which Indicates associated value. This indicates the preset distance threshold.

[0096] Preferably, step 5 specifically comprises:

[0097] Step 5.1: Establishing a dynamic intersection scenario and safe obstacle avoidance topology;

[0098]

[0099] in, This represents a scene topology graph consisting of a set of nodes and a set of edges.

[0100] For each object in the global fused identity tracking set, assign a graph node vector representation:

[0101]

[0102] in, This represents the horizontal coordinate of an object within a unified geographic spatial coordinate system. This represents the vertical coordinate of an object within a unified geographic coordinate system. Represents the velocity vector or motion state characteristics of an object;

[0103] For two nodes whose distance is less than a set threshold, calculate the feature vector of the connected boundary of the interaction graph:

[0104]

[0105] in, This represents the countdown parameter for the expected collision. This represents the inverse distance weight term constructed based on spatial distance;

[0106] Step 5.2: Multi-layer spatiotemporal graph feature network aggregation and capture;

[0107]

[0108] in, For the graph network Layer node distribution feature representation matrix For the graph network Layer node distribution feature representation matrix , It is the adjacency probability matrix after edge feature extraction. To control the normalized degree matrix, The graph convolution kernel contains a weight matrix of deep parameters. This is a nonlinear computation layer;

[0109] Step 5.3: Direct generation of integrated decisions based on fully connected decoding;

[0110]

[0111]

[0112] in, Represents the feedforward decision matrix. This represents the fully connected bias compensation coefficient. This represents the graph convolution aggregation feature operator. This represents the activation function for classification probability. This represents the probability matrix of the state response to a given set of output instructions. This indicates the decoding operation that takes the category with the highest response probability. This represents the final tracking-decision integrated machine instruction signal sequence.

[0113] Preferably, the high-position detection box set It includes categories and confidence levels.

[0114] Preferably, the maximum allowable time deviation threshold Set to 5 milliseconds.

[0115] The beneficial effects of this invention are as follows:

[0116] 1. Innovation in Structural Cognition and Collaborative Association: This invention adopts an association method based on a 3x3 planar homography matrix to map and register two-dimensional pixels of images with structural domain bias to a unified GPS real geographic rectangular coordinate system for distance matching. Therefore, it can solve the problem in existing technologies that are limited by the dependence on re-identification depth appearance features, and the feature collapse and the surge in matching and recognition error rate caused by the rapid change of elevation angle. It achieves the effect of providing uninterrupted and high-accuracy entity re-anchoring under the condition of target appearance distortion, even when crossing the scale deformation barrier.

[0117] 2. Closed-loop architecture and integrated generation innovation: This invention adopts a method of constructing a connected graph topology between tracking targets and continuously cascading a temporal spatial graph convolutional decision decoding network containing a three-layer information update architecture. Therefore, it can solve the problems of excessive end-to-end latency and lagging on-site situation response caused by the passive execution of perception functions and the fragmentation of tracking process and instruction planning calculation in traditional systems. It achieves a low-latency "perception-decision consistency closed loop" effect, where the front-end multi-target real-time trajectory structure directly triggers the hardware to actively adjust the early warning. Attached Figure Description

[0118] Figure 1 This is a flowchart of the method of the present invention.

[0119] Figure 2 This is a diagram of the method architecture of the present invention.

[0120] Figure 3 This is a flowchart for extracting target attributes.

[0121] Figure 4 This is a flowchart of cross-perspective ROI mapping.

[0122] Figure 5 It is a detailed ROI mapping diagram from a cross-perspective view. Detailed Implementation

[0123] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0124] This invention proposes a five-step end-to-end processing method that integrates multi-view spatiotemporal alignment, cross-domain geographic coordinate mapping, and graph neural network (GCN) collaboration. Specifically, it includes the following steps:

[0125] Step 1: Accessing high- and low-bit video streams and aligning them with temporal multimodal modes based on network time synchronization protocols;

[0126] This step is primarily responsible for acquiring the raw video streams from multiple sensors in the system's front-end hardware. High- and low-position cameras add local universal time standard timestamps to the captured image frames as they are fed into the network. Then, on the central processing server, the Network Time Protocol (NTP) and time window filtering algorithms are used to perform frame-level alignment of videos from different sources, extracting image pairs with minimal matching errors for post-processing.

[0127] Step 2: Target detection and deep feature extraction based on the backbone network of CSPDarknet;

[0128] This step deploys a single-stage object detection network (specifically using the YOLOv8 architecture). After the image pairs are input, they sequentially pass through the CSPDarknet feature extraction module and the Feature Pyramid (FPN) multi-scale fusion module. This step does not update a large number of weights in the preceding convolutional layers. The prediction decoupled head outputs the physical bounding box coordinates of the objects present in the current frame and the object-level feature representation layer.

[0129] Step 3: Cross-view spatial mapping based on homography correction and geographic coordinate registration;

[0130] To address the technical challenge of appearance features collapsing due to abrupt changes in elevation and pitch angles, this step skips image texture comparison and utilizes the specific geometric relationships of the camera's field of view on the ground plane for dimensionality reduction transformation. Based on preset physical calibration anchor points, the Perspective Transformation Block is used to transform the coordinates of the underlying pixel space of the 2D image to a unified geospatial coordinate system, typically using GPS latitude and longitude coordinates or preset Cartesian coordinates, thus giving spatial references from different perspectives equidistant meaning.

[0131] Step 4: Trajectory association integration based on unified Euclidean space metric and bipartite graph matching;

[0132] This step integrates and processes data within a unified geospatial coordinate system, enabling the evolution from detection response points to continuous lifecycle trajectory objects. A state motion propagation module is constructed using a Kalman filter to predict historical motion directions, and then the Hungarian algorithm is used for calculation and allocation to obtain trajectories with the same lifecycle. Targets detected by different cameras that are close to each other are considered the same object as long as their predicted and observed positions in the unified geospatial coordinate system meet a distance threshold constraint, regardless of changes in appearance or color.

[0133] Step 5: Integrated generation of perception and decision-making based on multi-branch spatiotemporal graph convolutional network (ST-GCN);

[0134] After the previous perception task (tracking) is completed, it seamlessly and directly drives the feedforward decision generation stage. Extraction Sequence data is used to construct a target object graph network. Each tracked entity is mapped to a node, and the geographical distance relationships and relative velocity vectors between entities are defined as edges of the graph. The input is fed into a graph feature decision extractor composed of a 3-layer spatial structure aggregation and a temporal series network to extract dynamic topological change patterns. Then, a state machine soft decoding process is performed through a fully connected layer to output an action probability set. For example, it determines whether to "activate the left low-position main camera for a local close-up" or "issue a regional collision avoidance warning event frame."

[0135] Example:

[0136] This embodiment addresses the problems of severe cross-viewpoint deformation, perception-decision discontinuity, and coordinate domain isolation in high- and low-position camera collaborative scenarios, proposing corresponding localization and decision-making methods, including the following steps:

[0137] Step 1: High and low position video stream access and temporal multimodal alignment based on network time synchronization protocol: Based on the edge decentralized multicast collaborative architecture, high position cameras and low position cameras complete image capture at the front-end edge nodes and construct high-precision timestamps through network time protocol; then, a dynamic time window filtering algorithm is used to perform frame-level alignment of videos from different sources, and isolated frames with excessive latency or mismatch are removed, thereby generating a strictly synchronized image frame matrix sequence in the time dimension.

[0138] The processing procedure is as follows:

[0139] Step 1.1: Timestamp marking and edge node preprocessing;

[0140]

[0141] The EdgeCapture function is responsible for stripping heavy video stream attributes when image frames are fed into the network, retaining only the core information source, and assigning a high-precision timestamp at the very beginning. and High and low edge nodes only use the UDP protocol for point-to-point blind transmission and reception, avoiding the high bandwidth consumption caused by centralized cloud server decoding.

[0142] Step 1.2: Dynamic time window filtering and strict multimodal frame alignment;

[0143]

[0144] in, In this embodiment, the maximum allowable time deviation threshold is set to 5 milliseconds.

[0145] This step calculates the absolute difference between the timestamps of the high-bit image and the low-bit image. When the difference is less than... At that time, these two frames are paired and entered into the post-processing flow to obtain a high-bit image frame matrix sequence that is synchronized with each other in time. and low-bit image frame matrix sequence .

[0146] Step 2: Object detection and deep feature extraction based on the backbone network of the YOLO architecture;

[0147] A single-stage object detection network (such as a network with CSPDarknet and FPN structures) is used to perform single-stage inference detection on the high-bit and low-bit images aligned in step 1. This step does not perform reverse updates of network weights, but instead predicts the two-dimensional bounding box coordinates of the target, classification confidence scores, and dense feature representations by the decoupling head.

[0148] The processing procedure is as follows:

[0149] Step 2.1: Extraction and fusion of multi-scale feature pyramids;

[0150]

[0151] This step utilizes CSPDarknet to extract deep semantic features of the image and performs multi-scale fusion through Feature Pyramid Network (FPN) to take into account both small targets in high-angle view and large targets in low-angle view.

[0152] Step 2.2: Target solution based on the predicted decoupled head;

[0153]

[0154] The DecoupledHead function separates the classification and regression tasks, improving the localization accuracy of the detection boxes.

[0155] Step 3: Cross-view spatial mapping based on homography correction and bisection approximation anti-distortion boundary repair;

[0156] Extract the center point of the bottom edge of the target (grounding point), and use the established spatial perspective transformation matrix to convert the two-dimensional bottom pixel coordinates into unified geospatial coordinates; the unified geospatial coordinates can be GPS latitude and longitude coordinates or preset plane rectangular coordinates; when the target projection exceeds the boundary and the denominator tends to zero and coordinate overflow occurs, the binary forcing back repair algorithm is used to accurately lock the critical point of the field of view.

[0157] The processing procedure is as follows:

[0158] Step 3.1: Extract the pixel coordinates of the target grounding point;

[0159]

[0160] The ExtractBasePoint function extracts the physical center point of the bottom edge of the target bounding box. Because its ground height is approximately zero, it is the optimal feature landing point for stable perspective mapping in a two-dimensional plane model.

[0161] Step 3.2: Reverse perspective transformation and unified geospatial conversion;

[0162]

[0163]

[0164] in This is a 3×3 plane homography mapping matrix calculated by fixing at least four real-world points.

[0165] Divide the homogeneous coordinate parameters by the scale factor Then, the target's location in a unified geospatial coordinate system can be obtained. .

[0166] Step 3.3: Apply anti-perspective overflow regional anti-outbound geometric bisection approximation;

[0167] During the calculation →0, meaning a point falls into the perspective anomaly endpoint. The starting point is the starting point of the legal region. ,implement:

[0168]

[0169] Next, verify whether the inverse perspective projection back to the 2D field of view exceeds the boundary; if it does not exceed the boundary, then... Update the search range, otherwise backtrack; until... At this time This is the safety threshold corresponding to the outer edge of the field of view.

[0170] Step 4: Cross-end trajectory association and integration based on spatiotemporal decay trust mechanism and optimal matching of bipartite graph;

[0171] The absolute great circle geographical distance is calculated using the unified geospatial coordinate system mentioned above. This is combined with the absolute difference in transmission and reception time of network uploads to achieve Gaussian exponential trust loss allocation. After eliminating false positives, the Hungarian allocation algorithm is used to achieve spatiotemporal continuous association and integration of identity IDs.

[0172] The processing procedure is as follows:

[0173] Step 4.1: Motion state deduction and physical space verification of the great circle;

[0174]

[0175]

[0176] The Kalman filter is used to predict the current latitude and longitude of the target in the previous frame; the Haversine formula is used to calculate the true spherical great circle right-angle distance between the two points, which is greater than a threshold. Remove the association directly.

[0177] Step 4.2: Multi-terminal temporal nonlinear exponential trust fusion and discount calculation;

[0178]

[0179] In the formula For the time-dependent nonlinear exponential decay term, since This could be additional states extracted at different resolutions of the target; as latency increases, this item will decrease sharply even if the base detection confidence is high. This step ensures that the most recent physical parameters (regardless of whether they are generated from high or low bits) have a stronger weighting.

[0180] Step 4.3: Establishing the allocation cost matrix and integrating identity ID cleaning;

[0181]

[0182] The Hungarian algorithm is used to find the global minimum cost and distance metric. A stable set of associated results.

[0183] Step 5: Integrated hard decoding and generation of perception and decision-making based on multi-branch spatiotemporal graph convolutional network (ST-GCN);

[0184] A spatial security topology graph is established using the tracked continuous entities as nodes and the spatial distance as the network connection edge. The collision prevention and pre-evolution entity patterns are captured through the three-layer aggregation of the spatiotemporal graph network. Then, the device control and risk mitigation instructions of the local linkage are decoded and released at the fully connected layer.

[0185] The processing procedure is as follows:

[0186] Step 5.1: Establishing a dynamic intersection scenario and safe obstacle avoidance topology;

[0187]

[0188] Assign a graph node vector representation to each object in the tracking set. For two nodes whose distance is less than a set threshold, calculate the feature vector of the connected boundary of the interaction graph:

[0189]

[0190] TTC stands for Time To Collision (the countdown parameter for the expected collision threshold).

[0191] Step 5.2: Multi-layer spatiotemporal graph feature network aggregation and capture;

[0192]

[0193] This embodiment involves a 3-layer stack.

[0194] Step 5.3: Direct generation of integrated decisions based on fully connected decoding;

[0195]

[0196]

[0197] Through the feedforward decision matrix With compensation bias (Based on pre-reinforcement learning of the behavioral safety model), the collision avoidance connectivity implicit perception information from the upper layer is directly projected to the device action probability layer in a reduced dimension. Finally, the command is truncated based on the maximum index value.

[0198] Experimental comparison:

[0199] The performance of the method of the present invention was compared with that of various existing methods, and the results are shown in Table 1:

[0200] Table 1 Performance Comparison Table

[0201]

[0202] The experimental results clearly demonstrate that, facing the severe challenges of extreme multi-scale and severely compressed appearance at high and low altitudes in this field, the present invention, by constructing a stable spatiotemporally unified geographic location anchoring relationship, has enabled the tracking stability performance parameter IDF1 to significantly increase from 66.7% of the traditional strongest solution to 81.5%. Simultaneously, relying on the unique end-to-end network direct connection map situational connectivity features, the FPS continues to maintain a strong real-time standard of 25 frames per second, while suppressing control latency to less than 40 milliseconds, successfully proving the technical feasibility of the present invention.

Claims

1. A target detection and tracking method based on high and low perspective collaborative perception and decision-making integration, characterized in that, Includes the following steps: Step 1: Accessing high- and low-bit video streams and aligning them with temporal multimodal modes based on network time synchronization protocols; Data input: High-position camera RTSP pull source data Low-angle camera RTSP pull source data The system's internal standard clock signal ; When high-position and low-position cameras capture image frames and input them into the network, they are stamped with local universal time standard timestamps. Then, on the central processing server, the Network Time Protocol (NTP) and time window filtering algorithm are used to perform frame-level alignment on videos from different sources, and image pairs with small matching errors and set thresholds are extracted for post-processing. Data output: a sequence of high-bit image frame matrices that are synchronized in time. and the corresponding low-bit image frame matrix sequence ; Step 2: Target detection and deep feature extraction based on the backbone network of CSPDarknet; Data input: The sequence of high-bit image frame matrices output from step 1 With low-bit image frame matrix sequence ; High-bit image frame matrix sequence With low-bit image frame matrix sequence After input, the data passes through the CSPDarknet feature extraction module and the feature pyramid multi-scale fusion module in sequence. Without updating the weights of the preceding convolutional layer, the prediction decoupling head outputs the physical target bounding box coordinates and the target-level feature representation layer in the current frame. The detection and feature extraction process can be abstractly represented as follows: in, This represents the feature parsing and detection head network computation operations based on YOLO; Data output: High-order detection box set and high-level detection confidence High-density feature metric Low-level detection box set and low-position detection confidence Low-level dense feature vector measurement ; Step 3: Cross-view spatial mapping based on homography correction and geographic coordinate registration; Data input: The set of high-order detection boxes obtained in step 2 With low-level detection box set ; The dimension reduction transformation is performed by utilizing the geometric relationship of the camera's field of view on the ground plane; based on the preset physical calibration anchor points, the coordinates of the bottom pixel space of the two-dimensional image are transformed to a unified geospatial coordinate system using spatial perspective transformation; the unified geospatial coordinate system adopts GPS latitude and longitude coordinates or preset plane rectangular coordinates so that spatial references from different perspectives have a unified distance measurement basis. For the pixel coordinates of the target grounding point, the spatial mapping relationship is expressed as follows: in: : The 3x3 plane homography mapping matrix calculated by pre-extracting more than four calibration points in the field; : The calculated longitude or absolute lateral distance coefficient before being divided by the scale factor in the unified geospatial coordinate system after mapping; : The calculated latitude or absolute longitudinal distance coefficient value before dividing by the scale factor in a unified geospatial coordinate system after mapping; : Scaling homogeneous scale parameter; The actual longitude and latitudinal calculation results are as follows: and ; Data output: High-level entity location set in a unified geospatial coordinate system and low-level entity location set ; Step 4: Trajectory association and integration based on unified geospatial distance metric and bipartite map matching; Data input: The set of high-level entity locations in the unified geospatial coordinate system obtained in step 3. and low-level entity location set and asynchronous message time difference ; A state motion propagation module is constructed using Kalman filtering to predict historical motion directions. Then, the Hungarian algorithm is used to calculate and allocate the data, and trajectories with the same life cycle are obtained. Targets that are close to each other and are detected by different cameras are identified as the same object as long as their predicted and observed positions in a unified geographic spatial coordinate system meet the distance threshold constraint. In the trajectory association process, matching and assigning cost metric matrices are performed. Used to indicate the first The existing trajectory and the first The association cost between new input projection points: in and They represent the first and second generations obtained after derivation using the Kalman wave filter, respectively. The tracking entity's longitudinal and latitudinal physical prediction points are determined; then, a minimum cost allocation scheme is solved using Hungarian matching to eliminate... False positive binding boundary, The physical meter-level deviation threshold; Data output: Globally fused tracking trajectory set after correlation and integration ; Step 5: Integrated generation of perception and decision-making based on ST-GCN (Spontaneous-Geographed Graph Convolutional Network with Multi-Branch Branches); Data input: The global fusion tracking trajectory set obtained in step 4 ; extract Sequence data is used to construct a target object graph network; each tracked entity is mapped as a node, and the geographical distance relationship and relative velocity vector between entities are defined as the edges of the graph; the input is fed into a graph feature decision extractor composed of a 3-layer spatial structure aggregation and a temporal series network to extract dynamic topological change patterns, and then the output action probability set is performed by a state machine soft decoding through a fully connected layer; Data output: Final tracking-decision integrated machine instruction signal sequence .

2. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Timestamp marking and edge node preprocessing; in This means that when an image frame is fed into the network, the video stream attributes are stripped away, and only the source information is retained; and These represent the timestamps assigned to the high-bit and low-bit image frames when they are accessed at the edge node, respectively. and These represent video frames from the high-order and low-order bits, respectively. This represents the preprocessed high-bit video image frame; This represents the preprocessed low-bit video image frame; Step 1.2: Dynamic time window filtering and strict multimodal frame alignment; in, The maximum allowable time deviation threshold, This represents a dynamic time window filtering matrix, used to pair and align high-bit and low-bit video frames under time deviation constraints.

3. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 2, characterized in that, Step 2 specifically involves: Step 2.1: Extraction and fusion of multi-scale feature pyramids; in, This indicates the feature extraction module, which is used to extract deep semantic features from images; This indicates the feature pyramid multi-scale fusion module, used to complete multi-scale feature fusion; This represents a measure of high-density features. This represents a measure of low-order dense features; Step 2.2: Target solution based on the predictive decoupling head; in and These represent the classification confidence scores from high-level and low-level perspectives, respectively. This represents the prediction decoupling head, used to separate the classification task from the bounding box regression task; The final output of this step is: a set of high-order detection boxes. Confidence set and high-density feature vector measurement Low-level detection box set Confidence set and low-level dense feature vector measurement .

4. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 3, characterized in that, Step 3 specifically involves: Step 3.1: Extract the pixel coordinates of the target grounding point; in, This indicates the physical center point of the bottom edge of the target bounding box. This represents the target bounding box to be mapped. and These represent the horizontal and vertical pixel coordinates of the target grounding point, respectively. Step 3.2: Reverse perspective transformation and unified geospatial conversion; in, This represents the lateral coordinate components in the unified geospatial coordinate system after mapping. This represents the mapped vertical coordinate components. This indicates the scaling homogeneous scale parameter. This is the 3×3 planar homography mapping matrix obtained by calculating using at least four pre-defined real-world fixed points. This indicates the target's mapped location in a unified geographic spatial coordinate system; Step 3.3: Apply anti-perspective overflow regional anti-outbound geometric bisection approximation; During the calculation That is, a certain point falls into the perspective error black hole. The starting point is the legal zone. implement: in, This represents the intermediate geographic spatial point obtained from the current binary search calculation; , and , These represent the starting point and the ending point respectively. direction and Coordinate components in the direction; Represents the horizontal coordinate components. Represents the vertical coordinate components; Next, verify whether the inverse perspective projection back to the 2D field of view exceeds the boundary; if it does not exceed the boundary, then... Update the search range, otherwise backtrack, until... At this time This refers to the safety threshold corresponding to the outer edge of the field of view. and These represent the differences between the horizontal and vertical coordinates, respectively. This represents the convergence threshold.

5. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 4, characterized in that, Step 4 specifically involves: Step 4.1: Motion state deduction and physical space verification of the great circle; in, and These represent the Kalman filters respectively. Predicting the first The location coordinates of each tracked entity in a unified geospatial coordinate system. Indicates the first The tracking target at time The geographical location; and These represent the observation points of the target to be correlated from high and low perspectives, respectively; Indicates the allowed spatial distance threshold; This represents the Haversine great circle distance calculation function, which is the actual surface distance between latitude and longitude points; Step 4.2: Multi-terminal temporal nonlinear exponential trust fusion and discount calculation; in, This represents the time-weighted result after fusion. Indicates the number of data items participating in the fusion. Indicates the first Item detection confidence level, Indicates the time decay coefficient. This represents the target state value to be fused. This is a time-dependent nonlinear exponential decay term; Step 4.3: Establishing the allocation cost matrix and integrating identity ID cleaning; in, , Indicates the coordinates of the new input point; Using the Hungarian algorithm to find the global minimum cost and distance metric The stable association result set, in which Indicates associated value. This indicates the preset distance threshold.

6. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 5, characterized in that, Step 5 specifically involves: Step 5.1: Establishing a dynamic intersection scenario and safe obstacle avoidance topology; in, This represents a scene topology graph consisting of a set of nodes and a set of edges. For each object in the global fused identity tracking set, assign a graph node vector representation: in, This represents the horizontal coordinate of an object within a unified geographic spatial coordinate system. This represents the vertical coordinate of an object within a unified geographic coordinate system. Represents the velocity vector or motion state characteristics of an object; For pairs of nodes that are close to each other, calculate the feature vector of the connected boundary of the interaction graph: in, This represents the countdown parameter for the expected collision. This represents the inverse distance weight term constructed based on spatial distance; Step 5.2: Multi-layer spatiotemporal graph feature network aggregation and capture; in, For the graph network Layer node distribution feature representation matrix, For the graph network Layer node distribution feature representation matrix, , It is the adjacency probability matrix after edge feature extraction. To control the normalized degree matrix, The graph convolution kernel contains a weight matrix of deep parameters. This is a nonlinear computation layer; Step 5.3: Direct generation of integrated decisions based on fully connected decoding; in, Represents the feedforward decision matrix. This represents the fully connected bias compensation coefficient. This represents the graph convolution aggregation feature operator. This represents the activation function for classification probability. This represents the probability matrix of the state response to a given set of output instructions. This indicates the decoding operation that takes the category with the highest response probability. This represents the final tracking-decision integrated machine instruction signal sequence.

7. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 6, characterized in that, The high-position detection frame set It includes categories and confidence levels.

8. The integrated target detection and tracking method based on high and low perspective collaborative perception and decision-making as described in claim 7, characterized in that, The maximum allowable time deviation threshold Set to 5 milliseconds.

Images