Unmanned aerial vehicle perspective-based vehicle and pedestrian target recognition and tracking method and system

Abstract

The application provides a kind of based on unmanned aerial vehicle visual angle vehicle and people target identification and tracking method and system, it is related to unmanned aerial vehicle visual identification technical field, including: obtaining unmanned aerial vehicle image sequence, extract and fuse multi-scale and semantic features to identify target category;Target speed is compensated using unmanned aerial vehicle motion information;Target matching is carried out in combination with motion prediction and appearance similarity;According to the matching result, the identity of the target is associated and the motion trajectory is generated.The application effectively improves the identification accuracy and tracking stability of personnel and vehicle targets in the dynamic scene of unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) vision technology, and in particular to a method and system for human and vehicle target recognition and tracking based on the perspective of an UAV. Background Technology

[0002] In the field of UAV visual surveillance, the continuous and stable identification and tracking of ground moving targets, such as people and vehicles, is a key technology. Existing methods typically follow a workflow framework of "detection-feature extraction-data association".

[0003] However, drones inevitably generate their own motion during flight, including position translation and attitude changes, which causes pixel-level shifts in the background and target positions between consecutive frames that are not caused by the target's own motion. Existing technologies still have problems such as failing to effectively compensate for global pixel disturbances caused by the drone's own motion, often being relatively simple or lacking in feature utilization, and possibly relying only on single-scale appearance features or simple motion features. They also have limited discriminative power for different targets with significant scale changes, occlusion, or similar appearances. Summary of the Invention

[0004] This invention provides a method and system for human and vehicle target recognition and tracking based on the perspective of an unmanned aerial vehicle (UAV), which can at least solve some of the problems existing in the prior art.

[0005] A first aspect of this invention provides a method for human and vehicle target recognition and tracking based on an unmanned aerial vehicle (UAV) perspective, comprising:

[0006] The system acquires image frame sequences captured by the UAV and performs candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, it extracts multi-scale features and semantic features, performs bidirectional interpolation sampling on the multi-scale features to obtain alignment features, and obtains fusion features through channel splicing and convolution operations. It then performs nonlinear activation on the semantic features and the fusion features to obtain enhanced features and determines the category identifier.

[0007] The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity.

[0008] Based on the compensated motion speed, the predicted position and prediction covariance of each person target and vehicle target in the current frame are predicted in the next frame using the Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance. The semantic features of the target in the current frame and the target in the next frame are extracted and the appearance similarity is calculated. The cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity and the matching result is obtained by solving the Hungarian algorithm.

[0009] Based on the matching results, an identity association is established, and the coordinates of candidate areas belonging to the same person target or the same vehicle target are connected with the category identifier to generate a motion trajectory according to the identity association.

[0010] In one alternative implementation,

[0011] The process involves acquiring image frame sequences from a drone and performing candidate region localization to obtain candidate region coordinates. Multi-scale features and semantic features are then extracted based on these candidate region coordinates. Bidirectional interpolation sampling is performed on the multi-scale features to obtain alignment features, including:

[0012] A sequence of image frames captured by a drone is acquired, and convolutional feature extraction is performed on each frame in the sequence to obtain a feature map. Multiple anchor boxes of different scales and aspect ratios are generated at each spatial location of the feature map, and foreground and background classification is performed to obtain the foreground confidence. Bounding box coordinate regression is performed on each anchor box to obtain the bounding box regression offset. Multiple anchor boxes with the highest foreground confidence are selected, and the coordinates are adjusted according to the bounding box regression offset to obtain candidate regions. The center coordinates and bounding box coordinates of the candidate regions are extracted as the candidate region coordinates.

[0013] Based on the candidate region coordinates, candidate region image patches are extracted from the image frame sequence and multi-path parallel convolution processing is performed. Local detail features and global context features are extracted using convolution kernels with different receptive fields in the parallel convolution path to obtain the multi-scale features. The candidate region image patches are subjected to residual connection deep convolution to obtain high-dimensional semantic representation and serve as the semantic features.

[0014] The spatial resolutions of the local detail features and the global context features in the multi-scale features are calculated respectively, and the target resolution is determined. The spatial resolution of the local detail features is adjusted to the target resolution by performing bilinear down-interpolation, and the spatial resolution of the global context features is adjusted to the target resolution by performing bicubic up-interpolation, thus obtaining the alignment features.

[0015] In one alternative implementation,

[0016] The fused features are obtained through channel concatenation and convolution operations. Enhanced features are then obtained by applying nonlinear activation to the semantic features and the fused features, and category identifiers are determined, including:

[0017] The local detail features and global context features in the alignment features are concatenated in the channel dimension to obtain the concatenated features. The feature response intensity of the concatenated features at each spatial location is calculated and the sampling offset is determined. The sampling position of the convolution kernel is determined according to the sampling offset and convolution is performed to obtain multi-scale perception features. Global pooling and block local pooling are performed on the multi-scale perception features. Based on the pooling results and the preset gate unit, adaptive fusion weights are determined and the multi-scale perception features are weighted and summed to obtain the fused features.

[0018] Calculate the spatial similarity between the semantic features and the fused features and construct a spatial correlation matrix. Perform singular value decomposition on the spatial correlation matrix to obtain a left singular vector matrix and a right singular vector matrix and determine the dominant direction vector. Perform matrix multiplication on the fused features and the dominant direction vector to obtain projection coefficients. Reconstruct the fused features based on the projection coefficients to obtain a decoupled feature representation. Perform cross-modal mapping on the decoupled feature representation to obtain a semantic fusion feature and perform nonlinear activation mapping to obtain the enhanced feature.

[0019] The enhanced features are input into the classification head network to calculate the category prediction vector and solve for the category probability distribution. The category index with the largest probability value in the category probability distribution is selected as the category identifier.

[0020] In one alternative implementation,

[0021] The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity, including:

[0022] The system acquires the three-dimensional spatial coordinates of the UAV at consecutive moments and calculates the difference between the three-dimensional spatial coordinates at adjacent moments to obtain the position change. It also acquires the attitude information of the UAV at consecutive moments and calculates the difference between the attitude information at adjacent moments to obtain the attitude change. The position change is projected onto the image plane to obtain the translation pixel offset. The rotation angle and scaling ratio of the image plane are calculated based on the attitude change and converted into rotation pixel offset and scaling pixel offset. The pixel offset is obtained by combining the translation pixel offset. An affine transformation matrix is ​​constructed based on the pixel offset as the motion compensation matrix.

[0023] Extract the center coordinates of the bounding boxes of the personnel and vehicle targets corresponding to the category identifier in consecutive image frames, calculate the displacement vector of the center coordinates of the bounding boxes in consecutive image frames as the motion velocity vector, calculate the compensated velocity vector based on the motion velocity vector and the motion compensation matrix, determine the projection component of the compensated velocity vector in the main direction of the image plane, construct a projection matrix based on the projection component, and calculate the compensated motion velocity based on the compensated velocity vector and the projection matrix.

[0024] In one alternative implementation,

[0025] Based on the compensated motion velocity, the predicted position and prediction covariance of each person and vehicle target in the current frame are predicted in the next frame using a Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance, including:

[0026] The motion pattern features are obtained by sliding window sampling of the compensated motion speed and an adaptive state transition matrix is ​​constructed. The gradient change of the compensated motion speed between adjacent frames is calculated and a noise covariance matrix is ​​constructed. Based on the adaptive state transition matrix, the state propagation of the bounding box center coordinates of each person target and vehicle target in the current frame is performed to obtain a prior state estimate. Based on the adaptive state transition matrix and the noise covariance matrix, covariance propagation is performed to obtain a prior covariance estimate. The bounding box center coordinates and appearance feature vectors of each target in the detection result of the next frame are extracted to construct a joint observation matrix. Based on the prior covariance estimate and the joint observation matrix, the Kalman gain is calculated and the prior state estimate and the prior covariance estimate are updated to obtain the predicted position and the predicted covariance.

[0027] Extract the center coordinates of the bounding boxes of all detected personnel and vehicle targets in the next frame as the measured target positions and calculate the position difference vector between the measured and predicted positions. Perform eigenvalue decomposition on the predicted covariance to obtain eigenvalues ​​and eigenvectors and construct a regularized covariance matrix. Perform inverse transformation on the regularized covariance moments to obtain the inverse covariance matrix. Perform a weighted transformation on the position difference vector based on the inverse covariance matrix to obtain a weighted difference vector and calculate the inner product distance between the weighted difference vector and the position difference vector to obtain the Mahalanobis distance.

[0028] In one alternative implementation,

[0029] Semantic features of the target in the current frame and the target in the next frame are extracted and appearance similarity is calculated. A cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity, and the matching result is obtained by solving the Hungarian algorithm, including:

[0030] Multi-layer convolutional encoding is performed on the bounding box regions of each person target and vehicle target in the current frame and the next frame to obtain the semantic features of the current frame and the next frame. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the channel dimension to obtain the channel moment features of the current frame and the next frame. Channel similarity is obtained by calculating a similarity measure. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the spatial dimension to obtain the spatial moment features of the current frame and the next frame. Spatial similarity is obtained by calculating a similarity measure. The appearance similarity is calculated based on the channel similarity and the spatial similarity.

[0031] The Mahalanobis distance is converted into a position matching metric using a preset exponential function, and the appearance similarity is converted into an appearance matching metric using a preset activation function. The statistical dependency between the position matching metric and the appearance matching metric is calculated, and the position matching metric and the appearance matching metric are cost-based according to the statistical dependency to construct the cost matrix. The cost matrix is ​​marked with row minimum and column minimum values ​​to obtain an initial label matrix. Augmented path search is performed based on the initial label matrix, and the label values ​​are updated until all current frame targets and next frame targets are paired. The association pairing relationship between current frame targets and next frame targets is extracted based on the label matrix obtained from the last update to obtain the matching result.

[0032] In one alternative implementation,

[0033] Based on the matching results, an identity association is established, and the coordinates of candidate regions belonging to the same person or vehicle target are connected with the category identifier according to the identity association to generate a motion trajectory, including:

[0034] Extract the pairing relationship between the current frame target and the next frame target from the matching results, assign a unique identifier to each pairing relationship and bind it to the candidate area coordinates and category identifier of the corresponding personnel target or vehicle target, calculate the position change of the candidate area coordinates and the consistency of the category identifier between adjacent frames, and perform a reliability assessment on the unique identifier to obtain the identity confidence level. Based on the identity confidence level, filter the unique identifier to obtain valid identity identifiers and establish identity association relationships.

[0035] Based on the identity association, extract personnel or vehicle targets with the same valid identity identifier and obtain the corresponding candidate area coordinates and category identifiers to obtain the original coordinate sequence. Perform motion prediction on the original coordinate sequence to obtain the predicted coordinate sequence. Calculate the deviation between the original coordinate sequence and the predicted coordinate sequence. Correct the original coordinate sequence based on the deviation to obtain the corrected coordinate sequence and connect it with the corresponding category identifier to obtain the motion trajectory.

[0036] A second aspect of the present invention provides a human and vehicle target recognition and tracking system based on the perspective of an unmanned aerial vehicle (UAV), comprising:

[0037] The feature extraction module is used to acquire image frame sequences collected by the UAV and perform candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, multi-scale features and semantic features are extracted. The multi-scale features are subjected to bidirectional interpolation sampling to obtain alignment features. The fusion features are obtained through channel splicing and convolution operations. The semantic features and the fusion features are subjected to nonlinear activation to obtain enhanced features and determine the category identifier.

[0038] The velocity compensation module is used to obtain the position and attitude changes of the UAV, determine the pixel offset, and construct a motion compensation matrix. Based on the motion compensation matrix, the module performs affine transformation and vector projection on the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers to obtain the compensated motion velocity.

[0039] The target association module is used to predict the predicted position and prediction covariance of each person target and vehicle target in the current frame in the next frame based on the compensated motion speed using a Kalman filter algorithm, calculate the Mahalanobis distance between the predicted position and the measured target position in the next frame based on the prediction covariance, extract the semantic features of the target in the current frame and the target in the next frame and calculate the appearance similarity, construct a cost matrix based on the Mahalanobis distance and the appearance similarity and solve it using a Hungarian algorithm to obtain the matching result;

[0040] The trajectory generation module is used to establish identity associations based on the matching results and connect the coordinates of candidate areas belonging to the same person target or the same vehicle target with the category identifier to generate motion trajectories according to the identity associations.

[0041] A third aspect of the present invention provides an electronic device, comprising:

[0042] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke instructions stored in the memory to perform the aforementioned method.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0044] In this invention, by extracting multi-scale and semantic features of candidate regions and performing bidirectional interpolation sampling, channel splicing, and nonlinear activation, effective feature alignment and deep fusion are achieved, enhancing the representational ability of the features. This makes the discrimination of personnel and vehicle categories more accurate and reliable under complex backgrounds, target scale changes, and partial occlusion, providing high-quality initial detection results for subsequent tracking. A motion compensation matrix is ​​constructed using the position and attitude changes of the UAV, and affine transformation and vector projection are performed on the motion velocity vector of the detected target. This effectively compensates for the pixel-level offset caused by the UAV's own motion to target observation, reducing the tracking jitter and target loss risk caused by the UAV platform's violent maneuvers or high-frequency vibrations, and improving the stability of motion state estimation. Semantic feature similarity of target appearance is extracted and calculated, and a cost matrix is ​​constructed by integrating motion and appearance information. The optimal matching is solved using the Hungarian algorithm, achieving high-accuracy identity association in complex scenarios such as target intersection and brief occlusion, effectively suppressing identity jumps. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the human and vehicle target recognition and tracking method based on the perspective of an unmanned aerial vehicle (UAV) according to an embodiment of the present invention.

[0046] Figure 2 This is a flowchart of the high-order moment multidimensional matching tracking method for human and vehicle target recognition and tracking based on the perspective of an unmanned aerial vehicle (UAV) in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0049] Figure 1 This is a flowchart illustrating the human and vehicle target recognition and tracking method based on the perspective of an unmanned aerial vehicle (UAV) according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0050] The system acquires image frame sequences captured by the UAV and performs candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, it extracts multi-scale features and semantic features, performs bidirectional interpolation sampling on the multi-scale features to obtain alignment features, and obtains fusion features through channel splicing and convolution operations. It then performs nonlinear activation on the semantic features and the fusion features to obtain enhanced features and determines the category identifier.

[0051] The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity.

[0052] Based on the compensated motion speed, the predicted position and prediction covariance of each person target and vehicle target in the current frame are predicted in the next frame using the Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance. The semantic features of the target in the current frame and the target in the next frame are extracted and the appearance similarity is calculated. The cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity and the matching result is obtained by solving the Hungarian algorithm.

[0053] Based on the matching results, an identity association is established, and the coordinates of candidate areas belonging to the same person target or the same vehicle target are connected with the category identifier to generate a motion trajectory according to the identity association.

[0054] In one alternative implementation,

[0055] The process involves acquiring image frame sequences from a drone and performing candidate region localization to obtain candidate region coordinates. Multi-scale features and semantic features are then extracted based on these candidate region coordinates. Bidirectional interpolation sampling is performed on the multi-scale features to obtain alignment features, including:

[0056] A sequence of image frames captured by a drone is acquired, and convolutional feature extraction is performed on each frame in the sequence to obtain a feature map. Multiple anchor boxes of different scales and aspect ratios are generated at each spatial location of the feature map, and foreground and background classification is performed to obtain the foreground confidence. Bounding box coordinate regression is performed on each anchor box to obtain the bounding box regression offset. Multiple anchor boxes with the highest foreground confidence are selected, and the coordinates are adjusted according to the bounding box regression offset to obtain candidate regions. The center coordinates and bounding box coordinates of the candidate regions are extracted as the candidate region coordinates.

[0057] Based on the candidate region coordinates, candidate region image patches are extracted from the image frame sequence and multi-path parallel convolution processing is performed. Local detail features and global context features are extracted using convolution kernels with different receptive fields in the parallel convolution path to obtain the multi-scale features. The candidate region image patches are subjected to residual connection deep convolution to obtain high-dimensional semantic representation and serve as the semantic features.

[0058] The spatial resolutions of the local detail features and the global context features in the multi-scale features are calculated respectively, and the target resolution is determined. The spatial resolution of the local detail features is adjusted to the target resolution by performing bilinear down-interpolation, and the spatial resolution of the global context features is adjusted to the target resolution by performing bicubic up-interpolation, thus obtaining the alignment features.

[0059] Image frame sequences were acquired from a drone flying at an altitude between 80 and 120 meters. The image resolution was 3840×2160 pixels, and the frame rate was 30 frames per second. Convolutional feature extraction was performed on each frame of the acquired image frame sequence to obtain a feature map. A 5-layer convolutional neural network structure was used, with each layer having a 3×3 kernel size, a stride of 1, padding of 1, and a modified linear unit activation function. The first layer had 3 input channels and 64 output channels; the second layer had 64 input channels and 128 output channels; the third layer had 128 input channels and 256 output channels; the fourth layer had 256 input channels and 512 output channels; and the fifth layer had 512 input channels and 512 output channels. The spatial size of the convolutional feature map was 1 / 16 of the original image, i.e., 240×135 pixels.

[0060] Multiple anchor boxes with different scales and aspect ratios are generated at each spatial location in the feature map. The basic anchor box size is set to 32×32 pixels, the scale ratio is set to 0.5, 1.0, and 2.0, and the aspect ratio is set to 0.5, 1.0, and 1.5, for a total of 9 different combinations. Nine anchor boxes are generated at each spatial location, for a total of 240×135×9=291600 anchor boxes. For each anchor box, foreground and background classification is performed to obtain the foreground confidence score. This is achieved through two fully connected layers: the first layer has 256 neurons, and the second layer has 2 neurons, representing the binary classification confidence scores between the foreground and background. Simultaneously, bounding box coordinate regression is performed on each anchor box to obtain the bounding box regression offset. This is achieved through two fully connected layers: the first layer has 256 neurons, and the second layer has 4 neurons, representing the offsets of the center point coordinates and the width and height, respectively.

[0061] Based on the calculation results, the 300 anchor boxes with the highest foreground confidence were selected, and a non-maximum suppression algorithm was used with an intersection-over-union (IoU) threshold of 0.7, retaining 100 candidate regions. The coordinates of these candidate regions were adjusted based on the bounding box regression offset. The adjustment formula is as follows: the center coordinates of the candidate region equal the center coordinates of the anchor box plus the center coordinate offset multiplied by the anchor box width and height; the width and height of the candidate region equal the width and height of the anchor box multiplied by the exponential function value of the width and height offset. The center coordinates and bounding box coordinates were extracted from the adjusted candidate regions as candidate region coordinates, including four values: the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner.

[0062] Candidate region image patches are extracted from the image frame sequence based on candidate region coordinates. Each candidate region is cropped and scaled to a uniform size of 224×224 pixels. Multi-path parallel convolution processing is applied to the cropped image patches, with three parallel convolution paths designed: Path 1 uses a 3×3 convolution kernel to extract local detail features; Path 2 uses a 5×5 convolution kernel to extract medium-scale features; and Path 3 uses a 7×7 convolution kernel to extract global context features. Each path contains three convolutional layers with 256 kernels each, and the activation function is a rectified linear unit.

[0063] In the parallel convolutional paths, the 3×3 convolutional kernel in path one has a small receptive field, which is used to capture local details such as target edges and textures, and the output feature map size is 56×56 pixels; the 5×5 convolutional kernel in path two has a moderate receptive field, which is used to capture medium-scale features such as target parts, and the output feature map size is 28×28 pixels; the 7×7 convolutional kernel in path three has a large receptive field, which is used to capture global contextual features such as the overall outline of the target, and the output feature map size is 14×14 pixels.

[0064] Deep convolutions with residual connections are applied to candidate region image patches to obtain high-dimensional semantic representations as semantic features. The residual connections employ a 50-layer deep residual network structure, containing 16 residual blocks. Each residual block contains 3 convolutional layers and 1 identity mapping connection. The number of channels in the convolutional layers increases progressively from 64 to 2048, outputting a 2048-dimensional semantic feature vector. This semantic feature vector contains rich semantic information and can effectively distinguish between different categories of targets, such as people and vehicles.

[0065] The spatial resolutions of local detail features and global context features in the multi-scale feature set are calculated separately, and the target resolution is determined. The spatial resolution of local detail features is 56×56 pixels, the spatial resolution of global context features is 14×14 pixels, and the target resolution is set to 28×28 pixels. Bilinear downinterpolation is performed on the local detail features to adjust the spatial resolution from 56×56 to the target resolution of 28×28, and bicubic upinterpolation is performed on the global context features to adjust the spatial resolution from 14×14 to the target resolution of 28×28, resulting in aligned features.

[0066] In this embodiment, by constructing a multi-path parallel convolutional structure on candidate region image blocks and using convolutional kernels with different receptive fields in different paths to extract local detail features and global contextual features respectively, the problem of insufficient attention to local texture or overall semantic information by single-path convolutional networks is effectively compensated. This improves the recognition ability of complex backgrounds, occluded targets, and targets with morphological changes, and enhances the completeness and discriminative power of feature representation. By introducing a deep convolutional structure with residual connections to construct high-dimensional semantic representation of candidate regions, the gradient transfer ability and deep feature learning ability of the network are enhanced, improving the semantic discrimination ability and overall recognition accuracy of targets in complex scenes. By calculating the spatial resolution of local detail features and global contextual features and aligning them with the target resolution, and by using bilinear down-interpolation and bicubic up-interpolation to achieve spatial alignment of multi-scale features respectively, the problem of information fusion deviation caused by the resolution inconsistency between features of different scales is effectively solved, and the accuracy of target boundary representation and the stability of feature fusion are enhanced.

[0067] In one alternative implementation,

[0068] The fused features are obtained through channel concatenation and convolution operations. Enhanced features are then obtained by applying nonlinear activation to the semantic features and the fused features, and category identifiers are determined, including:

[0069] The local detail features and global context features in the alignment features are concatenated in the channel dimension to obtain the concatenated features. The feature response intensity of the concatenated features at each spatial location is calculated and the sampling offset is determined. The sampling position of the convolution kernel is determined according to the sampling offset and convolution is performed to obtain multi-scale perception features. Global pooling and block local pooling are performed on the multi-scale perception features. Based on the pooling results and the preset gate unit, adaptive fusion weights are determined and the multi-scale perception features are weighted and summed to obtain the fused features.

[0070] Calculate the spatial similarity between the semantic features and the fused features and construct a spatial correlation matrix. Perform singular value decomposition on the spatial correlation matrix to obtain a left singular vector matrix and a right singular vector matrix and determine the dominant direction vector. Perform matrix multiplication on the fused features and the dominant direction vector to obtain projection coefficients. Reconstruct the fused features based on the projection coefficients to obtain a decoupled feature representation. Perform cross-modal mapping on the decoupled feature representation to obtain a semantic fusion feature and perform nonlinear activation mapping to obtain the enhanced feature.

[0071] The enhanced features are input into the classification head network to calculate the category prediction vector and solve for the category probability distribution. The category index with the largest probability value in the category probability distribution is selected as the category identifier.

[0072] The local detail features and global context features in the alignment features are concatenated along the channel dimension to obtain the concatenated feature. The local detail features have 256 channels, the global context features have 512 channels, and the concatenated feature has 768 channels, while maintaining the spatial size of 28×28 pixels. The feature response intensity at each spatial location of the concatenated feature is calculated, and the response map is obtained by summing and averaging the feature channels. The response map has a size of 28×28 and a value range between 0 and 1. The sampling offset is determined based on the response intensity; the higher the response intensity, the larger the sampling offset, specifically within a range of ±3 pixels. The sampling offset is calculated in both the horizontal and vertical directions, forming a 28×28×2 offset matrix.

[0073] The sampling position of the convolutional kernel is determined based on the sampling offset, and convolution is performed to obtain multi-scale perceptual features. A 3×3 deformable convolutional kernel is used, and the sampling position of the convolutional kernel is dynamically adjusted according to the previously calculated offset matrix, enabling the convolutional kernel to adaptively focus on important features of the target region. The deformable convolution consists of three layers, with 768, 384, and 256 channels per layer, respectively. The kernel size is 3×3, and the activation function is a modified linear unit. The resulting multi-scale perceptual feature has a size of 28×28×256. The multi-scale perceptual feature can dynamically adjust the receptive field according to the feature importance at different locations, effectively improving the ability to recognize irregularly shaped targets.

[0074] Global pooling and block-based local pooling are performed on the multi-scale perceptual features. Global pooling uses average pooling to pool the 28×28×256 feature map into a 1×1×256 global feature vector. Block-based local pooling divides the feature map into 16 regions of 4×4 each, and average pooling is performed on each region individually to obtain a 16×1×256 local feature vector. Adaptive fusion weights are determined based on the pooling results and a pre-defined gating unit. The gating unit consists of two fully connected layers. The input is the concatenation of the global and local feature vectors, and the output is 17 weight coefficients, corresponding to the weights of the global and 16 local features. The first layer of the gating network has 512 neurons and uses a modified linear unit (MRU) activation function; the second layer has 17 neurons and uses a sigmoid function with an output value ranging from 0 to 1. The sum of all weight coefficients is 1. The multi-scale perceptual features are weighted and summed according to the calculated weight coefficients to obtain the fused feature with a size of 28×28×256.

[0075] The spatial similarity between semantic features and fused features is calculated, and a spatial correlation matrix is ​​constructed. The semantic features are 2048-dimensional vectors, and the fused features are 28×28×256. The fused features are flattened into a 784×256 matrix. A fully connected layer maps the semantic features from 2048 dimensions to 256 dimensions, ensuring that the number of channels in the semantic and fused features is consistent. The similarity between the two features is calculated, forming a 784×784 spatial correlation matrix, where each element represents the correlation strength between different spatial locations within the fused features. The similarity calculation uses a dot product operation, and the numerical values ​​are normalized to between 0 and 1.

[0076] Singular value decomposition (SVD) is performed on the spatial correlation matrix to obtain a left singular vector matrix, singular value vectors, and a right singular vector matrix. The singular values ​​are sorted from largest to smallest, and the singular vectors corresponding to the top 32 singular values ​​are selected as the dominant direction vectors based on their cumulative contribution rate. Matrix multiplication is performed between the fused features and the dominant direction vectors to obtain projection coefficients, which represent the components of the fused features in each dominant direction. The fused features are reconstructed based on the projection coefficients to obtain a decoupled feature representation with a size of 28×28×256. The decoupled feature representation can effectively separate different semantic attributes of the target, such as shape, texture, and pose, improving feature representation capabilities.

[0077] Semantic fusion features are obtained by performing cross-modal mapping on the decoupled feature representations. An attention mechanism is employed, using semantic features as the query vector and the decoupled feature representation as the key-value vector. Attention weights are calculated and summed. The attention module contains three projection matrices and one output matrix, with a hidden layer dimension of 512, eight heads, and an output feature dimension of 1024. The semantic fusion features are further processed through residual connections and layer normalization, followed by nonlinear activation mapping to obtain enhanced features. The nonlinear activation uses the GELU activation function, which, compared to the traditional modified linear unit, provides a smooth transition near zero and richer gradient information. The final enhanced feature is a 1024-dimensional vector.

[0078] The enhanced features are input into the classification head network to calculate the category prediction vector. The classification head network consists of two fully connected layers. The first layer has an input dimension of 1024 and an output dimension of 512, with a modified linear unit activation function. The second layer has an input dimension of 512 and an output dimension equal to the number of target categories (10), including pedestrians, cars, trucks, buses, motorcycles, bicycles, tricycles, electric vehicles, tractors, and other categories. The category prediction vector is normalized using a softmax function to obtain a category probability distribution, where the sum of the probability values ​​for each category is 1. The category index with the highest probability value in the category probability distribution is selected as the category identifier.

[0079] In this embodiment, by performing global pooling and block-based local pooling on multi-scale perceptual features, and combining this with a preset gating unit to determine adaptive fusion weights, dynamic weight allocation for information at different scales is achieved. This enables automatic enhancement of key scale features and suppression of redundant information based on image content, thereby improving the rationality and discriminative ability of multi-scale information fusion. By calculating the spatial similarity between semantic features and fused features and constructing a spatial correlation matrix, the separation and modeling of the main changing trends in the feature space are achieved. This effectively eliminates coupling interference between different features, reduces redundant correlation, and improves the independence and structural clarity of feature representation.

[0080] In one alternative implementation,

[0081] The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity, including:

[0082] The system acquires the three-dimensional spatial coordinates of the UAV at consecutive moments and calculates the difference between the three-dimensional spatial coordinates at adjacent moments to obtain the position change. It also acquires the attitude information of the UAV at consecutive moments and calculates the difference between the attitude information at adjacent moments to obtain the attitude change. The position change is projected onto the image plane to obtain the translation pixel offset. The rotation angle and scaling ratio of the image plane are calculated based on the attitude change and converted into rotation pixel offset and scaling pixel offset. The pixel offset is obtained by combining the translation pixel offset. An affine transformation matrix is ​​constructed based on the pixel offset as the motion compensation matrix.

[0083] Extract the center coordinates of the bounding boxes of the personnel and vehicle targets corresponding to the category identifier in consecutive image frames, calculate the displacement vector of the center coordinates of the bounding boxes in consecutive image frames as the motion velocity vector, calculate the compensated velocity vector based on the motion velocity vector and the motion compensation matrix, determine the projection component of the compensated velocity vector in the main direction of the image plane, construct a projection matrix based on the projection component, and calculate the compensated motion velocity based on the compensated velocity vector and the projection matrix.

[0084] The system acquires the UAV's three-dimensional spatial coordinates at consecutive moments, represented in a north-northeast coordinate system, comprising eastward, northward, and altitude components, in meters. The UAV records its three-dimensional spatial coordinates every 33 milliseconds via an onboard positioning module, achieving a coordinate accuracy better than 0.05 meters. The position change is calculated by the difference between the three-dimensional spatial coordinates at adjacent moments, represented as a three-dimensional vector containing eastward, northward, and altitude displacement components. For example, at two adjacent moments, the UAV's three-dimensional spatial coordinates are [105.32, 204.67, 80.21] meters and [105.45, 204.82, 80.25] meters, respectively, resulting in a calculated position change of [0.13, 0.15, 0.04] meters.

[0085] The attitude information of the UAV is acquired at continuous intervals, represented by Euler angles, including three components: roll, pitch, and yaw, in degrees. Attitude information is recorded every 33 milliseconds by the UAV's attitude estimation system, with an angle accuracy better than 0.1 degrees. The attitude change is calculated by the difference between attitude information at adjacent time points, represented as a three-dimensional vector containing changes in roll, pitch, and yaw angles. For example, at two adjacent time points with attitude information of [2.3, 30.5, 120.7] degrees and [2.5, 30.8, 121.2] degrees, the calculated attitude change is [0.2, 0.3, 0.5] degrees.

[0086] The positional change is projected onto the image plane to obtain the translational pixel offset. Based on the UAV camera's intrinsic and extrinsic parameter matrices, the positional change in 3D space is converted into a translational pixel offset on the image plane. In the camera's intrinsic parameter matrix, the focal length is 3500 pixels, and the principal point coordinates are [1920, 1080] pixels. The calculated translational pixel offset is a two-dimensional vector representing the horizontal and vertical translation on the image plane. For example, the aforementioned positional change of [0.13, 0.15, 0.04] meters corresponds to a translational pixel offset of [4.2, 5.5] pixels.

[0087] The rotation angle and scaling factor of the image plane are calculated based on the attitude change, and then converted into rotation pixel offset and scaling pixel offset. The rotation angle is mainly affected by the change in yaw angle, and is calculated by multiplying the change in yaw angle by the cosine function value, where the independent variable is the pitch angle. The scaling factor is mainly affected by the changes in pitch angle and altitude, and is calculated by subtracting the change in altitude divided by the sum of the UAV altitude and the change in pitch angle multiplied by the tangent function value of the pitch angle. For example, when the change in yaw angle is 0.5 degrees, the pitch angle is 30.8 degrees, the change in altitude is 0.04 meters, and the UAV altitude is 80.25 meters, the calculated rotation angle is 0.43 degrees, and the scaling factor is 0.9994.

[0088] To convert the rotation angle into a rotation pixel offset, for each pixel in the image, calculate its vector relative to the image center, rotate this vector to obtain the rotated position, and the difference between the positions before and after rotation is the rotation pixel offset for that point. Similarly, to convert the scaling ratio into a scaling pixel offset, for each pixel in the image, calculate its vector relative to the image center, multiply this vector by the scaling ratio to obtain the scaled position, and the difference between the positions before and after scaling is the scaling pixel offset for that point. For a target with image center coordinates [1920, 1080] pixels and bounding box center coordinates [1500, 900] pixels, the calculated rotation pixel offset is [1.2, -0.9] pixels, and the scaling pixel offset is [-0.25, -0.11] pixels.

[0089] The pixel offset is obtained by combining the translation pixel offset, rotation pixel offset, and scaling pixel offset. The pixel offset is calculated as the vector sum of the three; for example, the vector sum of [4.2, 5.5] with [1.2, -0.9] and [-0.25, -0.11] is [5.15, 4.49] pixels. An affine transformation matrix is ​​constructed based on the pixel offset as the motion compensation matrix. The affine transformation matrix is ​​a 3×3 matrix, where the first two rows correspond to the linear transformation part, containing rotation and scaling information; the third row and third column are fixed at 1; and the first two rows and third column correspond to the translation component. For example, when the rotation angle is 0.43 degrees, the scaling ratio is 0.9994, and the translation pixel offset is [4.2, 5.5] pixels, the first two rows and first two columns of the constructed affine transformation matrix are [[0.9993, 0.0075], [-0.0075, 0.9993]], and the third column of the first two rows is [4.2, 5.5].

[0090] Extract the bounding box center coordinates of the personnel and vehicle targets corresponding to the category identifiers in consecutive image frames. The bounding box center coordinates are represented as two-dimensional vectors in pixels. For example, the bounding box center coordinates of a vehicle target in two consecutive image frames are [1500, 900] and [1512, 908] pixels, respectively. Calculate the displacement vector of the bounding box center coordinates in consecutive image frames as the motion velocity vector. The displacement vector is equal to the bounding box center coordinates of the next frame minus the bounding box center coordinates of the previous frame. For example, [1512, 908] minus [1500, 900] equals [12, 8] pixels. This vector represents the apparent motion of the target on the image plane. Considering an image frame rate of 30 frames per second, this translates to an actual velocity of [360, 240] pixels per second.

[0091] The compensated velocity vector is calculated based on the motion velocity vector and the motion compensation matrix. The motion compensation matrix is ​​applied to the motion velocity vector to remove the parallax effect caused by the drone's own motion, and the parallax vector calculated by the pixel offset is subtracted from the motion velocity vector. For example, when the motion velocity vector is [12, 8] pixels and the pixel offset is [5.15, 4.49] pixels, the compensated velocity vector is [6.85, 3.51] pixels, which translates to an actual velocity of [205.5, 105.3] pixels per second.

[0092] The projection component of the compensated velocity vector onto the principal direction of the image plane is determined. The principal direction of the image plane is determined by cluster analysis of the motion directions of multiple targets in the scene, and mainly includes two directions: horizontal and vertical. For road scenes, the horizontal direction usually corresponds to the road direction, and the vertical direction corresponds to the road's transverse direction. The projection component of the compensated velocity vector onto the principal direction is calculated, and the projection component is equal to the dot product of the velocity vector and the unit principal direction vector. For example, when the principal direction is the horizontal direction [1, 0], and the compensated velocity vector is [6.85, 3.51] pixels, the projection component is 6.85 pixels.

[0093] A projection matrix is ​​constructed based on the projection components, and the compensated motion velocity is calculated based on the compensated velocity vector and the projection matrix. The projection matrix is ​​a diagonal matrix, with diagonal elements corresponding to the main direction being 1 and other directions being 0. For example, when the main direction is horizontal, the diagonal elements of the projection matrix are [1, 0]. Multiplying the compensated velocity vector by the projection matrix yields the compensated motion velocity, which is the velocity component in the main direction. When the compensated velocity vector is [6.85, 3.51] pixels and the diagonal elements of the projection matrix are [1, 0], the compensated motion velocity is [6.85, 0] pixels, which translates to an actual velocity of 205.5 pixels per second.

[0094] In this embodiment, by extracting the center coordinates of the bounding boxes of personnel and vehicle targets in consecutive image frames and calculating the motion velocity vector, and then correcting the velocity vector using a motion compensation matrix, the true motion state of the target is separated and restored. This effectively eliminates false velocity components caused by platform motion, avoids misjudging the UAV's own motion as the target's motion, and improves the accuracy and reliability of motion estimation. By extracting the projection component of the compensated velocity vector in the main direction of the image plane and constructing a projection matrix to calculate the compensated motion velocity, the main motion trend of the target is accurately depicted, which can more accurately reflect the dynamic characteristics of the target in the main direction of travel and enhance the ability to analyze the target's behavioral trends.

[0095] In one alternative implementation,

[0096] Based on the compensated motion velocity, the predicted position and prediction covariance of each person and vehicle target in the current frame are predicted in the next frame using a Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance, including:

[0097] The motion pattern features are obtained by sliding window sampling of the compensated motion speed and an adaptive state transition matrix is ​​constructed. The gradient change of the compensated motion speed between adjacent frames is calculated and a noise covariance matrix is ​​constructed. Based on the adaptive state transition matrix, the state propagation of the bounding box center coordinates of each person target and vehicle target in the current frame is performed to obtain a prior state estimate. Based on the adaptive state transition matrix and the noise covariance matrix, covariance propagation is performed to obtain a prior covariance estimate. The bounding box center coordinates and appearance feature vectors of each target in the detection result of the next frame are extracted to construct a joint observation matrix. Based on the prior covariance estimate and the joint observation matrix, the Kalman gain is calculated and the prior state estimate and the prior covariance estimate are updated to obtain the predicted position and the predicted covariance.

[0098] Extract the center coordinates of the bounding boxes of all detected personnel and vehicle targets in the next frame as the measured target positions and calculate the position difference vector between the measured and predicted positions. Perform eigenvalue decomposition on the predicted covariance to obtain eigenvalues ​​and eigenvectors and construct a regularized covariance matrix. Perform inverse transformation on the regularized covariance moments to obtain the inverse covariance matrix. Perform a weighted transformation on the position difference vector based on the inverse covariance matrix to obtain a weighted difference vector and calculate the inner product distance between the weighted difference vector and the position difference vector to obtain the Mahalanobis distance.

[0099] Motion pattern features were obtained by sampling the compensated motion speed using a sliding window. The sliding window size was set to 5 frames, and the compensated motion speed of the target was extracted from 5 consecutive frames to form a motion temporal sequence. For example, the compensated motion speeds of a vehicle target in 5 consecutive frames were [205.5, 0.0], [204.8, 1.2], [206.3, -0.8], [205.9, 0.5], and [206.1, -0.3] pixels per second, respectively. Principal component analysis was performed on the aforementioned compensated motion speed data to extract the main motion pattern features. The analysis results show that the current target's motion is mainly concentrated in the horizontal direction, with a stable speed of approximately 206 pixels per second, and relatively small fluctuations in the vertical direction, belonging to a typical straight-line driving mode on a road.

[0100] An adaptive state transition matrix is ​​constructed, which describes the evolution of the target state from the current frame to the next frame. The state vector contains two parts: target position and velocity, with a dimension of 4, representing horizontal position, vertical position, horizontal velocity, and vertical velocity, respectively. The state transition matrix is ​​a 4×4 matrix, where the top-left 2×2 submatrix is ​​the identity matrix, the top-right 2×2 submatrix is ​​the time interval multiplied by the identity matrix, the bottom-left 2×2 submatrix is ​​the zero matrix, and the bottom-right 2×2 submatrix is ​​the adaptive velocity decay coefficient multiplied by the identity matrix. The time interval is the time difference between adjacent frames, typically 0.033 seconds. The velocity decay coefficient is dynamically adjusted according to the target's motion stability, approaching 1 when the motion is stable and decreasing to 0.8 when the motion changes drastically; for example, for the aforementioned vehicle target, the velocity decay coefficient is 0.98.

[0101] Calculate the gradient change of the compensated motion velocity between adjacent frames. The gradient change is equal to the current frame velocity minus the previous frame velocity. For example, the gradient change between the second and first frames is [204.8, 1.2] minus [205.5, 0.0], resulting in [-0.7, 1.2] pixels per second. Perform statistical analysis on the gradient changes across multiple consecutive frames, calculating their mean and variance. Construct a noise covariance matrix based on the gradient changes. The noise covariance matrix is ​​a 4×4 diagonal matrix, with the diagonal elements representing the position noise variance and velocity noise variance from top left to bottom right. The position noise variance is set to 2.0 square pixels, while the velocity noise variance is dynamically adjusted based on the statistical results of the gradient changes. For example, for the aforementioned vehicle target, the horizontal velocity noise variance is 1.2 square pixels per second, and the vertical velocity noise variance is 0.8 square pixels per second.

[0102] Based on the adaptive state transition matrix, a priori state estimate is obtained by performing state propagation on the bounding box center coordinates of each person and vehicle target in the current frame. For example, the state vector of a vehicle target in the current frame is [1500, 900, 206.1, -0.3], where the first two elements are the bounding box center coordinates in pixels; the last two elements are the compensated motion velocity in pixels per second. Multiplying the state vector by the state transition matrix yields the prior state estimate [1506.8, 899.99, 202.0, -0.29], indicating that the predicted position of the target in the next frame is [1506.8, 899.99] pixels, and the velocity is [202.0, -0.29] pixels per second.

[0103] The prior covariance estimate is obtained through covariance propagation based on the adaptive state transition matrix and the noise covariance matrix. The state covariance matrix of the current frame is a 4×4 diagonal matrix, with the diagonal elements from top left to bottom right representing the variances of horizontal position estimation, vertical position estimation, horizontal velocity estimation, and vertical velocity estimation, respectively, initially set to [4.0, 4.0, 4.0, 4.0]. Matrix operations are performed on the state transition matrix, the current state covariance matrix, and the noise covariance matrix to obtain the prior covariance estimate matrix, updating the diagonal elements to [10.2, 8.7, 5.1, 3.6], representing the uncertainty in position and velocity estimation.

[0104] A joint observation matrix is ​​constructed by extracting the bounding box center coordinates and appearance feature vectors of each target from the detection results of the next frame. The bounding box center coordinates are two-dimensional vectors, such as [1508, 901] pixels. The appearance feature vectors are extracted using a deep convolutional neural network with a dimension of 256 to capture the visual features of the target. The joint observation matrix has a dimension of 2×4, with the left 2×2 submatrix being the identity matrix and the right 2×2 submatrix being the zero matrix, indicating that only position information is observed, and velocity needs to be inferred through the state equation. The observation noise covariance matrix is ​​a 2×2 diagonal matrix with all diagonal elements being 3.0, representing the variance of the observation noise.

[0105] The Kalman gain is calculated based on the prior covariance estimate and the joint observation matrix. The Kalman gain is a 4×2 matrix, calculated using the prior covariance estimate, the joint observation matrix, and the observation noise covariance matrix. For example, for the vehicle target mentioned above, the calculated Kalman gain is [[0.63, 0.0], [0.0, 0.58], [0.32, 0.0], [0.0, 0.27]]. The prior state estimate and prior covariance estimate are updated based on the Kalman gain to obtain the predicted position and predicted covariance. The observed value is [1508, 901] pixels, and the difference between this observed value and the position portion of the prior state estimate [1506.8, 899.99] is [1.2, 1.01] pixels. Multiplying this difference by the Kalman gain and adding it to the prior state estimate yields the updated state estimate [1507.6, 900.6, 202.4, -0.02], where the first two elements represent the predicted position and the last two elements represent the predicted velocity. Simultaneously, the covariance matrix is ​​updated, with the diagonal elements decreasing to [3.8, 3.6, 3.5, 2.6], indicating a reduction in uncertainty after observation fusion.

[0106] Extract the center coordinates of the bounding boxes of all detected personnel and vehicle targets in the next frame as the measured target positions. For example, if three targets are detected in the next frame, with bounding box center coordinates of [1508, 901], [1200, 850], and [1700, 920] pixels, calculate the position difference vector between the measured target position and the predicted position [1507.6, 900.6], obtaining [0.4, 0.4], [-307.6, -50.6], and [192.4, 19.4] pixels, respectively.

[0107] The predicted covariance is decomposed using eigenvalues ​​and eigenvectors. The covariance at corresponding positions in the first 2×2 submatrices of the predicted covariance matrix is ​​decomposed to obtain eigenvalues ​​[3.8, 3.6] and corresponding eigenvectors. A regularized covariance matrix is ​​constructed; this is a 2×2 diagonal matrix where the diagonal elements are the square roots of the eigenvalues, i.e., [1.95, 1.9]. An inverse transformation is performed on the regularized covariance matrix to obtain the covariance inverse matrix, where the diagonal elements are the reciprocals of the eigenvalues, i.e., [0.26, 0.28].

[0108] The position difference vector is weighted by applying a weighted transformation based on the covariance inverse matrix to obtain a weighted difference vector. This position difference vector is then multiplied by the covariance inverse matrix; for example, for the first target, the weighted difference vector is [0.1, 0.11]. The Mahalanobis distance is obtained by calculating the inner product distance between the weighted difference vector and the position difference vector. For the three targets mentioned above, the calculated Mahalanobis distances are 0.09, 87.6, and 52.3, respectively. The smaller the Mahalanobis distance, the closer the measured and predicted positions are, and the greater the probability of a target match. For example, the first target with a Mahalanobis distance of 0.09 is clearly a correct match, while the other two targets are judged as different targets because their Mahalanobis distances are too large.

[0109] In this embodiment, a noise covariance matrix is ​​constructed by calculating the velocity gradient change between adjacent frames, enabling the system noise estimation to reflect the current motion fluctuation level. This achieves adaptive modeling of uncertainty, appropriately increasing the covariance estimation when the target motion changes drastically and reducing uncertainty when the motion is stable. This improves the rationality and stability of filter updates, reduces error accumulation, and performs state propagation and covariance propagation based on the adaptive state transition matrix. Combined with the joint observation matrix of the next frame, the Kalman gain is calculated to complete the state update, achieving joint correction of target position and appearance information. This improves the utilization rate of observation information, enhances the target re-identification capability under occlusion, intersection, and short-term loss conditions, and improves the robustness of multi-target tracking. By performing feature decomposition on the prediction covariance and constructing a regularized covariance matrix, and then obtaining the inverse covariance matrix through inverse transformation, the Mahalanobis distance is calculated by weighting the position difference vector. This achieves an accurate matching metric based on uncertainty constraints, fully considers the distribution characteristics of prediction errors, reasonably suppresses high uncertainty directions, improves the accuracy of data association, and reduces the mismatch rate.

[0110] In one alternative implementation,

[0111] Semantic features of the target in the current frame and the target in the next frame are extracted and appearance similarity is calculated. A cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity, and the matching result is obtained by solving the Hungarian algorithm, including:

[0112] Multi-layer convolutional encoding is performed on the bounding box regions of each person target and vehicle target in the current frame and the next frame to obtain the semantic features of the current frame and the next frame. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the channel dimension to obtain the channel moment features of the current frame and the next frame. Channel similarity is obtained by calculating a similarity measure. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the spatial dimension to obtain the spatial moment features of the current frame and the next frame. Spatial similarity is obtained by calculating a similarity measure. The appearance similarity is calculated based on the channel similarity and the spatial similarity.

[0113] The Mahalanobis distance is converted into a position matching metric using a preset exponential function, and the appearance similarity is converted into an appearance matching metric using a preset activation function. The statistical dependency between the position matching metric and the appearance matching metric is calculated, and the position matching metric and the appearance matching metric are cost-based according to the statistical dependency to construct the cost matrix. The cost matrix is ​​marked with row minimum and column minimum values ​​to obtain an initial label matrix. Augmented path search is performed based on the initial label matrix, and the label values ​​are updated until all current frame targets and next frame targets are paired. The association pairing relationship between current frame targets and next frame targets is extracted based on the label matrix obtained from the last update to obtain the matching result.

[0114] Multi-layer convolutional encoding is performed on the bounding box regions of each person and vehicle target in the current frame and the next frame to obtain the semantic features of the current frame and the next frame. The multi-layer convolutional encoding adopts a residual network structure, containing 5 residual blocks. Each residual block consists of two 3×3 convolutional layers and an identity mapping. The bounding box regions are adjusted to a uniform size of 224×224 pixels and input to the convolutional network. The first layer has 64 convolutional kernels, and the subsequent layers have 128, 256, 512, and 512 kernels respectively. After 5 residual blocks and a global average pooling layer, a 512-dimensional semantic feature vector is obtained. For example, the first 5 elements of the semantic feature vector of a vehicle target in the current frame are [0.72, -0.45, 0.18, -0.33, 0.91], and the first 5 elements of the semantic feature vector of the corresponding target in the next frame are [0.74, -0.43, 0.15, -0.36, 0.93].

[0115] Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame along the channel dimension to obtain the channel moment features of the current frame and the next frame. Higher-order moment statistics include calculating three statistical measures: mean, variance, and skewness. For a 512-dimensional semantic feature vector, 3×512-dimensional channel moment features are calculated. For example, the first three values ​​of the mean in the current frame's channel moment features are [0.22, -0.15, 0.08], the first three values ​​of the variance are [0.05, 0.03, 0.02], and the first three values ​​of the skewness are [0.12, -0.07, 0.04]. Channel similarity is obtained by calculating a similarity measure. The cosine similarity measure is used, which is the dot product of the channel moment feature vectors of the current frame and the next frame after normalization, resulting in a scalar value ranging from -1 to 1. For example, the channel similarity calculation result for the aforementioned vehicle target is 0.97, indicating a high degree of similarity in channel features.

[0116] Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the spatial dimension to obtain the spatial moment features of the current frame and the next frame. The higher-order moment statistics in the spatial dimension are obtained by analyzing the feature maps output by the convolutional layers. The feature map output by the last residual block is selected, with a size of 7×7×512. For each 7×7 feature map, the mean, variance, and skewness of the corresponding spatial distribution are calculated to obtain 3×512-dimensional spatial moment features. For example, the first three values ​​of the mean in the spatial moment features of the next frame are [0.35, 0.28, -0.12], the first three values ​​of the variance are [0.08, 0.06, 0.04], and the first three values ​​of the skewness are [0.15, -0.09, 0.05]. Spatial similarity is obtained through a similarity measure, also calculated using cosine similarity, to obtain a scalar value. For example, the spatial similarity calculation result for the aforementioned vehicle target is 0.95, indicating a high similarity in the spatial feature distribution.

[0117] Appearance similarity is calculated based on channel similarity and spatial similarity. The appearance similarity is a weighted average of channel similarity and spatial similarity, with the weights dynamically adjusted according to the target category. For vehicle targets, the channel similarity weight is 0.4, and the spatial similarity weight is 0.6; for personnel targets, the channel similarity weight is 0.6, and the spatial similarity weight is 0.4. For example, the appearance similarity of the aforementioned vehicle target is calculated as 0.4 × 0.97 + 0.6 × 0.95 = 0.958.

[0118] The Mahalanobis distance is converted into a positional matching metric using a predefined exponential function. The exponential function is the negative power of the Mahalanobis distance, where the base is a natural constant. For example, a target with a Mahalanobis distance of 0.09 will have a converted positional matching metric of 0.914; a target with a Mahalanobis distance of 87.6 will have a converted positional matching metric close to 0, indicating an extremely low positional matching degree. Appearance similarity is converted into an appearance matching metric using a predefined activation function. The activation function is the sigmoid function, which maps appearance similarity to between 0 and 1. For example, a target with an appearance similarity of 0.958 will have a converted appearance matching metric of 0.963, indicating a high appearance matching degree.

[0119] The statistical dependency between location matching and appearance matching metrics is calculated. By analyzing historical data, the Pearson correlation coefficient between the two metrics is calculated; a coefficient of 0.73 indicates a strong positive correlation. Based on the statistical dependency, the location matching and appearance matching metrics are cost-transformed, with a cost equal to 1 minus the corresponding matching metric. The location matching and appearance matching metrics are then multiplied by weighted coefficients and summed. These weighted coefficients are dynamically adjusted based on the correlation coefficient; the weight for location matching is 0.65, and the weight for appearance matching is 0.35. A cost matrix is ​​constructed, with dimensions equal to the number of targets in the current frame multiplied by the number of targets in the next frame. For example, if there are 3 targets in the current frame and 4 targets in the next frame, the cost matrix dimension is 3×4.

[0120] The initial label matrix is ​​obtained by marking the minimum value in each row and column of the cost matrix. For each row, the minimum cost value is found and marked as 1, with the remaining elements marked as 0; for each column, the minimum cost value is found and marked as 1, with the remaining elements marked as 0. For example, if a row of the cost matrix has values ​​[0.08, 0.78, 0.92, 0.65], and the minimum value is 0.08, then that row is marked as 1, and the remaining rows are marked as 0. If a row or column has no element marked as 1, an element is randomly selected and marked as 1. The resulting initial label matrix has the same dimension as the cost matrix, and all elements are either 0 or 1.

[0121] Augmenting path search and label value update are performed based on the initial label matrix. A Hungarian algorithm framework is employed, starting from the unmatched row and searching for feasible augmenting paths. An augmenting path is a path that starts from an unmatched row, passes through a series of alternating unlabeled and labeled edges, and reaches an unmatched column. Once an augmenting path is found, the label states on the path are inverted; that is, labels originally marked 1 are changed to 0, and labels originally marked 0 are changed to 1, thus adding a matching pair. This process is repeated until all targets in the current frame are paired with targets in the next frame, or no new augmenting paths can be found.

[0122] The matching results are obtained by extracting the association pairing relationship between the current frame target and the next frame target based on the tag matrix obtained from the last update. In the final tag matrix, each element marked as 1 corresponds to a matching pair, the row index corresponds to the current frame target number, and the column index corresponds to the next frame target number. If the number of targets in the current frame is less than that in the next frame, the extra targets in the next frame are considered newly appearing targets; if the number of targets in the current frame is more than that in the next frame, the unmatched targets in the current frame are considered disappeared targets. For example, if there are 3 targets in the current frame and 4 targets in the next frame, the final tag matrix is ​​a 3×4 matrix, where the positions of the element with a value of 1 are (0,0), (1,2), and (2,1), indicating that the first target in the current frame matches the first target in the next frame, the second target in the current frame matches the third target in the next frame, the third target in the current frame matches the second target in the next frame, and the fourth target in the next frame is a newly appearing target.

[0123] In this embodiment, by converting Mahalanobis distance into a location matching metric and appearance similarity into an appearance matching metric, and fusing them within a unified metric space, the matching weights can be adaptively adjusted based on the correlation between location and appearance. This reduces the impact of a single feature failure on the overall matching result, thereby improving the rationality and robustness of multi-feature fusion. By marking the minimum row and column values ​​of the cost matrix and combining it with augmented path search to achieve global optimal matching updates, the association between the target in the current frame and the target in the next frame is optimized and allocated under overall constraints. This avoids the erroneous association problem caused by local optima and improves the matching accuracy and consistency in multi-target scenarios.

[0124] Figure 2 This is a flowchart of the high-order moment multidimensional matching tracking method for human and vehicle target recognition and tracking based on the perspective of an unmanned aerial vehicle (UAV) in an embodiment of the present invention.

[0125] In one alternative implementation,

[0126] Based on the matching results, an identity association is established, and the coordinates of candidate regions belonging to the same person or vehicle target are connected with the category identifier according to the identity association to generate a motion trajectory, including:

[0127] Extract the pairing relationship between the current frame target and the next frame target from the matching results, assign a unique identifier to each pairing relationship and bind it to the candidate area coordinates and category identifier of the corresponding personnel target or vehicle target, calculate the position change of the candidate area coordinates and the consistency of the category identifier between adjacent frames, and perform a reliability assessment on the unique identifier to obtain the identity confidence level. Based on the identity confidence level, filter the unique identifier to obtain valid identity identifiers and establish identity association relationships.

[0128] Based on the identity association, extract personnel or vehicle targets with the same valid identity identifier and obtain the corresponding candidate area coordinates and category identifiers to obtain the original coordinate sequence. Perform motion prediction on the original coordinate sequence to obtain the predicted coordinate sequence. Calculate the deviation between the original coordinate sequence and the predicted coordinate sequence. Correct the original coordinate sequence based on the deviation to obtain the corrected coordinate sequence and connect it with the corresponding category identifier to obtain the motion trajectory.

[0129] The matching results are used to extract the pairing relationships between targets in the current frame and targets in the next frame. The matching results are stored in an association matrix, where a value of 1 indicates a match between the corresponding row and column targets, and a value of 0 indicates a mismatch. By traversing the association matrix and extracting all positions where the value is 1, the pairing relationships can be obtained. For example, if the element at position (2, 3) in the association matrix is ​​1, it means that target number 2 in the current frame is paired with target number 3 in the next frame. If there are 4 targets in the current frame and 5 targets in the next frame, the extracted pairing relationships are {(0, 1), (1, 0), (2, 3), (3, 4)}, indicating that target 0 in the current frame matches target 1 in the next frame, target 1 in the current frame matches target 0 in the next frame, target 2 in the current frame matches target 3 in the next frame, target 3 in the current frame matches target 4 in the next frame, and target 2 in the next frame is a newly appearing target.

[0130] Each pairing is assigned a unique identifier and bound to the candidate region coordinates and category identifier of the corresponding personnel or vehicle target. The identifier is an integer value, incrementing from 1. A new identifier is assigned to the first pairing; for pairings containing targets with existing identifiers, the existing identifier is retained. The candidate region coordinates are the coordinates of the four vertices of the target's bounding box, in the format [top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, bottom-right y-coordinate]. The category identifier is an integer value: 1 represents personnel, 2 represents small vehicles, and 3 represents large vehicles. For example, for pairing (0, 1), the candidate region coordinates of target 0 in the current frame are [230, 150, 270, 210], and the category identifier is 1; the candidate region coordinates of target 1 in the next frame are [235, 155, 275, 215], and the category identifier is 1. If the pairing relationship appears for the first time, an identity identifier 1 is assigned, forming a binding relationship {1: {current frame: (0, [230, 150, 270, 210], 1), next frame: (1, [235, 155, 275, 215], 1)}}.

[0131] The identity confidence score is obtained by calculating the positional change of candidate region coordinates between adjacent frames and the consistency of the category identifier, and then performing a reliability assessment on the unique identifier. The positional change is equal to the coordinates of the center point of the candidate region in the next frame minus the coordinates of the center point of the candidate region in the current frame, and then calculating their Euclidean distance. The consistency judgment of the category identifier checks whether the category identifiers of two frames are the same. The reliability assessment is based on two factors: positional change and category consistency. If the positional change is below a threshold, such as 20 pixels, the positional consistency is high; if the category identifiers are the same, the category consistency is high. The identity confidence score is calculated as a weighted average of the positional consistency score and the category consistency score, with weights of 0.7 and 0.3, respectively. For example, for the above pairing relationship, the positional change is 7.07 pixels, which is below the threshold, so the positional consistency score is 0.95; the category identifiers are the same, so the category consistency score is 1.0; the calculated identity confidence score is 0.7 × 0.95 + 0.3 × 1.0 = 0.965.

[0132] Valid identities are obtained by filtering unique identifiers based on identity confidence levels, and identity associations are established. An identity confidence threshold of 0.8 is set; identities with a confidence level higher than the threshold are considered valid. For example, identity identifier 1 has a confidence level of 0.965, which is higher than the threshold and is therefore a valid identity; identity identifier 4 has a confidence level of 0.73, which is lower than the threshold and is therefore an invalid identity. Establishing identity associations involves concatenating all frame target information corresponding to each valid identity. For example, valid identity identifier 1 is associated with target information in 10 consecutive frames, forming an association sequence {frame 1: (target number, coordinates, category), frame 2: (target number, coordinates, category), ..., frame 10: (target number, coordinates, category)}.

[0133] Based on identity association, personnel or vehicle targets with the same valid identity are extracted, and their corresponding candidate region coordinates and category identifiers are obtained and arranged to form the original coordinate sequence. For each valid identity, candidate region coordinates are extracted from all associated frames and arranged in frame order to form the original coordinate sequence. For example, for identity 1, candidate region coordinates of 10 consecutive frames are extracted: {[230, 150, 270, 210], [235, 155, 275, 215], ..., [265, 185, 305, 245]}, and the corresponding category identifier sequence: {1, 1, ..., 1}. Each element of the original coordinate sequence contains four values, representing the upper-left and lower-right coordinates of the target bounding box.

[0134] Motion prediction is performed on the original coordinate sequence to obtain the predicted coordinate sequence. A Kalman filter is used for motion prediction, and the state vector contains the target position and velocity. The state transition model is a uniform linear motion model. The initial state is the first two frames of the original coordinate sequence. The predicted values ​​for subsequent frames are obtained recursively through the state transition equation. For example, the bounding box center point coordinates of the first two frames of the original coordinate sequence are [250, 180] and [255, 185], and the calculation speed is [5, 5] pixels / frame. The predicted center point coordinates of the third frame are [260, 190]. The predicted center point coordinates are expanded to bounding box coordinates to obtain the predicted coordinate sequence. For example, the predicted bounding box coordinates of the third frame are [240, 160, 280, 220]. This process is repeated for all frames to obtain the complete predicted coordinate sequence.

[0135] Calculate the deviation between the original coordinate sequence and the predicted coordinate sequence. The deviation is equal to the Euclidean distance between the center points of the original and predicted coordinates. For example, if the center point of the original third frame bounding box is [258, 188] and the predicted center point is [260, 190], the deviation is 2.83 pixels. Calculate the deviation for all frames to obtain the deviation sequence. Perform statistical analysis on the deviation sequence to calculate the mean deviation and standard deviation, which are used to determine the smoothness of the trajectory.

[0136] The original coordinate sequence is corrected based on the deviation to obtain a corrected coordinate sequence, which is then combined with the corresponding category identifier to form the motion trajectory. The correction process uses a weighted average method, where the weights are inversely proportional to the deviation; the larger the deviation, the smaller the weight of the original coordinates and the larger the weight of the predicted coordinates. A deviation threshold of 10 pixels is set. When the deviation is less than the threshold, the weight of the original coordinates is 0.8, and the weight of the predicted coordinates is 0.2; when the deviation is greater than the threshold, the weight of both the original and predicted coordinates is 0.5. For example, in the third frame, the deviation is 2.83 pixels, which is less than the threshold. The corrected bounding box coordinates are 0.8 × [240, 160, 280, 220] + 0.2 × [242, 162, 282, 222] = [240.4, 160.4, 280.4, 220.4]. The correction is applied to all frames to obtain the corrected coordinate sequence. The corrected coordinate sequence is then combined with the category identifier sequence to form a complete motion trajectory representation. For example, the motion trajectory of identity identifier 1 is {frame 1: ([230, 150, 270, 210], 1), frame 2: ([235, 155, 275, 215], 1), ..., frame 10: ([265.4, 185.4, 305.4, 245.4], 1)}.

[0137] Special cases encountered during tracking are handled. When a target is temporarily occluded, causing detection failure, the missing frames are filled using predicted coordinates, with a maximum of three consecutive missing frames allowed. For example, if a target is occluded and not detected in the 5th frame, the coordinates in the 5th frame are predicted using the state of the 4th frame and marked as predicted filling. When targets split or merge, such as when people get out of a vehicle, the event type is determined based on area changes and relative positional relationships, and the identification is adjusted accordingly. For example, if a target's area suddenly decreases by 30% and a new target appears nearby, it is determined as a splitting event, a new identification is assigned to the new target, and the splitting relationship is recorded in the association record.

[0138] In this embodiment, by calculating the positional change of candidate region coordinates between adjacent frames and the consistency of category identifiers, the reliability of unique identity identifiers is evaluated and valid identity identifiers are obtained through screening. This enables the identity verification process to have dynamic credibility judgment capabilities, and can adaptively evaluate the reliability of identity based on motion stability and category consistency, reducing the risk of false association and false inheritance, and improving the accuracy and stability of identity management. By extracting the original coordinate sequence of targets with the same valid identity identifiers and calculating the deviation in combination with motion prediction results for correction, the trajectory data is smoothly optimized, which can effectively eliminate trajectory fluctuations caused by detection noise and instantaneous jitter, improve the continuity and smoothness of the trajectory, and enhance the realistic expression of the motion path.

[0139] A second aspect of the present invention provides a human and vehicle target recognition and tracking system based on the perspective of an unmanned aerial vehicle (UAV), comprising:

[0140] The feature extraction module is used to acquire image frame sequences collected by the UAV and perform candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, multi-scale features and semantic features are extracted. The multi-scale features are subjected to bidirectional interpolation sampling to obtain alignment features. The fusion features are obtained through channel splicing and convolution operations. The semantic features and the fusion features are subjected to nonlinear activation to obtain enhanced features and determine the category identifier.

[0141] The velocity compensation module is used to obtain the position and attitude changes of the UAV, determine the pixel offset, and construct a motion compensation matrix. Based on the motion compensation matrix, the module performs affine transformation and vector projection on the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers to obtain the compensated motion velocity.

[0142] The target association module is used to predict the predicted position and prediction covariance of each person target and vehicle target in the current frame in the next frame based on the compensated motion speed using a Kalman filter algorithm, calculate the Mahalanobis distance between the predicted position and the measured target position in the next frame based on the prediction covariance, extract the semantic features of the target in the current frame and the target in the next frame and calculate the appearance similarity, construct a cost matrix based on the Mahalanobis distance and the appearance similarity and solve it using a Hungarian algorithm to obtain the matching result;

[0143] The trajectory generation module is used to establish identity associations based on the matching results and connect the coordinates of candidate areas belonging to the same person target or the same vehicle target with the category identifier to generate motion trajectories according to the identity associations.

[0144] A third aspect of the present invention provides an electronic device, comprising:

[0145] A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke instructions stored in the memory to perform the aforementioned method.

[0146] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0147] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human and vehicle target recognition and tracking based on the perspective of an unmanned aerial vehicle (UAV), characterized in that, include: The system acquires image frame sequences captured by the UAV and performs candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, it extracts multi-scale features and semantic features, performs bidirectional interpolation sampling on the multi-scale features to obtain alignment features, and obtains fusion features through channel splicing and convolution operations. It then performs nonlinear activation on the semantic features and the fusion features to obtain enhanced features and determines the category identifier. The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity. Based on the compensated motion speed, the predicted position and prediction covariance of each person target and vehicle target in the current frame are predicted in the next frame using the Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance. The semantic features of the target in the current frame and the target in the next frame are extracted and the appearance similarity is calculated. The cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity and the matching result is obtained by solving the Hungarian algorithm. Based on the matching results, an identity association is established, and the coordinates of candidate areas belonging to the same person target or the same vehicle target are connected with the category identifier to generate a motion trajectory according to the identity association.

2. The method according to claim 1, characterized in that, The process involves acquiring image frame sequences from a drone and performing candidate region localization to obtain candidate region coordinates. Multi-scale features and semantic features are then extracted based on these candidate region coordinates. Bidirectional interpolation sampling is performed on the multi-scale features to obtain alignment features, including: A sequence of image frames captured by a drone is acquired, and convolutional feature extraction is performed on each frame in the sequence to obtain a feature map. Multiple anchor boxes of different scales and aspect ratios are generated at each spatial location of the feature map, and foreground and background classification is performed to obtain the foreground confidence. Bounding box coordinate regression is performed on each anchor box to obtain the bounding box regression offset. Multiple anchor boxes with the highest foreground confidence are selected, and the coordinates are adjusted according to the bounding box regression offset to obtain candidate regions. The center coordinates and bounding box coordinates of the candidate regions are extracted as the candidate region coordinates. Based on the candidate region coordinates, candidate region image patches are extracted from the image frame sequence and multi-path parallel convolution processing is performed. Local detail features and global context features are extracted using convolution kernels with different receptive fields in the parallel convolution path to obtain the multi-scale features. The candidate region image patches are subjected to residual connection deep convolution to obtain high-dimensional semantic representation and serve as the semantic features. The spatial resolutions of the local detail features and the global context features in the multi-scale features are calculated respectively, and the target resolution is determined. The spatial resolution of the local detail features is adjusted to the target resolution by performing bilinear down-interpolation, and the spatial resolution of the global context features is adjusted to the target resolution by performing bicubic up-interpolation, thus obtaining the alignment features.

3. The method according to claim 1, characterized in that, The fused features are obtained through channel concatenation and convolution operations. Enhanced features are then obtained by applying nonlinear activation to the semantic features and the fused features, and category identifiers are determined, including: The local detail features and global context features in the alignment features are concatenated in the channel dimension to obtain the concatenated features. The feature response intensity of the concatenated features at each spatial location is calculated and the sampling offset is determined. The sampling position of the convolution kernel is determined according to the sampling offset and convolution is performed to obtain multi-scale perception features. Global pooling and block local pooling are performed on the multi-scale perception features. Based on the pooling results and the preset gate unit, adaptive fusion weights are determined and the multi-scale perception features are weighted and summed to obtain the fused features. Calculate the spatial similarity between the semantic features and the fused features and construct a spatial correlation matrix. Perform singular value decomposition on the spatial correlation matrix to obtain a left singular vector matrix and a right singular vector matrix and determine the dominant direction vector. Perform matrix multiplication on the fused features and the dominant direction vector to obtain projection coefficients. Reconstruct the fused features based on the projection coefficients to obtain a decoupled feature representation. Perform cross-modal mapping on the decoupled feature representation to obtain a semantic fusion feature and perform nonlinear activation mapping to obtain the enhanced feature. The enhanced features are input into the classification head network to calculate the category prediction vector and solve for the category probability distribution. The category index with the largest probability value in the category probability distribution is selected as the category identifier.

4. The method according to claim 1, characterized in that, The position and attitude changes of the UAV are obtained to determine the pixel offset and construct a motion compensation matrix. Based on the motion compensation matrix, the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers are subjected to affine transformation and vector projection to obtain the compensated motion velocity, including: The system acquires the three-dimensional spatial coordinates of the UAV at consecutive moments and calculates the difference between the three-dimensional spatial coordinates at adjacent moments to obtain the position change. It also acquires the attitude information of the UAV at consecutive moments and calculates the difference between the attitude information at adjacent moments to obtain the attitude change. The position change is projected onto the image plane to obtain the translation pixel offset. The rotation angle and scaling ratio of the image plane are calculated based on the attitude change and converted into rotation pixel offset and scaling pixel offset. The pixel offset is obtained by combining the translation pixel offset. An affine transformation matrix is ​​constructed based on the pixel offset as the motion compensation matrix. Extract the center coordinates of the bounding boxes of the personnel and vehicle targets corresponding to the category identifier in consecutive image frames, calculate the displacement vector of the center coordinates of the bounding boxes in consecutive image frames as the motion velocity vector, calculate the compensated velocity vector based on the motion velocity vector and the motion compensation matrix, determine the projection component of the compensated velocity vector in the main direction of the image plane, construct a projection matrix based on the projection component, and calculate the compensated motion velocity based on the compensated velocity vector and the projection matrix.

5. The method according to claim 1, characterized in that, Based on the compensated motion velocity, the predicted position and prediction covariance of each person and vehicle target in the current frame are predicted in the next frame using a Kalman filter algorithm. The Mahalanobis distance between the predicted position and the measured target position in the next frame is calculated based on the prediction covariance, including: The motion pattern features are obtained by sliding window sampling of the compensated motion speed and an adaptive state transition matrix is ​​constructed. The gradient change of the compensated motion speed between adjacent frames is calculated and a noise covariance matrix is ​​constructed. Based on the adaptive state transition matrix, the state propagation of the bounding box center coordinates of each person target and vehicle target in the current frame is performed to obtain a prior state estimate. Based on the adaptive state transition matrix and the noise covariance matrix, covariance propagation is performed to obtain a prior covariance estimate. The bounding box center coordinates and appearance feature vectors of each target in the detection result of the next frame are extracted to construct a joint observation matrix. Based on the prior covariance estimate and the joint observation matrix, the Kalman gain is calculated and the prior state estimate and the prior covariance estimate are updated to obtain the predicted position and the predicted covariance. Extract the center coordinates of the bounding boxes of all detected personnel and vehicle targets in the next frame as the measured target positions and calculate the position difference vector between the measured and predicted positions. Perform eigenvalue decomposition on the predicted covariance to obtain eigenvalues ​​and eigenvectors and construct a regularized covariance matrix. Perform inverse transformation on the regularized covariance moments to obtain the inverse covariance matrix. Perform a weighted transformation on the position difference vector based on the inverse covariance matrix to obtain a weighted difference vector and calculate the inner product distance between the weighted difference vector and the position difference vector to obtain the Mahalanobis distance.

6. The method according to claim 1, characterized in that, Semantic features of the target in the current frame and the target in the next frame are extracted and appearance similarity is calculated. A cost matrix is ​​constructed based on the Mahalanobis distance and the appearance similarity, and the matching result is obtained by solving the Hungarian algorithm, including: Multi-layer convolutional encoding is performed on the bounding box regions of each person target and vehicle target in the current frame and the next frame to obtain the semantic features of the current frame and the next frame. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the channel dimension to obtain the channel moment features of the current frame and the next frame. Channel similarity is obtained by calculating a similarity measure. Higher-order moment statistical analysis is performed on the semantic features of the current frame and the next frame in the spatial dimension to obtain the spatial moment features of the current frame and the next frame. Spatial similarity is obtained by calculating a similarity measure. The appearance similarity is calculated based on the channel similarity and the spatial similarity. The Mahalanobis distance is converted into a position matching metric using a preset exponential function, and the appearance similarity is converted into an appearance matching metric using a preset activation function. The statistical dependency between the position matching metric and the appearance matching metric is calculated, and the position matching metric and the appearance matching metric are cost-based according to the statistical dependency to construct the cost matrix. The cost matrix is ​​marked with row minimum and column minimum values ​​to obtain an initial label matrix. Augmented path search is performed based on the initial label matrix, and the label values ​​are updated until all current frame targets and next frame targets are paired. The association pairing relationship between current frame targets and next frame targets is extracted based on the label matrix obtained from the last update to obtain the matching result.

7. The method according to claim 1, characterized in that, Based on the matching results, an identity association is established, and the coordinates of candidate regions belonging to the same person or vehicle target are connected with the category identifier according to the identity association to generate a motion trajectory, including: Extract the pairing relationship between the current frame target and the next frame target from the matching results, assign a unique identifier to each pairing relationship and bind it to the candidate area coordinates and category identifier of the corresponding personnel target or vehicle target, calculate the position change of the candidate area coordinates and the consistency of the category identifier between adjacent frames, and perform a reliability assessment on the unique identifier to obtain the identity confidence level. Based on the identity confidence level, filter the unique identifier to obtain valid identity identifiers and establish identity association relationships. Based on the identity association, extract personnel or vehicle targets with the same valid identity identifier and obtain the corresponding candidate area coordinates and category identifiers to obtain the original coordinate sequence. Perform motion prediction on the original coordinate sequence to obtain the predicted coordinate sequence. Calculate the deviation between the original coordinate sequence and the predicted coordinate sequence. Correct the original coordinate sequence based on the deviation to obtain the corrected coordinate sequence and connect it with the corresponding category identifier to obtain the motion trajectory.

8. A human and vehicle target recognition and tracking system based on the perspective of an unmanned aerial vehicle (UAV), used to implement the method of any one of claims 1-7, characterized in that, include: The feature extraction module is used to acquire image frame sequences collected by the UAV and perform candidate region localization to obtain candidate region coordinates. Based on the candidate region coordinates, multi-scale features and semantic features are extracted. The multi-scale features are subjected to bidirectional interpolation sampling to obtain alignment features. The fusion features are obtained through channel splicing and convolution operations. The semantic features and the fusion features are subjected to nonlinear activation to obtain enhanced features and determine the category identifier. The velocity compensation module is used to obtain the position and attitude changes of the UAV, determine the pixel offset, and construct a motion compensation matrix. Based on the motion compensation matrix, the module performs affine transformation and vector projection on the motion velocity vectors of the personnel and vehicle targets corresponding to the category identifiers to obtain the compensated motion velocity. The target association module is used to predict the predicted position and prediction covariance of each person target and vehicle target in the current frame in the next frame based on the compensated motion speed using a Kalman filter algorithm, calculate the Mahalanobis distance between the predicted position and the measured target position in the next frame based on the prediction covariance, extract the semantic features of the target in the current frame and the target in the next frame and calculate the appearance similarity, construct a cost matrix based on the Mahalanobis distance and the appearance similarity and solve it using a Hungarian algorithm to obtain the matching result; The trajectory generation module is used to establish identity associations based on the matching results and connect the coordinates of candidate areas belonging to the same person target or the same vehicle target with the category identifier to generate motion trajectories according to the identity associations.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images