Event identification method and device for aerial video, equipment and medium

Abstract

The invention relates to an event identification method, device and equipment for an aerial video and a medium, and belongs to the technical field of video identification, and the event identification method for the aerial video comprises the following steps: carrying out global modeling on obtained unmanned aerial vehicle aerial video data based on a dynamic frame selection module to obtain a key frame sequence related to an event; performing feature extraction on the key frame sequence based on a spatial feature extraction module to obtain spatial appearance features of key frames; carrying out difference and correlation calculation between adjacent frames on the key frame sequence based on a motion enhancement module to obtain action motion features, and fusing the action motion features with the spatial appearance features to obtain a fused feature sequence; and performing global modeling on the fused feature sequence based on a multi-scale time relation modeling module to obtain event global time sequence features, classifying the event global time sequence features through a classifier to obtain an event category result, and improving the event identification accuracy and reasoning efficiency.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and in particular to an event recognition method, apparatus, device, and medium for aerial video. Background Technology

[0002] Computer vision plays a crucial role in the perception and understanding of drone video data. In various mission scenarios, using computer vision technology to analyze video data captured by drone cameras has become a key link in promoting the improvement of drone intelligence. Among them, aerial video event recognition, as an important direction for video content understanding, is of great significance for improving the drone's autonomous judgment and behavioral decision-making capabilities in complex environments. With the rapid development of deep learning technology, video recognition technology has made significant progress in many application scenarios, providing technical support for event recognition in drone aerial videos.

[0003] Due to the complexity and unique nature of aerial photography environments, existing methods still face numerous challenges in practical applications. On the one hand, aerial events are typically characterized by temporal sparsity and randomness, with a large number of redundant frames obscuring the core actions of the events. This makes it difficult for traditional uniform sampling or fixed-interval sampling strategies to effectively capture critical moments, resulting in wasted model training resources and missed event detections. On the other hand, issues such as viewpoint drift and frame misalignment caused by the movement of the drone camera itself severely interfere with the motion feature extraction process based on frame sequence modeling, making it difficult to stably express the temporal correlation information of events. Furthermore, most existing methods rely on static spatial information, making it difficult to simultaneously consider the modeling of local target motion changes and global temporal relationships. When faced with events that have strong temporal dependencies, small action amplitudes, or inconsistent durations, the recognition performance is insufficient to meet the needs of practical applications.

[0004] Therefore, current drone aerial video event recognition technology still has significant shortcomings in dealing with issues such as event temporal sparsity, inaccurate motion modeling, and inefficient keyframe selection, resulting in low accuracy in drone aerial video event recognition. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, apparatus, device and medium for event recognition of aerial video to solve the technical problem of low accuracy in event recognition of drone aerial video.

[0006] To address the aforementioned issues, in a first aspect, the present invention provides an event recognition method for aerial video, used to perform event recognition on drone aerial video based on a constructed event recognition model, wherein the event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module; The event recognition method for the aerial video includes: The dynamic frame selection module is used to perform global modeling on the acquired UAV aerial video data to obtain a sequence of key frames related to the event. Based on the spatial feature extraction module, feature extraction is performed on the key frame sequence to obtain the spatial appearance features of the key frames. Based on the motion enhancement module, the keyframe sequence is subjected to difference and correlation calculation between adjacent frames to obtain motion features. The motion features are then fused with the spatial appearance features to obtain a fused feature sequence. The multi-scale temporal relationship modeling module performs global modeling on the fused feature sequence to obtain global temporal features of the event. The global temporal features of the event are then classified by a classifier to obtain the event category results.

[0007] In one possible implementation, the dynamic frame selection module includes a feature extraction submodule and a policy network submodule, wherein the network structure of the feature extraction submodule is an X3D network; the step of performing global modeling on the acquired UAV aerial video data based on the dynamic frame selection module to obtain a sequence of keyframes related to the event includes: The drone aerial video data is preprocessed to obtain aerial video features; The X3D network is used to perform coarse-grained global feature extraction on the aerial video features to obtain global frame-level features; The policy network submodule is used to extract spatiotemporal features from the global frame-level features to obtain a key frame sequence related to the event. The quality of the key frame combination is estimated by multiple Monte Carlo sampling with the goal of minimizing the expected loss of the key frame combination in the key frame sequence.

[0008] In one possible implementation, the preprocessing of the UAV aerial video data to obtain aerial video features includes: The drone aerial video is divided into multiple segments of equal length using a sparse sampling strategy, and each segment is randomly sampled to obtain aerial video features.

[0009] In one possible implementation, the expected loss of the keyframe combination is: , in, This is a collection of videos from aerial video data. A set of frames in a video. The number of frames sampled from the video frame set. The loss for sampling frames in the policy network, The total loss for sampling frames in the policy network. The expected loss of aerial video data, For the expected loss of video frames, For video No. Frame loss.

[0010] In one possible implementation, the spatial feature extraction module includes a target feature extraction network, a scene feature extraction network, and a feature fusion module; the step of extracting features from the keyframe sequence based on the spatial feature extraction module to obtain the spatial appearance features of the keyframes includes: Based on the target feature extraction network, local feature extraction is performed on the foreground target region in the keyframe sequence to obtain target features; The scene features are obtained by performing global feature extraction on the keyframe sequence based on the scene feature extraction network. The feature fusion module fuses the target features and scene features to obtain the spatial appearance features of the keyframe.

[0011] In one possible implementation, the motion enhancement module includes an inter-frame difference submodule, an inter-frame correlation submodule, and a feature fusion submodule; the step of performing inter-frame difference and correlation calculations on the keyframe sequence based on the motion enhancement module to obtain motion features, and fusing the motion features with the spatial appearance features to obtain a fused feature sequence includes: Based on the inter-frame difference module, differential operation is performed on adjacent frames in the keyframe sequence to obtain differential features; Based on the inter-frame correlation submodule, local neighborhood multiplication is performed on adjacent frames in the keyframe sequence to obtain correlation features; Based on the feature fusion submodule, the differential features and correlation features are spliced ​​together and fused with channel attention to obtain motion features. The motion features are then fused with the spatial appearance features to obtain a fused feature sequence.

[0012] In one possible implementation, the multi-scale temporal relationship modeling module includes a time segmentation module, a local state modeling submodule, and a global temporal modeling submodule, wherein the network structure of the local state modeling submodule is a Mamba state-space network; the step of performing global modeling on the fused feature sequence based on the multi-scale temporal relationship modeling module to obtain global temporal features of the event includes: Based on the time-segmentation module, the fused feature sequence is segmented and sampled in the time dimension to obtain feature sequences at multiple scales; Based on the Mamba state space network, the feature sequences at multiple scales are modeled to obtain local temporal features; The global time modeling submodule performs convolution and activation processing on the fused feature sequence, and fuses the local temporal features and the activated feature sequence to obtain the global temporal features of the event.

[0013] Secondly, the present invention also provides an event recognition device for aerial video, used to perform event recognition on drone aerial video based on a constructed event recognition model, wherein the event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module; The event recognition device for the aerial video includes: The keyframe extraction unit is used to perform global modeling on the acquired UAV aerial video data based on the dynamic frame selection module to obtain a sequence of keyframes related to the event. A spatial appearance feature extraction unit is used to extract features from the key frame sequence based on the spatial feature extraction module to obtain the spatial appearance features of the key frames. The feature fusion unit is used to perform difference and correlation calculations between adjacent frames of the keyframe sequence based on the motion enhancement module to obtain motion features, and to fuse the motion features with the spatial appearance features to obtain a fused feature sequence. The event classification unit is used to perform global modeling on the fused feature sequence based on the multi-scale temporal relationship modeling module to obtain global temporal features of the event, and to classify the global temporal features of the event through a classifier to obtain the event category result.

[0014] Thirdly, the present invention also provides an event recognition device, comprising: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the event recognition method for aerial video as described above.

[0015] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the event recognition method for aerial video described in any one of the above-mentioned method items.

[0016] The beneficial effects of this invention are as follows: The event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module; based on the dynamic frame selection module, global modeling is performed on the acquired UAV aerial video data to obtain a sequence of key frames related to the event; the dynamic frame selection module is used to perform key screening of the aerial video, reducing interference from redundant frames and lowering the overall computational load, thus improving the model's focusing ability; based on the spatial feature extraction module, feature extraction is performed on the key frame sequence to obtain the spatial appearance features of the key frames; based on the motion enhancement module, the difference and correlation calculations between adjacent frames of the key frame sequence are performed to obtain motion features; the motion features are fused with the spatial appearance features to obtain a fused feature sequence; combining the motion enhancement module and the spatial feature extraction module effectively integrates... By combining spatial appearance and local motion features, the model enhances its ability to express time-related events and significantly improves its ability to perceive dynamic changes in targets. Based on a multi-scale temporal relationship modeling module, the fused feature sequence is globally modeled to obtain global temporal features of the event. These global temporal features are then classified using a classifier to obtain event category results. Furthermore, the multi-scale temporal relationship modeling module performs multi-scale temporal sampling processing on the fused features to capture the short-term and long-term temporal dependencies of events, improving adaptability to different behavior durations. Through dynamic frame filtering, motion enhancement, and multi-scale temporal modeling strategies, the model effectively solves the problems of keyframe sparsity, motion blur, and insufficient temporal modeling capabilities faced by traditional video recognition models when processing time-related events, thereby improving the accuracy and inference efficiency of event recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an embodiment of the event recognition method for aerial video provided by the present invention; Figure 2 A schematic diagram of the dynamic frame sampling module of the aerial video event recognition method provided by the present invention; Figure 3 This is a schematic diagram of the strategy network structure of the event recognition method for aerial video provided by the present invention; Figure 4 A schematic diagram of the motion enhancement module of the aerial video event recognition method provided by the present invention; Figure 5A schematic diagram of the structure of the multi-scale temporal relationship modeling module for the event recognition method of aerial video provided by the present invention; Figure 6 A schematic diagram of the event recognition model for the event recognition method of aerial video provided by the present invention; Figure 7 A schematic diagram of the structure of an embodiment of the event recognition device for aerial video provided by the present invention; Figure 8 This is a schematic diagram of an embodiment of the event recognition device provided by the present invention. Detailed Implementation

[0019] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0020] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] This invention discloses a method, apparatus, device, and medium for event recognition in aerial video, which can be used in a computer. The method, apparatus, or computer-readable storage medium involved in this invention can be integrated with the aforementioned apparatus or can be relatively independent.

[0022] One specific embodiment of the present invention discloses an event recognition method for aerial video, which can be executed by a computer, specifically by one or more processors of the computer. For example... Figure 1 As shown, the event recognition method for aerial video is based on a constructed event recognition model to perform event recognition on drone aerial video. The event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module. The event recognition method for aerial video includes: S101. Based on the dynamic frame selection module, perform global modeling on the acquired UAV aerial video data to obtain the key frame sequence related to the event; It should be noted that using the dynamic frame selection module to perform key filtering on aerial video significantly reduces redundant calculations and improves the model's focusing ability.

[0023] S102. Based on the spatial feature extraction module, feature extraction is performed on the key frame sequence to obtain the spatial appearance features of the key frames. S103. Based on the motion enhancement module, perform differential and correlation calculations between adjacent frames of the key frame sequence to obtain motion features. Then, fuse the motion features with the spatial appearance features to obtain the fused feature sequence.

[0024] It should be noted that by combining the motion enhancement module and the spatial feature extraction module, the spatial appearance and local motion features are effectively integrated, thereby improving the ability to express time-related events.

[0025] S104. The multi-scale temporal relationship modeling module performs global modeling on the fused feature sequence to obtain global temporal features of the event. The global temporal features of the event are then classified by a classifier to obtain the event category results. It should be noted that by capturing long-term and short-term time dependencies through a multi-scale time modeling mechanism, the adaptability to the duration of different behaviors is improved. Through dynamic frame filtering mechanism, motion enhancement mechanism and multi-scale time modeling strategy, the accuracy of event recognition and inference efficiency are improved.

[0026] In some embodiments, in step S101, the acquired UAV aerial video data is globally modeled based on the dynamic frame selection module to obtain key frame sequences related to events. The acquired UAV aerial video data includes video data containing time-related behaviors collected from the UAV aerial platform, including various dynamic events captured in scenarios such as traffic management, environmental monitoring, disaster relief, and goods delivery, but not limited to the UCF-ARG dataset and a self-built football match dataset. It features sparse event occurrence, large scene changes, and significant camera movement. The UAV aerial video data is preprocessed to obtain aerial video features. A sparse sampling strategy is used to divide the UAV aerial video into multiple segments of equal length, and each segment is randomly sampled to obtain aerial video features. A sparse sampling strategy similar to TSN is used to divide the video into n segments of equal length, and one frame is randomly sampled from each segment to obtain n RGB type video frames. n is set to 8, and the original image resolution of the sampled video frames is... After performing data augmentation processing on the original image, including random resizing, multi-scale cropping, and random horizontal flipping, the final image has a resolution of [resolution value missing]. The image frames of pixels are used as input to the model to obtain the drone aerial video features, which form a candidate set of frames. The dynamic frame selection module includes a feature extraction submodule and a policy network submodule. The feature extraction submodule uses an X3D network to perform coarse-grained global feature extraction on the aerial video features, obtaining global frame-level features. The policy network submodule then performs spatiotemporal feature extraction on the global frame-level features to obtain a sequence of key frames related to the event. The goal is to minimize the expected loss of the key frame combination in the key frame sequence. Multiple Monte Carlo sampling is used to estimate the quality of the key frame combination. The policy network submodule combines global average pooling, temporal convolution, channel normalization, and activation functions to generate an importance weight sequence of frames, guiding the non-replacement weighted sampling process of key frames. Finally, the frame-level combination is output for subsequent feature extraction. The dynamic frame selection module addresses the issue of redundant frames in aerial videos. It models the importance of candidate frames using a policy network. First, it uniformly samples T frames along the time dimension to form a candidate frame set. A lightweight X3D network extracts coarse global contextual features for each frame, which are then input into the policy network for further encoding spatiotemporal features and outputting attention weights for each frame, resulting in an attention sequence. This attention sequence is used for weighted sampling to select the combination of keyframes that best represent the event. The training objective of the policy network is to minimize the expected loss of keyframe combinations in the event recognition task, thereby improving the model's frame selection quality and computational efficiency. The dynamic frame selection module adaptively identifies and focuses on the frames with the most information in the video, improving event recognition efficiency without sacrificing accuracy. For a schematic diagram of the dynamic frame selection module, please refer to [link to schematic diagram]. Figure 2 ,like Figure 2 As shown, for the input aerial video, firstly, T frames are sampled evenly along the time dimension as candidate frames. Downsample it to a lower resolution A new set of frames is obtained. Then use a lightweight X3D network ( At a lower resolution, quickly extract coarse global frame-level features. Its global frame-level features are: , in, This refers to global frame-level features, i.e., global video representation. , For frame number, For the first Frame context encoding information, The first video after downsampling frame; Then, the global frame-level features are input into the policy network of the policy network submodule. In this process, spatiotemporal features are further extracted and the optimal set of motion frames is determined. The optimal set of motion frames is the sequence of keyframes related to the event. For a schematic diagram of the policy network structure, please refer to [link to diagram]. Figure 3 The policy network generates frame weights based on temporal convolution, channel normalization, and activation functions. It then uses a no-replacement weighted sampling strategy to generate frame combinations and trains based on minimizing the expected combination loss using a set-based collaborative approach. The task is to select the frame most relevant to the event task, from Network Receive Global Context Features Next, spatiotemporal features are further encoded, and it is determined which frames can be used as salient segments of the video. The specific formula is as follows: , , , , in, for Global average pooling in the spatial dimension, , To perform temporal convolution operations in the time dimension, the kernel size is... , This is a normalization operation for the channel dimension. For activation function, The convolution kernel is The 2D convolution performs a fully connected operation along the channel dimension, resulting in an output channel of 1. To expand the features along the channel dimension, after... The weight sequence is obtained after the activation function. , Policy Network From the original Selecting from frames For each task-related frame, generate a weight set. , Weights in the original Weighted sampling is performed on the frame to obtain Selected frames ,Right now: , , in, For each frame The weight of the sample is such that the larger the weight value, the greater the probability of it being sampled. This is a weighted sampling algorithm; sampling is based on weights. Perform multiple samplings until the selected sample is found. A keyframe, if frame Already selected (i.e.) Belongs to the selected frame set If a frame is selected, its weight is set to 0 in the next sampling, indicating that the already selected frame will not be selected again; otherwise, the weight is set to 0. Take , i.e., frame The original weights; to make the sampled weights... Frames contain more task-related information compared to other frames in the event recognition task. To minimize the expected loss of keyframe combinations in the keyframe sequence, multiple Monte Carlo samplings are used to estimate the quality of the keyframe combinations. The expected loss of the keyframe combinations is: , in, This is a collection of videos from aerial video data. A set of frames in a video. The number of frames sampled from the video frame set. The loss for sampling frames in the policy network, The total loss for sampling frames in the policy network. The expected loss of aerial video data, The expected loss of a video frame is the value of the video frame. The smaller the expected loss of a video frame, the more task-related information is contained in the video. For video No. Frame loss; the goal of the policy network is to select an optimal combination of frames, rather than selecting several independent frames. Frame combinations are a comprehensive representation of these frames, and they can better reflect the content and features of the entire video. While individual frames may be significant on their own, their combination cannot well represent the overall features of the video. Monte Carlo sampling estimation is used. To eliminate Compared to Differentiability, in order to achieve end-to-end training, Cannot be used directly As a monitoring signal, it is necessary to Let's break it down further: , The process involves two steps: First, the total expectation is decomposed into the average of the expected loss for each frame, utilizing the linear property of expectation; second, the expected loss for each selected frame is further decomposed by introducing conditional probability. This indicates that before it has been selected. In the case of frames, the first The sampling probability of a frame, i.e., the selection of the first frame Frame number, the first The frame loss is a conditional distribution, reflecting the sequential selection process in dynamic sampling, where each step depends on the results of previous selections; the third step transforms the conditional probability into a specific weighting form. This represents the total number of frames in the video. For the first Frame in The weights used in step selection are set such that the weights of already selected frames are set to zero to avoid duplicate selections. The denominator is a normalization of the weights to ensure the validity of the probability distribution. Therefore, the expected value is a weighted average of the losses of all frames. The complex combinatorial optimization problem is transformed into a differentiable weighted summation form, allowing the model to optimize the frame sampling strategy using stochastic gradient descent. Then, Monte Carlo sampling is used to approximate the expectation. The expectation is approximated by calculating the number of Monte Carlo samples, i.e.: , in, N frames are sampled from the total number of frames T. For the first M samples were taken in total. To perform weighted sampling of the probability distribution based on given weights, Let J be the weight of the j-th frame at the m-th sampling. The final representation is: , The selected M samples were calculated using a weighted average method. Frame loss.

[0027] The dynamic frame selection module adaptively selects frames containing key motion information based on the rate of change of motion features between frames, reducing redundancy and improving recognition efficiency.

[0028] In some embodiments, in step S102, feature extraction is performed on the keyframe sequence based on the spatial feature extraction module to obtain the spatial appearance features of the keyframe. The spatial feature extraction module includes a target feature extraction network, a scene feature extraction network, and a feature fusion module. The target feature extraction network performs local feature extraction on the foreground target region in the keyframe sequence to obtain target features; the scene feature extraction network performs global feature extraction on the keyframe sequence to obtain scene features; and the feature fusion module fuses the target features and scene features to obtain the spatial appearance features of the keyframe.

[0029] In some embodiments, in step S103, the motion enhancement module performs inter-frame difference and correlation calculations on the keyframe sequence to obtain motion features. These motion features are then fused with the spatial appearance features to obtain a fused feature sequence. The motion enhancement module includes an inter-frame difference submodule, an inter-frame correlation submodule, and a feature fusion submodule. The inter-frame difference submodule performs difference operations on adjacent frames in the keyframe sequence to obtain difference features. This inter-frame difference submodule is used to perform difference operations on adjacent frames in the channel dimension to extract directional changes representing local target motion. The inter-frame correlation submodule performs local neighborhood multiplication operations on adjacent frames in the keyframe sequence. The inter-frame correlation submodule obtains correlation features by constructing regional dynamic relationships between adjacent frames through local neighborhood multiplication or spatial convolution operations, which is used to model target matching and motion trajectories in space. Based on the feature fusion submodule, differential features and correlation features are concatenated and fused with channel attention to obtain motion features. These motion features are then fused with the spatial appearance features to obtain a fused feature sequence. The feature fusion submodule is used to concatenate differential features and correlation features with channel attention and output a structured enhanced temporal event feature representation, i.e., motion features. Its inter-frame differential submodule is the CMM (Channel-wise Multiplication Model) in the STM network. The Motion Module (CMM) simulates motion features similar to optical flow through temporal difference operations. Compared to optical flow, the CMM internalizes motion feature modeling, enhancing the model's ability to utilize motion information while significantly reducing computational load and improving model efficiency. The CMM uses content displacement between two consecutive frames to reflect changes in motion. Since motion causes spatial displacement of the same object between adjacent frames, leading to motion representation mismatches and making direct difference calculation impossible, a convolution kernel is used for the subsequent frame. The 2D convolution is used to simulate motion offset; For a schematic diagram of the motion enhancement module that combines inter-frame difference and correlation, please refer to [link / reference]. Figure 4 ,like Figure 4 As shown, the motion enhancement module takes frame features as input, uses pairs of adjacent appearance features to generate frame-by-frame motion features, and then adds the obtained motion features to the appearance features for final prediction. The motion enhancement module extracts motion features by combining subtraction and multiplication operations, i.e., calculating the inter-frame difference and correlation between adjacent frames. Subtraction is widely used to estimate optical flow, which represents the magnitude of changes in spatial features. Using subtraction to infer the smoothness of changes helps in understanding motion features in spatial location. Features of adjacent frames and The subtraction calculation is performed using the following formula: , Then we get a size of The tensor; spatial multiplication between two features can be a spatial feature appearance similarity correlation, and similarity can represent the probability of corresponding spatial appearances in a frame, thus it can be used to understand the dynamics of actions. To improve computational efficiency, it utilizes... Local multiplication in the neighborhood replaces global multiplication to calculate features of adjacent frames. and In spatial location The multiplication operation is as follows: , in, For the set of relative positions in the neighborhood space of adjacent frames (P is set to 9 by default), , ≡ × ,in, Given the maximum offset within the neighborhood, we obtain a value of size [value]. The multiplication tensor is then converted into the original features through a convolutional layer. To maintain consistent feature shapes, the size of the feature representation is determined by using the original frame feature representation for the last frame of subtraction and multiplication operations. Specifically, given T frames, and For the frame features before and after the operation, use filling ,Right now After performing subtraction and multiplication operations, the resulting multiplication tensor and tensor are concatenated using concat and channel attention fusion to obtain motion features.

[0030] Video is essentially composed of a sequence of image frames. The motion information of events in a video is implicit in the temporal information between frames. Therefore, to accurately identify the event category in aerial video, it is necessary not only to consider the visual features of the current scene, but also to pay attention to the motion trend in the time dimension. Visual features provide static information such as color, shape, and texture, which are important bases for identifying event categories. However, in dynamic scenes, event identification not only depends on instantaneous visual information, but also on the time-related dynamic event dataset. It is difficult for the model to simply determine the event category through spatial features. It is also necessary to analyze the motion relationship and change pattern between objects. Therefore, the motion trend in the time dimension is particularly important. Thus, by inputting keyframe sequences into the motion enhancement module, the model uses the difference operation between adjacent frames and the local spatial correlation calculation method to obtain motion features and fuse them with appearance features, which significantly improves the model's ability to perceive dynamic changes of targets.

[0031] In some embodiments, in step S104, the fused feature sequence is globally modeled based on the multi-scale temporal relationship modeling module to obtain global temporal features of the event. The multi-scale temporal relationship modeling module includes a time segmentation module, a local state modeling submodule, and a global temporal modeling submodule. The network structure of the local state modeling submodule is a Mamba state space network. Based on the time segmentation module, the fused feature sequence is segmented and sampled in the time dimension to obtain feature sequences at multiple scales. Based on the Mamba state space network, the feature sequences at multiple scales are modeled to obtain local temporal features. Based on the global temporal modeling submodule, the fused feature sequence is convolutionally and activated to fuse the local temporal features and the activated feature sequence to obtain global temporal features of the event. For a schematic diagram of the multi-scale temporal relationship modeling module, please refer to [link to schematic diagram]. Figure 5 ,like Figure 5 As shown, the fused feature sequence is divided into 8 segments, 4 segments, and 2 segments by the time segmentation module to capture the local temporal relationships under different behavior durations. Then, the Mamba state space structure combined with the multi-scale time segmentation strategy is used to model the time granularity of 2 segments, 4 segments, and 8 segments in parallel. The Mamba is a selective state space model (SSM) that independently models long-term inter-frame dependencies at each time scale. Finally, the global time modeling submodule is used to fuse the event global temporal features to obtain a unified multi-scale event global temporal feature. The global temporal features of the event are classified using a classifier to obtain the event category results. The motion enhancement module extracts higher-level features from the motion frame set to predict event categories, and this is done from the policy network. Receive a length of The set of frames, whose length is The set of frames is a sequence of keyframes related to the event. Then, after global modeling by the multi-scale temporal relationship modeling module, the global temporal features of the event are input into the classifier. The classifier outputs the recognition results of the event in the aerial video. Formally, the classifier... Direct processing The original resolution is frames ,Right now , in, The final probability score for each category.

[0032] The event recognition model is optimized end-to-end using a joint loss function, which is as follows: , in, The global feature encoding loss is the loss of the X3D network. The total loss of the policy network, The loss of the classification network; for a schematic diagram of the event recognition model, please refer to [link / reference]. Figure 6 ,like Figure 6 As shown, the dynamic frame selection module filters the most discriminative key frames in the video, significantly reducing redundant computation; the motion enhancement module integrates inter-frame difference and correlation information with the spatial feature extraction module, enhancing the model's ability to model dynamic changes in events; the multi-scale temporal relationship modeling module captures the short-term and long-term dependencies of events using a state-space model; and finally, a classifier completes the event recognition, taking into account both spatial structure information and temporal change features. This approach effectively addresses issues such as event sparsity, complex perspectives, and target motion blur, achieving excellent recognition results on multiple aerial video datasets, demonstrating high practical value and promising prospects for wider application.

[0033] In summary, the event recognition method for aerial video provided by this invention is used to identify events in UAV aerial video based on a constructed event recognition model. The event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module. The dynamic frame selection module performs global modeling on the acquired UAV aerial video data to obtain a sequence of key frames related to the event. The spatial feature extraction module extracts features from the key frame sequence to obtain the spatial appearance features of the key frames. The motion enhancement module performs difference and correlation calculations between adjacent frames in the key frame sequence to obtain motion features. These motion features are then fused with the spatial appearance features to obtain a fused feature sequence. The multi-scale temporal relationship modeling module performs global modeling on the fused feature sequence to obtain global temporal features of the event. A classifier is then used to classify these global temporal features to obtain the event category results, thus improving the accuracy and inference efficiency of event recognition.

[0034] To better implement the event recognition method for aerial video in this embodiment of the invention, based on the event recognition method for aerial video, correspondingly, as follows: Figure 7 As shown, this embodiment of the invention also provides an event recognition device for aerial video. The event recognition device 700 for aerial video is used to perform event recognition on drone aerial video based on a constructed event recognition model. The event recognition model includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module. The aerial video event recognition device 700 includes: The keyframe extraction unit 701 is used to perform global modeling on the acquired UAV aerial video data based on the dynamic frame selection module to obtain a keyframe sequence related to the event. The spatial appearance feature extraction unit 702 is used to extract features from the key frame sequence based on the spatial feature extraction module to obtain the spatial appearance features of the key frames. The feature fusion unit 703 is used to perform difference and correlation calculations between adjacent frames on the key frame sequence based on the motion enhancement module to obtain motion features, and to fuse the motion features with the spatial appearance features to obtain a fused feature sequence. The event classification unit 704 is used to perform global modeling on the fused feature sequence based on the multi-scale time relationship modeling module to obtain global temporal features of the event, and to classify the global temporal features of the event through a classifier to obtain the event category result.

[0035] like Figure 8 As shown, the present invention also provides an event recognition device 800, which can be a computing device such as a mobile terminal, desktop computer, laptop, handheld computer, and server. The event recognition device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the event recognition device 800 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0036] In some embodiments, memory 802 may be an internal storage unit of the event recognition device 800, such as a hard disk or memory of the event recognition device 800. In other embodiments, memory 802 may be an external storage device of the event recognition device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the event recognition device 800. Further, memory 802 may include both internal and external storage units of the event recognition device 800. Memory 802 is used to store application software and various types of data installed on the event recognition device 800, such as program code for installing the event recognition device 800. Memory 802 may also be used to temporarily store data that has been output or will be output. In one embodiment, memory 802 stores an event recognition program for aerial video, which can be executed by processor 801 to implement the event recognition method for aerial video according to various embodiments of the present invention.

[0037] In some embodiments, processor 801 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as an event recognition method for aerial video.

[0038] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display identification information from the event recognition program of the aerial video and to display a visual user interface. Components 801-803 of the event recognition device 800 communicate with each other via a system bus.

[0039] In some embodiments, when the processor 801 executes the event recognition program for aerial video in the memory 802, it implements each step of the event recognition method for aerial video as described in the above embodiments. Since the event recognition method for aerial video has been described in detail above, it will not be repeated here.

[0040] Accordingly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps or functions of the aerial video event recognition method provided in the above-described method embodiments.

[0041] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0042] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for event recognition in aerial video, characterized in that, This is used to perform event recognition on drone aerial videos based on a constructed event recognition model, which includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module. The event recognition method for the aerial video includes: The dynamic frame selection module is used to perform global modeling on the acquired UAV aerial video data to obtain a sequence of key frames related to the event. Based on the spatial feature extraction module, feature extraction is performed on the key frame sequence to obtain the spatial appearance features of the key frames. Based on the motion enhancement module, the keyframe sequence is subjected to difference and correlation calculation between adjacent frames to obtain motion features. The motion features are then fused with the spatial appearance features to obtain a fused feature sequence. The multi-scale temporal relationship modeling module performs global modeling on the fused feature sequence to obtain global temporal features of the event. The global temporal features of the event are then classified by a classifier to obtain the event category results.

2. The event recognition method for aerial video according to claim 1, characterized in that, The dynamic frame selection module includes a feature extraction submodule and a policy network submodule. The network structure of the feature extraction submodule is an X3D network. The step of globally modeling the acquired UAV aerial video data based on the dynamic frame selection module to obtain a sequence of keyframes related to the event includes: The drone aerial video data is preprocessed to obtain aerial video features; The X3D network is used to perform coarse-grained global feature extraction on the aerial video features to obtain global frame-level features; The policy network submodule is used to extract spatiotemporal features from the global frame-level features to obtain a key frame sequence related to the event. The quality of the key frame combination is estimated by multiple Monte Carlo sampling with the goal of minimizing the expected loss of the key frame combination in the key frame sequence.

3. The event recognition method for aerial video according to claim 2, characterized in that, The preprocessing of the drone aerial video data to obtain aerial video features includes: The drone aerial video is divided into multiple segments of equal length using a sparse sampling strategy, and each segment is randomly sampled to obtain aerial video features.

4. The event recognition method for aerial video according to claim 2, characterized in that, The expected loss of the keyframe combination is: ， in, This is a collection of videos from aerial video data. A set of frames in a video. The number of frames sampled from the video frame set. The loss for sampling frames in the policy network, The total loss for sampling frames in the policy network. The expected loss of aerial video data, For the expected loss of video frames, For video No. Frame loss.

5. The event recognition method for aerial video according to claim 3, characterized in that, The spatial feature extraction module includes a target feature extraction network, a scene feature extraction network, and a feature fusion module; the step of extracting features from the keyframe sequence based on the spatial feature extraction module to obtain the spatial appearance features of the keyframes includes: Based on the target feature extraction network, local feature extraction is performed on the foreground target region in the keyframe sequence to obtain target features; The scene features are obtained by performing global feature extraction on the keyframe sequence based on the scene feature extraction network. The feature fusion module fuses the target features and scene features to obtain the spatial appearance features of the keyframe.

6. The event recognition method for aerial video according to claim 5, characterized in that, The motion enhancement module includes an inter-frame difference submodule, an inter-frame correlation submodule, and a feature fusion submodule. The process involves calculating the difference and correlation between adjacent frames of the keyframe sequence based on the motion enhancement module to obtain motion features, and then fusing these motion features with the spatial appearance features to obtain a fused feature sequence, including: Based on the inter-frame difference module, differential operation is performed on adjacent frames in the keyframe sequence to obtain differential features; Based on the inter-frame correlation submodule, local neighborhood multiplication is performed on adjacent frames in the keyframe sequence to obtain correlation features; Based on the feature fusion submodule, the differential features and correlation features are spliced ​​together and fused with channel attention to obtain motion features. The motion features are then fused with the spatial appearance features to obtain a fused feature sequence.

7. The event recognition method for aerial video according to claim 6, characterized in that, The multi-scale temporal relationship modeling module includes a time segmentation module, a local state modeling submodule, and a global temporal modeling submodule. The network structure of the local state modeling submodule is a Mamba state-space network. The step of performing global modeling on the fused feature sequence based on the multi-scale temporal relationship modeling module to obtain global temporal features of the event includes: Based on the time-segmentation module, the fused feature sequence is segmented and sampled in the time dimension to obtain feature sequences at multiple scales; Based on the Mamba state space network, the feature sequences at multiple scales are modeled to obtain local temporal features; The global time modeling submodule performs convolution and activation processing on the fused feature sequence, and fuses the local temporal features and the activated feature sequence to obtain the global temporal features of the event.

8. An event recognition device for aerial video, characterized in that, This is used to perform event recognition on drone aerial videos based on a constructed event recognition model, which includes a dynamic frame selection module, a spatial feature extraction module, a motion enhancement module, and a multi-scale temporal relationship modeling module. The event recognition device for the aerial video includes: The keyframe extraction unit is used to perform global modeling on the acquired UAV aerial video data based on the dynamic frame selection module to obtain a sequence of keyframes related to the event. A spatial appearance feature extraction unit is used to extract features from the key frame sequence based on the spatial feature extraction module to obtain the spatial appearance features of the key frames. The feature fusion unit is used to perform difference and correlation calculations between adjacent frames of the keyframe sequence based on the motion enhancement module to obtain motion features, and to fuse the motion features with the spatial appearance features to obtain a fused feature sequence. The event classification unit is used to perform global modeling on the fused feature sequence based on the multi-scale temporal relationship modeling module to obtain global temporal features of the event, and to classify the global temporal features of the event through a classifier to obtain the event category result.

9. An event recognition device, characterized in that, Including memory and processor; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the event recognition method for aerial video as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the event recognition method for aerial video as described in any one of claims 1-7.

Images