Unmanned aerial vehicle video identification method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of unmanned aerial vehicles, in particular to an unmanned aerial vehicle video recognition method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: carrying out video frame sampling on a target aerial video to obtain a video frame set; obtaining spatial-temporal features, global scene features and local target features corresponding to the target aerial video according to the video frame set; fusing the global scene features and the local target features to obtain fused features; and obtaining a video recognition result according to the spatial-temporal features and the fusion features. According to the unmanned aerial vehicle video recognition method and device, the electronic equipment and the computer readable storage medium provided by the invention, the technical effect of improving the accuracy of an unmanned aerial vehicle video recognition result can be realized.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicles (UAVs), specifically to a UAV video recognition method and apparatus, electronic equipment, and computer-readable storage medium. Background Technology

[0002] With the popularization of drone technology and the development of deep learning technology, video-based intelligent recognition technology has made significant progress. Research on intelligent recognition methods for low-altitude environments from drone aerial videos is of great value for improving the perception and cognitive capabilities of drones.

[0003] Compared to traditional ground-view videos, drone aerial videos face unique challenges: on the one hand, the wide field of view of drones introduces a large amount of complex background information, increasing the difficulty of extracting key scene and target features; on the other hand, the global scene information in aerial videos can depict the overall environment in which the event occurs, while local target features carry detailed information about the actors.

[0004] However, existing recognition methods fail to fully consider these visual characteristics in low-altitude aerial videos, resulting in low recognition accuracy of drone aerial videos in complex scenes. Summary of the Invention

[0005] In view of this, it is necessary to provide a drone video recognition method and apparatus, electronic device and computer-readable storage medium to achieve the technical effect of improving the accuracy of drone video recognition results.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for drone video recognition, comprising:

[0007] Video frames are sampled from the aerial video of the target to obtain a set of video frames; Based on the video frame set, obtain the spatiotemporal features, global scene features, and local target features corresponding to the target aerial video; The fused features are obtained by fusing the global scene features and the local target features; The video recognition result is obtained based on the spatiotemporal features and the fusion features.

[0008] In one possible embodiment, obtaining the spatiotemporal features corresponding to the target aerial video based on the video frame set includes: Perform 3D convolution operation on the video frame set to obtain two-dimensional spatial features; The two-dimensional spatial features are unfolded into one-dimensional features; Position embedding is performed on the one-dimensional feature to obtain spatial position information; The spatial location information is subjected to layer normalization to obtain the spatiotemporal features.

[0009] In one possible embodiment, obtaining global scene features corresponding to the target aerial video based on the video frame set includes: The video frame set is subjected to layer normalization processing to obtain the set layer normalization result; The Mamba branch output is obtained by efficiently performing sequence modeling on the normalized result of the set layer through visual Mamba branching. By using multi-head self-attention branches, the global dependency relationship of the normalization result of the set layer is modeled to obtain the multi-head self-attention branch output; The Mamba branch output and the multi-head self-attention branch output are concatenated to obtain the branch concatenation result; The branch splicing results are subjected to layer normalization and random path discarding to obtain the global scene features.

[0010] In one possible embodiment, obtaining local target features corresponding to the target aerial video based on the set of video frames includes: Perform a one-dimensional temporal convolution on the set of video frames to obtain the temporal features of the set; The temporal features of the set are reconstructed in two-dimensional space to obtain the reconstructed features; The reconstructed features are then subjected to convolution and pooling to obtain the pooling result. The pooling result is subjected to convolution and upsampling to obtain the upsampling result; The upsampling results are then concatenated along the channel dimension to obtain the channel concatenation result; The local target features are obtained by unfolding the channel splicing results in one-dimensional space.

[0011] In one possible embodiment, fusing the global scene features and the local target features to obtain the fused features includes: Align the global scene features and the local target features by channel dimension; The global scene features and the local target features, after being aligned by channel dimension, are concatenated by channel dimension to obtain the feature concatenation result; The feature concatenation result is subjected to global average pooling to obtain the global average pooling result. Linear activation is applied to the global average pooling result to obtain the fusion weights; The fused feature is obtained based on the fusion weight and the feature concatenation result.

[0012] In one possible embodiment, obtaining the video recognition result based on the spatiotemporal features and the fused features includes: The spatiotemporal features and the fused features are subjected to residual connections based on a fully connected layer to obtain residual connection results; The environmental category that satisfies the preset conditions for the model's total loss function is obtained from the residual stitching result as the video recognition result.

[0013] In one possible embodiment, it further includes: The total loss function of the model is constructed based on the cross-entropy loss function.

[0014] Secondly, this application also provides a drone video recognition device, comprising: A frame acquisition module is used to sample video frames from the aerial video of the target to obtain a set of video frames. The feature extraction module is used to obtain the spatiotemporal features, global scene features and local target features corresponding to the target aerial video based on the video frame set; A feature fusion module is used to fuse the global scene features and the local target features to obtain fused features; A video recognition module is used to obtain video recognition results based on the spatiotemporal features and the fusion features.

[0015] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the UAV video recognition method described in any of the above implementations.

[0016] Fourthly, this application also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the UAV video recognition method described in any of the above implementations.

[0017] The beneficial effects of this application are: Compared with related technologies, the UAV video recognition method, apparatus, electronic device, and computer-readable storage medium provided in this application sample video frames of the target aerial video captured by the UAV and requiring recognition, obtaining a video frame set composed of multiple video frames. The spatiotemporal features, global scene features, and local target features corresponding to the target aerial video are obtained through the video frame set. Global scene features are extracted for global analysis of the target aerial video, and local target features are extracted for enhanced analysis of local targets in the target aerial video. Then, the global scene features and local target features are fused to obtain fused features, enabling combined analysis of local targets and global background in the target aerial video. Further combining spatiotemporal features and fused features yields the video recognition result. By fusing and processing these three features, the accuracy of UAV video recognition results can be improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the drone video recognition method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the process for obtaining spatiotemporal features in the UAV video recognition method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the process of obtaining global scene features in the UAV video recognition method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the process for obtaining local target features in the UAV video recognition method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the process of fusing global scene features and local target features in the UAV video recognition method provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of the drone video recognition device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0022] The terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] This application provides a method and apparatus for drone video recognition, an electronic device, and a computer-readable storage medium, which are described below.

[0025] Please refer to Figure 1 The drone video recognition method provided in this application includes: Step S101: Sample video frames from the target aerial video to obtain a set of video frames.

[0026] In this step, video frame sampling of the target aerial video specifically involves extracting video frames from the target aerial video, and all extracted video frames form a video frame set. In this embodiment, a uniform sampling method is used to sample the target aerial video. Uniform sampling involves extracting sample frames from all video frames of the target aerial video at fixed frame intervals, ensuring that the sampled frames uniformly cover the entire time range of the target aerial video. Uniform sampling results in a uniform distribution of sampled frames, which can completely cover the entire time period of the video and avoid excessive information loss in a certain period; the sampling results are predictable, facilitating subsequent unified feature analysis and recognition processing.

[0027] It is understood that the aforementioned method of uniform sampling for sampling video frames of the target aerial video is merely an example of the specific method for sampling video frames of the target aerial video in this embodiment. In some other embodiments of this application, a keyframe sampling method can also be used to sample video frames of the target aerial video. Keyframe sampling involves quantifying the visual similarity of adjacent frames, filtering out key frames that represent significant changes in the scene, eliminating redundant frames with highly similar content, and focusing on frames in the video that have incremental information. Keyframe sampling accurately preserves the core information of scene changes, significantly reduces redundant frames, and reduces the computational load of subsequent recognition processing. Alternatively, in some other embodiments of this application, a temporal interval sampling method can also be used to sample video frames of the target aerial video. Temporal interval sampling uses the time axis of the video as the sampling basis, extracting corresponding frames at fixed time lengths to ensure that the sampled frames are evenly distributed in the time dimension. The sampling results of temporal interval sampling directly correspond to the time dimension, facilitating subsequent analysis of scene / target change trends along the time axis; it can accurately match the time observation requirements of the business.

[0028] Furthermore, in this embodiment, after sampling video frames from the target aerial video to obtain sampled frames, data enhancement is performed on the sampled frames to obtain a set of video frames composed of data-enhanced RGB format video frames. Data enhancement of the sampled frames can be achieved through geometric transformations, including random translation, rotation, scaling, horizontal / vertical flipping, random cropping, and affine transformation; at the pixel level, brightness, contrast, saturation, and hue can be adjusted, Gaussian noise or salt-and-pepper noise can be added, and Gaussian blurring and histogram equalization can be performed; color gamut transformations can also be performed, such as RGB to grayscale and HSV space parameter adjustment; in addition, mixed enhancement methods such as random erasing, mosaic, and image stitching can be used, and some enhancements adapted to aerial photography characteristics also include random viewpoint transformation, local area stretching or compression, etc.

[0029] Step S102: Obtain the spatiotemporal features corresponding to the target aerial video based on the video frame set.

[0030] Please refer to Figure 2 In this embodiment, obtaining spatiotemporal features specifically includes the following steps: Step S201: Perform 3D convolution operation on the video frame set to obtain two-dimensional spatial features.

[0031] In this step, the specific calculation formula for performing 3D convolution on the video frame set is as follows:

[0032] in, A collection of video frames. Features in two-dimensional space The function is a 3D convolution function with a kernel size of 3×P×P. In this embodiment, P=16. It is understood that the aforementioned P=16 is only a specific example in this embodiment. In some other embodiments of this application, P can also be other constant values ​​set according to actual needs.

[0033] in, The convolutional function is a core convolutional operation function designed for 3D structured data in deep learning, and it is also a key tool for capturing spatiotemporal features. Its core principle is to slide a 3D convolutional kernel (with dimensions of depth × height × width) position by position along the 3D dimensions of the input data, and perform multiplication and addition operations on the local 3D region covered at each position, thereby extracting the 3D spatial correlation features or spatiotemporal coupling features of the data. The function takes a 5-dimensional tensor as input, with two common input formats: one is a channel-first input format (including batch size, number of channels, depth, height, and width), and the other is a channel-last input format (including batch size, depth, height, width, and number of channels). The output is also a 5-dimensional feature tensor, the size of which is determined by the kernel size, stride, and padding method. The stride controls the sliding interval of the kernel in the three dimensions of depth, height, and width, while padding can prevent the feature map size from being excessively reduced after convolution, ensuring dimensionality adaptation.

[0034] Step S202: Expand the two-dimensional spatial features into one-dimensional features.

[0035] In this step, the specific use The function expands two-dimensional spatial features into one-dimensional features.

[0036] Functions are the fundamental core functions in numerical computation and deep learning frameworks for adjusting the dimensional structure of tensors / arrays. Their core essence is to redefine the dimensional organization of data without changing the values ​​and total number of data elements themselves, adjusting only the "view" of the data rather than the data itself. The core rule of the function is that the product of all dimensions in the target dimension must be exactly the same as the total number of elements in the original tensor / array. If this is not satisfied, a dimension incompatibility error will be thrown. To simplify the operation, the function supports using "-1" as a wildcard. Simply specify other dimensions, and the system will automatically calculate the dimension value that matches the total number of elements at that position (e.g., for a tensor with shape (2,3,4)). Given (6, -1), the second dimension will be automatically deduced to be 4). Data elements will be filled in a fixed storage order during the reshaping process. Most mainstream frameworks default to the C-style "row-first" order, that is, the last dimension is filled first, and then the dimensions are traversed forward in turn to adapt to different data storage habits.

[0037] Step S203: Perform position embedding on the one-dimensional features to obtain spatial location information.

[0038] In this step, the specific use The function represents the position embedding of one-dimensional features to obtain spatial location information, which, combined with the aforementioned step S202, can be expressed as a formula:

[0039] in, The weight matrix is ​​a core function in deep learning for mapping discrete features to continuous low-dimensional dense vectors. Its core function is to solve the high-dimensional sparsity problem of discrete features while capturing semantic relationships or potential patterns between features. Its core principle is to construct a trainable weight matrix. The number of rows in the matrix corresponds to the total number of discrete feature categories, the number of columns corresponds to the target embedding dimension, and the unique index of each discrete feature corresponds to a row in the matrix. The function maps the index to a low-dimensional dense vector in that row using a "lookup table" approach. The weight matrix is ​​continuously updated as the model trains, allowing the vectors to gradually learn the semantic similarity of the features. In this embodiment, the specific calculation formula for position embedding of one-dimensional features is as follows: , in, It is a learnable projection matrix. x N Input features X The N Features of each location E pos This is the location embedding vector that adopts a normal distribution.

[0040] Step S204: Perform layer normalization on the spatial location information to obtain spatiotemporal features.

[0041] In this step, the specific use The (layer normalization) function performs layer normalization processing on spatial location information. The function is a core normalization tool in deep learning used to stabilize model training and mitigate internal covariate bias. The core principle of the function is to independently calculate the mean and variance for all feature dimensions of each sample in the input tensor, and then normalize the feature distribution to a standard distribution with a mean of 0 and a variance of 1 using the standardization formula (eivalue minus mean, divided by standard deviation). In this embodiment, the calculation formula for layer normalization of spatial location information is as follows:

[0042] in, It is a spatiotemporal characteristic.

[0043] Step S103: Obtain the global scene features corresponding to the target aerial video based on the video frame set.

[0044] Please refer to Figure 3 In this embodiment, obtaining global scene features specifically includes the following steps: Step S301: Perform layer normalization processing on the video frame set to obtain the set layer normalization result.

[0045] In this step, the specific use The function performs layer normalization on the video frame set. It is understood that this step is largely the same as step S204 described above; for details, please refer to the specific descriptions in the aforementioned embodiments.

[0046] Step S302: Using the visual Mamba branch, perform efficient sequence modeling on the normalized results of the set layer to obtain the Mamba branch output.

[0047] In this step, the visual Mamba branch first obtains vectors from the normalized results of the ensemble layer through linear projection. x and z Then, from both the positive and negative directions, the vector... x Unfolding, in each direction, first perform a one-dimensional convolution on x, then use forward and backward SSM to obtain... y forward and y backward , y forward and y backward This refers to the output of the Mamba branch. The specific formula is as follows:

[0048] in, Output for Mamba branches, A collection of video frames. This is a function for handling branching in visual Mamba.

[0049] Step S303: Through multi-head self-attention branches, global dependency modeling is performed on the normalization results of the set layer to obtain the multi-head self-attention branch output.

[0050] In this step, the multi-head self-attention mechanism extracts local correlations and global dependencies from different feature subspaces simultaneously through multiple parallel "attention heads," and then fuses multi-dimensional features to improve the model's expressive power. The specific calculation formula is as follows: ; in, For multi-head self-attention branch output, The computation function corresponding to the multi-head self-attention mechanism specifically includes: ； ； ; in, Q , K , V These represent the query, key, and value, respectively. h To focus on the number of heads, Let be the projection matrix. ， ， The input features are projected as the query, key, and value in the attention mechanism, respectively. d For the input feature dimension, It refers to the number of heads to focus on. The function's purpose is to combine two or more data units end to end to form a new, longer result, while keeping the original data unchanged.

[0051] Step S304: Concatenate the Mamba branch output and the multi-head self-attention branch output to obtain the branch concatenation result.

[0052] In this step, the specific use The function concatenates the output of the Mamba branch and the output of the multi-head self-attention branch. function The specific formula is expressed as follows: .

[0053] Furthermore, the dynamic fusion weights for each location are calculated using a fusion gating network, as shown in the following formula: ; ; ; ; in, This represents the first-layer weight matrix. This represents the second-layer weight matrix, where b1 and b2 are bias vectors. This represents the operation of the sigmoid function; Based on dynamically calculated weights The Mamba branch output and the multi-head self-attention branch output are concatenated to obtain the branch concatenation result. The formula is as follows: ; in, This indicates element-wise multiplication.

[0054] Step S305: Perform layer normalization and random path discarding processing on the branch splicing results to obtain global scene features.

[0055] In this step, the formula is expressed as: ； ; in, For global scene features, It represents the DropPath probability in the random path dropping process. During training, paths are dropped according to probability, while all paths are retained during inference.

[0056] Step S104: Obtain the local target features corresponding to the target aerial video based on the set of video frames.

[0057] Please refer to Figure 4 In this embodiment, obtaining local target features specifically includes the following steps: Step S401: Perform a one-dimensional temporal convolution on the video frame set to obtain the temporal features of the set.

[0058] Step S402: Reconstruct the time series features of the set in two dimensions to obtain the reconstructed features.

[0059] Specifically, the formulas for steps S401 and S402 are as follows: .in, Conv1D For a one-dimensional temporal convolution of size 3, For set time series characteristics, Reshape Functions redefine the structure and layout of data while keeping the total number of elements (capacity) constant. This allows for adjustments to the input / output shape of a neural network to match the connection requirements between layers. For reconstructing features.

[0060] Step S403: Perform convolution processing on the reconstructed features and then perform pooling processing to obtain the pooling result.

[0061] Specifically, the formula for pooling after convolution processing of the reconstructed features in this step is expressed as follows: ; ; ; in, DWConvFor a 3×3 depthwise separable convolution, MaxPool It is a 3×3 max pooling operation with a step size of 2.

[0062] Step S404: Perform convolution and upsampling processing on the pooling results to obtain the upsampling results.

[0063] In this step, the formulas for convolution and upsampling of the pooling results are expressed as follows: ; ; ; in, Conv 1×1 A 1×1 2D convolution is used to reduce the number of channels. UpSample This is the upsampling function using the bilinear interpolation algorithm.

[0064] Step S405: Perform channel-dimensional concatenation on the upsampling results to obtain the channel concatenation result.

[0065] In this step, the formula for concatenating the upsampling results along the channel dimension is expressed as: ; ; .

[0066] in, X C This is the result of channel splicing.

[0067] Step S406: Expand the channel stitching result in one-dimensional space to obtain local target features.

[0068] In this step, the channel splicing result is expressed by a one-dimensional spatial expansion formula as follows: To obtain local target features .

[0069] Step S105: Fuse global scene features and local target features to obtain fused features.

[0070] Please refer to Figure 5 The process of fusing global scene features and local target features to obtain fused features in this step includes the following steps: Step S501: Align the global scene features and local target features according to the channel dimension.

[0071] In this step, 1×1 convolution and layer normalization operations are used to process the global scene features respectively. F G and local target features To achieve alignment between the two in the channel dimension, the formula is expressed as: ; .

[0072] Step S502: Perform channel-dimensional concatenation on the global scene features and local target features after channel-dimensional alignment to obtain the feature concatenation result.

[0073] In this step, the formula for concatenating the channel-dimensional aligned global scene features and local target features is expressed as follows: ,in This is the result of feature splicing.

[0074] Step S503: Perform global average pooling on the feature concatenation result to obtain the global average pooling result.

[0075] Step S504: Perform linear activation on the global average pooling result to obtain the fusion weights.

[0076] Step S505: Obtain the fused features based on the fusion weights and feature concatenation results.

[0077] In steps S503 to S505, channel statistics of the feature concatenation result are first generated using global average pooling, and then adaptive weights are learned through a linear layer. Simultaneously, channel dimensionality reduction is performed, and then based on this weight vector... Selectively fuse features. The formula is expressed as: ， ， ;in, GlobalPool For global average pooling, MLP is a linear layer with ReLU activation function. FC It is a fully connected layer. This is a feature of fusion. GlobalPool Global average pooling is a key dimensionality reduction and feature integration operation in convolutional neural networks. It compresses each feature map across the entire spatial dimension into a single representative value. It is mainly divided into global average pooling and global max pooling.

[0078] Step S106: Obtain the video recognition result based on the spatiotemporal features and fusion features.

[0079] In this step, residual connections are first performed on the spatiotemporal features and fused features based on the fully connected layer to obtain the residual connection result, which is expressed by the following formula: .in, The sliding window function is a downsampling operation in convolutional neural networks. Specifically, it takes the arithmetic mean of all values ​​within a local region of the input feature map and uses this average as the output for that region. It systematically reduces the spatial size of the feature map by using a sliding window approach.

[0080] Then, based on the residual stitching results, the environment category that satisfies the preset conditions for the model's total loss function is obtained as the video recognition result. The model's total loss function is the overall model loss function for executing the entire UAV video recognition method. Specifically, it is constructed based on the cross-entropy loss function, and the formula is expressed as: ,in, N It is the sample size. K It is the number of environmental categories. It is the first i Each sample in category c The real label on the sample i The true category equals c Then take 1, otherwise take 0. It is the first i Each sample in category c The predicted probability. In this step, that is, determining the residual connection result when the total loss function of the model satisfies the preset conditions. Corresponding category c As a result of video recognition.

[0081] Compared with related technologies, the UAV video recognition method provided in this embodiment samples video frames of the target aerial video captured by the UAV and to be identified, obtaining a video frame set composed of multiple video frames. The spatiotemporal features, global scene features, and local target features corresponding to the target aerial video are obtained through the video frame set. Global scene features are extracted for global analysis of the target aerial video, and local target features are extracted for enhanced analysis of local targets in the target aerial video. Then, the global scene features and local target features are fused to obtain fused features, which can combine and analyze the local targets and global background of the target aerial video. Further combining spatiotemporal features and fused features yields the video recognition result. By further combining spatiotemporal analysis of the target aerial video to obtain the video recognition result, the accuracy of UAV video recognition results can be improved through the fusion analysis and processing of these three features.

[0082] To better implement the UAV video recognition method in the embodiments of this application, based on the UAV video recognition method, correspondingly, such as Figure 6 As shown in the figure, this application embodiment also provides a drone video recognition device, which includes: The frame acquisition module 601 is used to sample video frames from the aerial video of the target to obtain a set of video frames. The feature extraction module 602 is used to obtain the spatiotemporal features, global scene features and local target features corresponding to the target aerial video based on the video frame set; The feature fusion module 603 is used to fuse global scene features and local target features to obtain fused features; The video recognition module 604 is used to obtain video recognition results based on spatiotemporal features and fusion features.

[0083] The drone video recognition device provided in the above embodiments can realize the technical solutions described in the above drone video recognition method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above drone video recognition method embodiments, and will not be repeated here.

[0084] Please refer to Figure 7 This application also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702, and a display 703. Figure 7 Only some components of the electronic device 700 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0085] In some embodiments, processor 701 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 702 or process data, such as the drone video recognition method in this application.

[0086] In some embodiments, processor 701 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 701 may be local or remote. In some embodiments, processor 701 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0087] In some embodiments, memory 702 may be an internal storage unit of electronic device 700, such as a hard disk or memory of electronic device 700. In other embodiments, memory 702 may also be an external storage device of electronic device 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 700.

[0088] Furthermore, the memory 702 may include both internal storage units of the electronic device 700 and external storage devices. The memory 702 is used to store application software and various types of data installed on the electronic device 700.

[0089] In some embodiments, display 703 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 703 is used to display information from electronic device 700 and to display a visual user interface. Components 701-703 of electronic device 700 communicate with each other via a system bus.

[0090] In one embodiment, when the processor 701 executes the drone video recognition program in the memory 702, the following steps can be performed: Video frames are sampled from the aerial video of the target to obtain a set of video frames; Based on the set of video frames, obtain the spatiotemporal features, global scene features, and local target features corresponding to the aerial video of the target; The fused features are obtained by fusing global scene features and local target features; The video recognition results are obtained based on spatiotemporal features and fusion features.

[0091] It should be understood that when the processor 701 executes the drone video recognition program in the memory 702, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0092] Furthermore, this application does not specifically limit the type of electronic device 700 mentioned in the embodiments. Electronic device 700 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, electronic device 700 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0093] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the UAV video recognition methods provided in the above-described method embodiments.

[0094] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0095] The above provides a detailed description of the drone video recognition method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for video recognition of unmanned aerial vehicles (UAVs), characterized in that, include: Video frames are sampled from the aerial video of the target to obtain a set of video frames; Based on the video frame set, obtain the spatiotemporal features, global scene features, and local target features corresponding to the target aerial video; The fused features are obtained by fusing the global scene features and the local target features; The video recognition result is obtained based on the spatiotemporal features and the fusion features.

2. The UAV video recognition method according to claim 1, characterized in that, The step of obtaining the spatiotemporal features corresponding to the target aerial video based on the video frame set includes: Perform 3D convolution operation on the video frame set to obtain two-dimensional spatial features; The two-dimensional spatial features are unfolded into one-dimensional features; Position embedding is performed on the one-dimensional feature to obtain spatial position information; The spatial location information is subjected to layer normalization to obtain the spatiotemporal features.

3. The UAV video recognition method according to claim 1, characterized in that, Based on the set of video frames, global scene features corresponding to the target aerial video are obtained, including: The video frame set is subjected to layer normalization processing to obtain the set layer normalization result; The Mamba branch output is obtained by efficiently performing sequence modeling on the normalized result of the set layer through visual Mamba branching. By using multi-head self-attention branches, the global dependency relationship of the normalization result of the set layer is modeled to obtain the multi-head self-attention branch output; The Mamba branch output and the multi-head self-attention branch output are concatenated to obtain the branch concatenation result; The branch splicing results are subjected to layer normalization and random path discarding to obtain the global scene features.

4. The UAV video recognition method according to claim 1, characterized in that, Based on the set of video frames, local target features corresponding to the aerial video of the target are obtained, including: Perform a one-dimensional temporal convolution on the set of video frames to obtain the temporal features of the set; The temporal features of the set are reconstructed in two-dimensional space to obtain the reconstructed features; The reconstructed features are then subjected to convolution and pooling to obtain the pooling result. The pooling result is subjected to convolution and upsampling to obtain the upsampling result; The upsampling results are then concatenated along the channel dimension to obtain the channel concatenation result; The local target features are obtained by unfolding the channel splicing results in one-dimensional space.

5. The UAV video recognition method according to claim 1, characterized in that, The fusion of the global scene features and the local target features to obtain the fused features includes: Align the global scene features and the local target features by channel dimension; The global scene features and the local target features, after being aligned by channel dimension, are concatenated by channel dimension to obtain the feature concatenation result; The feature concatenation result is subjected to global average pooling to obtain the global average pooling result. Linear activation is applied to the global average pooling result to obtain the fusion weights; The fused feature is obtained based on the fusion weight and the feature concatenation result.

6. The UAV video recognition method according to claim 1, characterized in that, The process of obtaining video recognition results based on the spatiotemporal features and the fusion features includes: The spatiotemporal features and the fused features are subjected to residual connections based on a fully connected layer to obtain residual connection results; The environmental category that satisfies the preset conditions for the model's total loss function is obtained from the residual stitching result as the video recognition result.

7. The UAV video recognition method according to claim 6, characterized in that, Also includes: The total loss function of the model is constructed based on the cross-entropy loss function.

8. A drone video recognition device, characterized in that, include: A frame acquisition module is used to sample video frames from the aerial video of the target to obtain a set of video frames. The feature extraction module is used to obtain the spatiotemporal features, global scene features and local target features corresponding to the target aerial video based on the video frame set; A feature fusion module is used to fuse the global scene features and the local target features to obtain fused features; A video recognition module is used to obtain video recognition results based on the spatiotemporal features and the fusion features.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the UAV video recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the UAV video recognition method according to any one of claims 1 to 7.

Images