Multi-modal low-altitude high-density high-isomerism unmanned aerial vehicle target identification method and system

Abstract

The invention discloses a multi-mode low-altitude high-density high-isomerism unmanned aerial vehicle target rapid and accurate identification method, and discloses a system and a medium of the multi-mode low-altitude high-density high-isomerism unmanned aerial vehicle target rapid and accurate identification method. The method not only breaks through the precision bottleneck of the traditional YOLO in the aspect of small target detection and the limitation of insufficient context sensing ability, but also remarkably improves the target recognition robustness of the model in a complex background. Through a high-quality feature reconstruction and scale adaptive mechanism, the system can realize refined and multi-category target detection at different scale levels, provides accurate positioning and intelligent identification capabilities for an electric power inspection unmanned aerial vehicle, and also provides real-time, stable and extensible visual basis support for quick response and target locking of an anti-unmanned aerial vehicle system. In the future, the framework can also be expanded to various unmanned aerial vehicle cooperation scenes such as forest fire prevention, traffic monitoring, disaster emergency and the like, and has good universality and engineering application prospects.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of overhead power transmission and distribution line operation and maintenance, in particular to a multi-modal low-altitude high-density high-heterogeneous unmanned aerial vehicle target identification method and system. BACKGROUND

[0002] In the field of unmanned aerial vehicle power inspection, image recognition and target detection technology has become one of the core technologies, and its research theory is extensive and in-depth. In practical application, a single operation of a single inspection unmanned aerial vehicle can generate a large amount of high-definition video data, which can reach TB or even PB level after long-term accumulation. Although these huge data streams are very rich in information capacity, they face a serious "information sparsity" problem.

[0003] Specifically, in these massive video data, the video clips that actually contain security threats (such as black flying unmanned aerial vehicle intrusion) account for a very small proportion. This typical "data enrichment but information sparsity" phenomenon makes the traditional full-amount video analysis method extremely inefficient - most of the computing resources are wasted on processing redundant clips without targets.

[0004] How to quickly filter out key video clips that may contain threat targets while filtering out a large amount of redundant data without information value under the condition of limited computing resources? This requires us to design an efficient pre-screening scheme that can significantly reduce computing complexity and storage requirements while ensuring detection performance.

[0005] Only by solving the efficiency challenge brought about by information sparsity can we truly realize real-time and reliable monitoring of security threats by the power inspection system and provide solid protection for the safety of power infrastructure. SUMMARY

[0006] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a multi-modal low-altitude high-density high-heterogeneous unmanned aerial vehicle target identification method, which uses an innovative two-stage detection method of "filtering-recognition". This method overturns the traditional full-amount video analysis paradigm, quickly retrieves key video clips that may contain targets first, and converts the "needle-in-a-haystack" full-amount detection into efficient target-oriented screening, fundamentally improving the computing efficiency and response speed of the system.

[0007] The present application constructs an efficient two-stage identification framework of "hash fast screening → accurate target detection". In the first stage, the AVHash algorithm quickly screens candidate video clips that may contain targets; in the second stage, only these candidate clips are applied to the computationally intensive SAM-YOLO high-precision detection algorithm to achieve accurate target identification and positioning.

[0008] The layered design effectively solves the fundamental contradiction between limited computing resources and real-time identification demand in a low-altitude high-density monitoring environment. By concentrating most of the computing resources on a small amount of high-value data, the overall efficiency of the system is improved by orders of magnitude, while maintaining a high level of detection performance.

[0009] The application further provides a system with the fusion method for evaluating dynamic measurement uncertainty.

[0010] The multi-modal low-altitude high-density high-heterogeneous unmanned aerial vehicle target rapid and accurate identification method according to the first aspect of the application is characterized in that it comprises the following steps:

[0011] For a given video with synchronized audio-visual signals, extract key frame images and corresponding audio segments of the video;

[0012] Map the extracted key frame images and corresponding audio segments to a shared latent space through a visual encoder and an audio encoder;

[0013] Implement multi-modal feature fusion using cross-attention mechanisms and gated attention, and generate semantic representations of the video segments through a multi-modal Transformer encoder;

[0014] Apply a target identification algorithm to the candidate video segments for target detection and identification, and output the class, location and mask information of the target.

[0015] The multi-modal low-altitude high-density high-heterogeneous unmanned aerial vehicle target rapid and accurate identification method according to the application has at least the following beneficial effects: the semantic segmentation capability of SAM is deeply integrated with the detection framework of YOLO, innovative methods such as multi-modal feature fusion, multi-scale pyramid enhancement and dynamic target perception are proposed, the theoretical application of target detection technology in multi-modal and multi-scale scenarios is enriched, a new theoretical approach and method system are provided for the field of unmanned aerial vehicle target detection, and the development and innovation of related theories are promoted. In the field of power inspection, this technology can significantly improve the detection accuracy and real-time performance of unmanned aerial vehicles for multi-scale targets, effectively reduce missed and false detections, and provide reliable protection for the safe operation of power facilities. By accurately identifying the identity of unmanned aerial vehicles, countermeasures against black unmanned aerial vehicles are implemented, the efficiency of intelligent collaborative security systems is improved, and economic losses caused by unmanned aerial vehicle safety problems are reduced. In addition, this technology can also be applied to other low-altitude fields such as smart cities and traffic management, and has wide practical application prospects and social and economic benefits.

[0016] According to some embodiments of the application, each key frame image is divided into M image blocks, and the visual encoder encodes the image blocks in order to represent the image.

[0017] According to some embodiments of the present invention, the audio segment corresponding to the keyframe image is divided into M audio blocks of equal size, and the audio encoder encodes the audio blocks in an orderly manner into the audio corresponding to the image for representation.

[0018] According to some embodiments of the present invention, the cross-attention mechanism is a multi-head cross-attention mechanism with a linear fully connected layer, and the gated attention is implemented by a gated attention fusion module, which accepts features of two modalities as input and connects the features of each modality row by row.

[0019] According to some embodiments of the present invention, the target recognition algorithm is the SAM-YLO algorithm, which includes:

[0020] The perception center module is used to extract visual features from the input raw image;

[0021] The scheduling hub module is used to schedule and fuse features at different levels and scales.

[0022] The output layer module is used to transform the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, so as to achieve accurate target localization and fine-grained recognition.

[0023] According to some embodiments of the present invention, the target recognition algorithm also employs a three-branch resolution adaptive detection head to process near-range, medium-range, and far-range targets respectively, with each branch using dilated convolutions with different dilation rates.

[0024] According to some embodiments of the present invention, the target recognition algorithm employs a cross-task joint loss function, which includes classification loss, detection box regression loss, and segmentation mask loss.

[0025] According to a second aspect of the present invention, rapid and accurate identification of multimodal, low-altitude, high-density, and highly heterogeneous unmanned aerial vehicle (UAV) targets is characterized by comprising:

[0026] The hash retrieval module can perform fast hash retrieval on the input video to filter out candidate video segments that may contain the target; including:

[0027] The video extraction element is capable of extracting key frame images and corresponding audio segments from a given video with synchronized audio-visual signals.

[0028] The mapping element is capable of mapping the extracted keyframe images and corresponding audio segments into a shared latent space through a visual encoder and an audio encoder;

[0029] The multimodal fusion element can achieve multimodal feature fusion using cross-attention mechanism and gated attention, and generate the semantic representation of the video segment through a multimodal Transformer encoder;

[0030] The target recognition module applies a target recognition algorithm to detect and recognize targets in the video clip, and outputs the target's category, location, and mask information.

[0031] According to some embodiments of the present invention, each keyframe image is divided into M image blocks, and the visual encoder encodes the image blocks into images in an orderly manner for representation.

[0032] According to some embodiments of the present invention, the audio segment corresponding to the keyframe image is divided into M audio blocks of equal size, and the audio encoder encodes the audio blocks in an orderly manner into the audio corresponding to the image for representation.

[0033] According to some embodiments of the present invention, the cross-attention mechanism is a multi-head cross-attention mechanism with a linear fully connected layer, and the gated attention is implemented by a gated attention fusion module, which accepts features of two modalities as input and connects the features of each modality row by row.

[0034] According to some embodiments of the present invention, the target recognition algorithm is the SAM-YLO algorithm, which includes:

[0035] The perception center module is used to extract visual features from the input raw image;

[0036] The scheduling hub module is used to schedule and fuse features at different levels and scales.

[0037] The output layer module is used to transform the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, so as to achieve accurate target localization and fine-grained recognition.

[0038] According to some embodiments of the present invention, the target recognition algorithm also employs a three-branch resolution adaptive detection head to process near-range, medium-range, and far-range targets respectively, with each branch using dilated convolutions with different dilation rates.

[0039] According to some embodiments of the present invention, the target recognition algorithm employs a cross-task joint loss function, which includes classification loss, detection box regression loss, and segmentation mask loss.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0042] Figure 1 This is a schematic diagram of a video hash retrieval architecture with a nested Transformer architecture, representing a fast and accurate multimodal low-altitude, high-density, and highly heterogeneous UAV target identification method according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the SAM-YOLO architecture for a multimodal, low-altitude, high-density, and highly heterogeneous UAV target fast and accurate identification method according to an embodiment of the present invention.

[0044] Figure 3 for Figure 1 The diagram shows a nested Transformer architecture for video hash retrieval where 4ViT acts as an image encoder.

[0045] Figure 4 A schematic diagram of the AST structure in the multimodal, low-altitude, high-density, and highly heterogeneous UAV target rapid and accurate identification method provided in the embodiments of the present invention;

[0046] Figure 5 This is a schematic diagram of the multi-head attention Transformer layer in the fast and accurate identification method for multimodal low-altitude high-density and highly heterogeneous UAV targets provided in the embodiments of the present invention;

[0047] Figure 6 This is a schematic diagram of the cross-modal attention mechanism in the fast and accurate multimodal low-altitude high-density and highly heterogeneous UAV target identification method provided in the embodiments of the present invention;

[0048] Figure 7 The structural block diagram of the multimodal, low-altitude, high-density, highly heterogeneous UAV target rapid and accurate identification system provided in the embodiments of the present invention is shown. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0050] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0051] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0052] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0053] Example 1

[0054] To effectively resolve the contradiction between massive video data and sparse targets in power line inspection scenarios, this invention proposes a fast and accurate multimodal low-altitude high-density and highly heterogeneous UAV target identification method, which includes at least the following steps:

[0055] Step S100: Quickly filter out candidate video segments that may contain the target using the AVHash algorithm.

[0056] Step S200: Apply a target recognition algorithm to the candidate video segment to detect and recognize the target, and output the target's category, location, and mask information.

[0057] In step S100, the AVHash algorithm provides a method to convert video from raw signals into a final semantic representation. This method first extracts N (e.g., 25) keyframe images and divides their audio into N sound waveform segments. Then, they are mapped to a shared latent space through separate visual Transformer and audio Transformer networks. Finally, the audiovisual embeddings are combined into the final video representation using a cross-attention mechanism and av-Transformer.

[0058] Given V videos, represented as The model aims to learn a hash function that maps each video to a compact binary code. Specifically, each video consists of N keyframes. and N audio clips Extracting ViT features from keyframes Extracting AST features from audio segments Where I and A correspond to the image and audio modalities, respectively. These modal features are then input into the respective branches of the model (see...). Figure 1As shown), the features are sequentially passed through individual Transformer encoders to generate image / audio embeddings, cross-attention for intermodal interactions, gated attention for feature fusion, a multimodal Transformer encoder, and a hash layer to generate the video embedding. The video embedding is then generated using the Sign(·) function. i The embedding binarization is used to obtain the final hash code b. i ∈{-1,+1} q , where q represents the code length.

[0059] The video hashing model consists of the following modules: "input", "modality-specific encoder", "cross attention", "gated attention", "multimodal encoder", "hash layer" and "loss function".

[0060] Regarding the target recognition algorithm in step S200 above, a multi-scale UAV target detection framework (such as...) is developed that integrates the Segment Anything Model (SAM) and YOLO series algorithms. Figure 2 As shown, this framework aims to leverage the advantages of SAM in semantic segmentation and boundary awareness, combined with YOLO's end-to-end high-efficiency detection capabilities, to construct a high-precision, robust detection system for complex multi-scale scenes. The core technical path of this framework is "semantic segmentation-guided feature enhancement + multi-scale pyramid optimization." By introducing semantic segmentation maps and boundary masks generated by SAM into the YOLO backbone feature extraction network, it enhances the focus on regions of interest and achieves structured alignment of multi-scale information. Simultaneously, based on a pyramid-style multi-resolution feature fusion mechanism, it further strengthens the model's ability to express the differences in morphology, size, and texture of targets within different distance ranges (including near, medium, and far distances), thereby achieving efficient collaboration from pixel-level target localization to identity category determination for UAVs.

[0061] Example 2

[0062] To further describe the fast and accurate identification method for multimodal low-altitude high-density and highly heterogeneous UAV targets provided in this application, another embodiment of this application is described in more detail based on Embodiment 1.

[0063] The step S100 above, which uses the AVHash algorithm to quickly filter out candidate video segments that may contain the target, specifically includes:

[0064] The model's input is an image-audio data pair. Superficially, the input is a video... i (i = 1, ..., V), which is actually composed of N keyframe-audio segment pairs.

[0065] For a keyframe Where (H, W) represents its resolution and C is the number of channels, we reshape it into a series of flattened 2D blocks. Where P is the block size, and M = HW / P 2 This represents the number of blocks generated. To accommodate the input of subsequent modules, linear projection is used to map the dimensions of the blocks to the input dimensions, as shown in Equation (1):

[0066]

[0067] Where I cls It is a learnable token added at the beginning of the serialized block embedding, where E represents linear projection and indicates the embedding at the corresponding position. j} 0 This represents the final input for the j-th keyframe. Then, it is input into the image encoder, for example, the Clip-ViT model (e.g., ...). Figure 3 As shown), it encodes ordered image blocks into an image representation. The image encoder contains L... I Each image encoder module consists of a multi-head self-attention block (MSA) and an MLP layer. The serialized image block embeddings are fed into the MSA layer, then through the MLP layer. The output of the (l-1)th MLP layer becomes the input to the l-th MSA layer. The image encoder formula is expressed as:

[0068]

[0069] in By performing the above procedure, we can obtain video v i Keyframe I j Image representation Image encoder output It comprises two parts, (1+M), corresponding to CLS and M image patches respectively. In the following sections, only the first part of the output will be used, abbreviated as... This is the final output of the image encoder. Then... Processed through an adapter:

[0070]

[0071] in It is a 768-dimensional keyframe. The N keyframes are processed according to formulas (1), (2), (3), and (4) to obtain the image feature sequence.

[0072] For an audio segment data First, a t-second audio signal is converted into a 128-dimensional log Mel spectrogram. This is processed by applying a 25ms Hamming window every 10ms, resulting in a 128×100t spectrogram. This spectrogram is then divided into M equal-sized patches, and each 2D patch is flattened into a C-dimensional vector. Thus, an audio segment can be represented as... Where C = 128 × 100 t / M.

[0073] Linear projection is still used to map the dimensions of the blocks to the input dimensions of the AST (Audio Spectrogram Transformer), and learnable positional embeddings P are added. a-pos As shown in formula (5):

[0074]

[0075] Where a cls It is a learnable token added at the beginning of the serialized block embedding, where E represents linear projection (in Figure 4 E in Chinese [·] Let a represent the result after linear projection. j E,P a-pos Represents a cls and Embed at the corresponding position. j} 0 Indicates video v i The final input for the j-th audio segment.

[0076] AST (Audio Spectrogram Transformer) adopts the Transformer encoder architecture and is powered by L A Layer coding blocks constitute. {w j} 0 After being input into the encoder, the video v can be obtained. i audio segment a j audio representation Similarly, using only the [CLS] part, abbreviated as As the final output of AST.

[0077] Then Processed through an adapter:

[0078]

[0079] in It is a 768-dimensional keyframe. The above processing is applied to N audio segments to obtain the audio feature sequence.

[0080] At this point, we will have video v i The keyframe sequence and the audio sequence are represented as two vector sequences. and Before feeding them into subsequent modules, a CLS token sequence needs to be added before the function, i.e.

[0081]

[0082] in and These correspond to the positional encoding of time-series audio and time keyframes, respectively.

[0083] Modality-specific encoders. The video hash model contains two modality-specific encoders designed to capture the unique features inherent in each modality. A standard Transformer is used as the audio and visual encoders. Figure 5 As shown.

[0084] Each encoder consists of L S It consists of several Transformer layers, each mainly consisting of Multi-Head-Self-Attention (MSA) (with A multi-heads). S The system consists of an MLP layer and an MLP layer. The output of the (l-1)th MLP layer is fed into the l-th multi-head self-attention (MSA) layer. Layer normalization (LN) is applied before each layer, and residual connections are merged after each layer. The MLP layer contains two linear fully connected layers with nonlinear GELU activation functions:

[0085]

[0086] Where l = 1, ..., L S Let m ∈ {A, I} represent the audio or visual modality. By following the process described above, the video v can be obtained. i Potential representation of audio and keyframe features Note the output. Each encoder's Sm contains (1+N) parts corresponding to the preset marker CLS and N audio / visual features. Then, the embedding of the first part is used, called... As a whole, the embedding of the second part is used, called As a representation of each feature.

[0087] Cross-modal attention. After encoding unimodal features, cross-attention is used to capture the semantic interaction between audio and visual modalities. For example... Figure 6 As shown.

[0088] Specifically, a multi-head cross-attention mechanism with a linear fully connected layer is used:

[0089]

[0090] Where m1 represents the target mode and m2 represents the source mode, the MCA calculation formula is given below:

[0091]

[0092] as well as

[0093]

[0094] Where W, and It is a learnable matrix, A c Equivalent to A A→I Or A I→A Therefore, the representation z rich in cross-modal information is obtained through the following formula. A→I , z I→A

[0095]

[0096]

[0097] Gated Attention Fusion Module. To fuse the semantics of different modalities, gated attention is introduced to determine the importance of each modality. For simplicity, this module is implemented using linear layers and the Tanh activation function. The module takes features from both modalities as input and concatenates the features of each modality row-wise. The output of the module is fed into a Softmax function to ensure that the sum of the weights of the two modalities equals 1.

[0098] [α,1-α]=Softmax(tanh(FC([z I→A ,z A→I ]))) (19)

[0099] The final fusion feature is described as follows:

[0100] z AI =α·z I→A +(1-α)·z A→I (20)

[0101] Multimodal encoder. To obtain the final representation of the video after fusing features from two modalities, a multimodal Transformer encoder is introduced. This encoder consists of L... M It consists of A Transformer layers, each Transformer layer consisting of multi-head self-attention (MSA) (with A multi-heads). MIt consists of an MLP layer and an input z. AI Previously, in z AI Add a cls token to the beginning, and its corresponding output serves as the final representation of the video:

[0102]

[0103] Where l = 1, ..., L M By following the process described above, video v can be obtained. i Potential representation Output Each encoder also contains a (1+N) part corresponding to a preset tag CLS and N multimodal features. The embedding of the first part is then employed. As its final expression.

[0104] In order to obtain video v i The final hash code is obtained by adding a hash layer on top of the multimodal encoder, namely an FC network with a Tanh(·) function. Formally, given video v i The output of its multimodal encoder Convert to a q-dimensional binary real-valued vector g i pass:

[0105]

[0106] Then forward it to the final binary code b i The Sign(·) function, i.e.

[0107] b i =Sign(g i )∈{-1,+1} q (25)

[0108] The network is trained using the InfoNCE loss. The InfoNCE loss consists of three classes of samples: an anchor sample a, a positive sample p, and n. o negative samples Anchor samples and positive samples belong to the same class, while anchor samples and negative samples belong to different classes. This InfoNCE loss maximizes the consistency between positive pairs and the inconsistency between positive and negative samples, and its formula is:

[0109]

[0110] Where a, p, and n represent the embedding vectors of the anchor sample, positive sample, and negative sample, respectively. sim(·,·) is the similarity function (e.g., cosine similarity), and τ is a hyperparameter controlling the discriminative power of the negative samples. This study constructed five types of positive-negative sample pairs:

[0111]

[0112] as well as

[0113]

[0114] Where L1 represents the alignment of audio and video signals in a common latent semantic space, and L2 represents the video-based contrastive loss in the video space. Then, the overall objective can be constructed as follows:

[0115] L=αL1+L2 (29)

[0116] The hyperparameter α is used here. By minimizing the InfoNCE loss, the video hashing model can learn discriminative hash codes, thereby achieving efficient large-scale video retrieval.

[0117] Given a new video, we can first compute its real-valued class binary vector using a trained video hash model:

[0118] g oos =AVHash(v oos (30)

[0119] AVHash is the video hash model, and this vector is then formatted into the final binary hash code:

[0120] b oos =Sign(g oos (31)

[0121] Furthermore, regarding step S200 described in Embodiment 1 above, a target recognition algorithm is applied to the candidate video segment to perform target detection and recognition, outputting the target's category, location, and mask information. Specifically, this includes:

[0122] The core of the multimodal feature fusion architecture design is to deeply couple the semantic segmentation capabilities of SAM with the detection framework of YOLO to solve the challenge of feature representation of multi-scale UAV targets in complex backgrounds. The following is a detailed description of the model structure.

[0123] 1) Backbone (the "sensory center" of multi-scale features)

[0124] Backbone serves as the "first stop" and "perceptual center" of the entire detection framework, primarily responsible for extracting rich and multi-layered visual features from the input raw image. Due to their large size variations and complex shapes, drone target detection relies on different levels of features: small targets typically depend on basic visual information such as texture and edges at the detail level, while large targets rely more on semantic-level overall contours and scene-related information. Therefore, Backbone needs to efficiently extract both fine-grained features and high-level semantic information.

[0125] The Backbone in this study consists of four key modules: CBS, C3, MP, and ELA N.

[0126] 1. CBS Module (orange block, labeled as (3,1)): This is the basic feature extraction unit, composed of convolution (Conv), batch normalization (BN), and the SiLU activation function. The convolution kernel size is 3×3, and the stride is 1, which can effectively capture detailed texture information in the image. Through the stacking of multiple CBS modules, the network gradually forms a comprehensive perception of basic visual features, thus laying the foundation for subsequent complex feature abstraction.

[0127] 2. C3 module (green block): A lightweight module based on CSPNet (Cross Stage Partial Network). By executing the main branch and the residual branch in parallel, it reduces the amount of computation and ensures the full flow of gradient information.

[0128] 3. MP Module (blue block, MaxPool): Max pooling is used to reduce the spatial size of the feature map while expanding the receptive field, enabling the model to focus on a wider range of contextual information. For distant or large drone targets, the MP module helps capture overall contour and structural information, improving the global scope of detection.

[0129] 4. ELAN Module (Green Block): This high-efficiency feature aggregation module uses multi-layer residual connections to achieve multi-stage information relay, promoting the continuous fusion of shallow detailed features and deep semantic features. The ELAN module effectively enhances feature representation capabilities, enabling the network to simultaneously consider the fine-grained details of small targets and the semantic integrity of large targets.

[0130] 2) Neck (the "scheduling hub" for feature interactions)

[0131] The Neck layer, serving as the model's "core scheduling layer," bears the crucial responsibility of multi-scale feature fusion and information interaction. This part employs a bidirectional feature pyramid structure combining FPN (Feature Pyramid Network) and PAN (Path Aggregation Network), along with the lightweight attention mechanism SimAM, to achieve precise scheduling and fusion of features at different levels and scales, thus achieving a balance between semantics and detail.

[0132] 1. Two-way feature pyramid structure

[0133] Top-down approach (FPN): Starting with high-level semantic features, the spatial resolution is first increased through upsampling, and then concatenated with mid-level features. This allows abstract semantic information to be "grounded," effectively supplementing low- and mid-level detailed features and enhancing the expressive power of small targets.

[0134] Bottom-up Path (PAN): Mid-level features are downsampled using max pooling and then concatenated with low-level features, enhancing the detail and semantic expression. This bidirectional feature fusion ensures that the Neck output features include both the global semantics of large targets and the edge and texture details of small targets, greatly improving the detection performance of multi-scale targets.

[0135] 2. SimAM Attention Mechanism (module in red box)

[0136] SimAM is a lightweight attention mechanism with no additional parameters that automatically adjusts the spatial and channel weights of feature maps based on neuron importance scores. By enhancing the response of important features and suppressing interference from irrelevant backgrounds, it effectively improves the model's focus on key target regions. For example, when detecting drones in power lines, SimAM can enhance the edge and contour features of the drone, reducing false positives caused by background trees and buildings, making the mask generated by the subsequent SAM module more accurately fit the target edges. This mechanism is particularly suitable for target detection in complex environments and low-contrast scenes.

[0137] 3. Other key modules

[0138] SPPCSPC module (red block): This module integrates Spatial Pyramid Pooling (SPP) and CSP structure, and uses multi-scale pooling (1×1, 3×3, 5×5) to capture spatial information of different sizes, enhancing the feature coverage and expressive ability of large targets.

[0139] C3, MP, and Concat modules: Continuing Backbone's lightweight design philosophy, they balance computational load and feature quality through downsampling and feature concatenation, ensuring the overall real-time performance and detection accuracy of the model.

[0140] 3) Prediction (the "result output layer" of the entire architecture)

[0141] The Prediction module is the "output layer" of the entire architecture. It is responsible for transforming the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, achieving accurate target localization and fine-grained recognition. This module achieves efficient output of feature maps through RepConv (reparameterized convolution).

[0142] 1. RepConv module (blue block): During the training phase, RepConv adopts a multi-branch convolutional structure to improve the model's expressive power and accuracy; during the inference phase, it merges multiple branches into a single convolution, which greatly speeds up the inference process and achieves a balance between accuracy and speed to meet the requirements of real-time detection.

[0143] 2. Feature Map Output: The final output consists of three feature maps at different scales, corresponding to the detection requirements of small, medium, and large targets, respectively. For the YOLO part, these feature maps are used to predict the location, class, and confidence of the target bounding boxes, achieving fast and accurate multi-target detection. For the SAM part, the feature maps serve as the core carrier for image embedding. Combined with the detection box information output by YOLO, they are processed by the SAM mask decoder to generate pixel-level instance segmentation masks for the targets, thus achieving seamless integration of detection and segmentation.

[0144] This design not only improves the accuracy of target recognition, but also provides a precise visual foundation for subsequent intelligent tasks such as target tracking, behavior analysis, and threat assessment.

[0145] Addressing the significant challenges of large target scale variations and intense background interference in power line inspection scenarios, leading to substantial false positives and false negatives due to complex backgrounds, this report designs and proposes an innovative three-branch resolution adaptive detection head structure and a loss function based on cross-task joint optimization (JLoss). This scheme fully leverages the advantages of the SAM and YOLO architectures, achieving robust detection and fine segmentation of UAV targets across different scales through differentiated feature extraction mechanisms and multi-task collaborative optimization strategies. This significantly improves the model's overall performance and practical value in complex power line inspection environments.

[0146] 4) Three-branch resolution adaptive detection head design

[0147] In power line inspection scenarios, the target scale varies greatly when using drones. This study innovatively designs a three-branch resolution adaptive detection head, dividing the overall detection task into three specialized branches: near-range, mid-range, and long-range. Each branch employs dilated convolution with different dilation rates to achieve refined capture and effective fusion of multi-scale features.

[0148] Near-field branch: A standard convolution with a dilation rate of d=1 is used to focus on detailed features on the drone's surface, such as model labels, paint textures, and propeller details. In this case, the receptive field size is determined by the convolution kernel size, calculated using the following formula:

[0149] RFnear=k+(k-1)(d-1)=k (32)

[0150] Where k is the kernel size.

[0151] Mid-range branch: Employs dilated convolutions with a dilation rate of d=2, designed to balance target details with contextual semantics, particularly suitable for capturing contour and structural features of UAVs at medium distances. The receptive field is calculated as follows:

[0152] RFmid=k+(k-1)(d-1)=k+(k-1) (33)

[0153] Long-range branch: A dilated convolution with an inflation rate of d=3 is employed. Addressing the characteristics of small target size and sparse information in long-range UAVs, the receptive field is further expanded to capture global contour features. The receptive field is calculated as follows:

[0154] RFfar=k+(k-1)(d-1)=k+2(k-1) (34)

[0155] The feature outputs of each branch are dynamically weighted and fused through a Scale-Aware Fusion Layer (SAFL). SAFL employs an adaptive weighting mechanism based on the mean of the feature maps, dynamically adjusting features at different scales using the following Softmax weight allocation formula:

[0156]

[0157] Where Fs represents the feature map of the s-th branch, avg(Fs) is its mean response, and λs are trainable weight coefficients. This mechanism ensures that the model can automatically adjust its attention to features of different scales under different scenarios and inputs, achieving feature complementarity and information sharing, and significantly improving the detection accuracy of small targets (pixel size < 80×80) and multi-scale targets.

[0158] 5) Construction of cross-task joint loss function

[0159] Traditional object detection networks (such as YOLOv8) primarily use classification loss (L... cls ) and bounding box regression loss (L boxHowever, due to the lack of constraints on the shape and mask of the target boundary, it is difficult to cope with the challenges of high false negative rates and inaccurate localization in complex backgrounds. To address this, this study introduces instance segmentation mask information from SAM (Segment Anything Model) and combines it with Dice Loss (Lmask) to construct a joint loss function J for both detection and segmentation tasks. Loss This achieves collaborative optimization of detection and segmentation tasks, and strengthens the guiding role of semantic segmentation on the detection boundary.

[0160] The joint loss function is defined as:

[0161] J Loss =L cls +L box +α L mask (36)

[0162] Where α = 0.5 is the weight coefficient for balancing the detection and segmentation tasks. The mathematical definitions of each component loss are as follows:

[0163] Classification loss. Cross-entropy loss is used to force the model to accurately classify the target category (such as whitelisted drones or birds):

[0164]

[0165] In the formula, N is the number of samples, C is the number of categories, and y i,c ∈{0,1} represents the true label. To predict probabilities.

[0166] Detection box regression loss. CIoU Loss is adopted, which introduces center point distance, scale, and aspect ratio constraints based on the Intersection over Union (IOU). The formula is as follows:

[0167]

[0168] Segmentation mask loss. Dice Loss is employed to improve segmentation accuracy in low-quality images (such as those with electromagnetic interference or those exposed to rain and fog) by enhancing the boundary distinction between the target and background.

[0169]

[0170] In the formula, H×W is the mask size, and M gt and M pred These are the real mask and the predicted mask, respectively.

[0171] Example 3

[0172] Another embodiment of the present invention provides a multimodal, low-altitude, high-density, highly heterogeneous unmanned aerial vehicle (UAV) target rapid and accurate identification system 20, such as...Figure 7 As shown, the system includes:

[0173] The hash retrieval module 201 is capable of performing fast hash retrieval on the input video to filter out candidate video segments that may contain the target.

[0174] The target recognition module 202 applies a target recognition algorithm to detect and recognize targets in the video segment, and outputs the target's category, location, and mask information.

[0175] The hash retrieval module 201 includes:

[0176] The video extraction element 2011 is capable of extracting key frame images and corresponding audio segments from a given video with synchronized audio-visual signals.

[0177] The mapping element 2012 is capable of mapping the extracted keyframe images and corresponding audio segments to a shared latent space through a visual encoder and an audio encoder;

[0178] The Multimodal Fusion Element 2013 can achieve multimodal feature fusion using cross-attention and gated attention mechanisms, and generate a semantic representation of the video segment through a multimodal Transformer encoder.

[0179] Furthermore, each keyframe image is divided into M image blocks, and the visual encoder encodes the image blocks into images in an orderly manner for representation.

[0180] Furthermore, the audio segment corresponding to the keyframe image is divided into M audio blocks of equal size, and the audio encoder encodes the audio blocks in an orderly manner into the audio corresponding to the image for representation.

[0181] Furthermore, the cross-attention mechanism is a multi-head cross-attention mechanism with a linear fully connected layer. The gated attention is implemented by the gated attention fusion module, which accepts features from two modalities as input and concatenates the features of each modality row by row.

[0182] Furthermore, the target recognition algorithm in the target recognition module 202 is the SAM-YLO algorithm, which includes:

[0183] The perception center module is used to extract visual features from the input raw image;

[0184] The scheduling hub module is used to schedule and fuse features at different levels and scales.

[0185] The output layer module is used to transform the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, so as to achieve accurate target localization and fine-grained recognition.

[0186] Furthermore, the target recognition algorithm also employs a three-branch resolution adaptive detection head to process near-range, medium-range, and far-range targets respectively, with each branch using dilated convolutions with different dilation rates.

[0187] Furthermore, the target recognition algorithm employs a cross-task joint loss function, which includes classification loss, detection box regression loss, and segmentation mask loss.

[0188] Example 4

[0189] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the above-described... Figure 1 The method shown is a fast and accurate identification method for multimodal, low-altitude, high-density, and highly heterogeneous UAV targets.

[0190] This application's embodiments, through high-quality feature reconstruction and scale adaptation mechanisms, enable the system to achieve refined, multi-category target detection at different scale levels. This provides precise positioning and intelligent recognition capabilities for power line inspection drones, while also offering real-time, stable, and scalable visual infrastructure support for rapid response and target locking in anti-drone systems. In the future, this framework can also be extended to various drone collaborative scenarios such as forest fire prevention, traffic monitoring, and disaster emergency response, demonstrating good versatility and promising engineering application prospects.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0193] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for rapid and accurate identification of multimodal, low-altitude, high-density, and highly heterogeneous unmanned aerial vehicle (UAV) targets, characterized in that: Includes the following steps: For a given video with synchronized audio-visual signals, extract the keyframe images and corresponding audio segments of the video; The extracted keyframe images and corresponding audio segments are mapped to a shared latent space using a visual encoder and an audio encoder. Multimodal feature fusion is achieved using cross-attention mechanism and gated attention, and the semantic representation of the video segment is generated by a multimodal Transformer encoder; The candidate video segments are subjected to target detection and recognition by a target recognition algorithm, and the target category, location and mask information are output.

2. The method according to claim 1, characterized in that, Each keyframe image is divided into M image blocks, and the visual encoder encodes the image blocks into images in an orderly manner for representation.

3. The method according to claim 2, characterized in that, The audio segment corresponding to the keyframe image is divided into M audio blocks of equal size, and the audio encoder encodes the audio blocks in an orderly manner into the audio corresponding to the image for representation.

4. The method according to claim 1, characterized in that, The cross-attention mechanism is a multi-head cross-attention mechanism with a linear fully connected layer. The gated attention is implemented by a gated attention fusion module, which accepts features from two modalities as input and concatenates the features of each modality row by row.

5. The method according to claim 1, characterized in that, The target recognition algorithm is the SAM-YLO algorithm, which includes: The perception center module is used to extract visual features from the input raw image; The scheduling hub module is used to schedule and fuse features at different levels and scales. The output layer module is used to transform the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, so as to achieve accurate target localization and fine-grained recognition.

6. The method according to claim 1, characterized in that, The target recognition algorithm also uses a three-branch resolution adaptive detection head to process near-range, medium-range, and far-range targets respectively, with each branch employing dilated convolutions with different dilation rates.

7. The method according to claim 1, characterized in that, The target recognition algorithm employs a cross-task joint loss function, which includes classification loss, bounding box regression loss, and segmentation mask loss.

8. A multimodal, low-altitude, high-density, highly heterogeneous unmanned aerial vehicle (UAV) target rapid and accurate identification system, characterized in that: include: The hash retrieval module can perform fast hash retrieval on the input video to filter out candidate video segments that may contain the target; including: The video extraction element is capable of extracting key frame images and corresponding audio segments from a given video with synchronized audio-visual signals. The mapping element is capable of mapping the extracted keyframe images and corresponding audio segments into a shared latent space through a visual encoder and an audio encoder; The multimodal fusion element can achieve multimodal feature fusion using cross-attention mechanism and gated attention, and generate the semantic representation of the video segment through a multimodal Transformer encoder; The target recognition module applies a target recognition algorithm to detect and recognize targets in the video clip, and outputs the target's category, location, and mask information.

9. The system according to claim 8, characterized in that, Each keyframe image is divided into M image blocks, and the visual encoder encodes the image blocks into images in an orderly manner for representation.

10. The system according to claim 9, characterized in that, The audio segment corresponding to the keyframe image is divided into M audio blocks of equal size, and the audio encoder encodes the audio blocks in an orderly manner into the audio corresponding to the image for representation.

11. The system according to claim 8, characterized in that, The cross-attention mechanism is a multi-head cross-attention mechanism with a linear fully connected layer. The gated attention is implemented by a gated attention fusion module, which accepts features from two modalities as input and concatenates the features of each modality row by row.

12. The system according to claim 8, characterized in that, The target recognition algorithm is the SAM-YLO algorithm, which includes: The perception center module is used to extract visual features from the input raw image; The scheduling hub module is used to schedule and fuse features at different levels and scales. The output layer module is used to transform the features extracted and fused from the front end into the final detection boxes and instance segmentation masks, so as to achieve accurate target localization and fine-grained recognition.

13. The system according to claim 8, characterized in that, The target recognition algorithm also uses a three-branch resolution adaptive detection head to process near-range, medium-range, and far-range targets respectively, with each branch employing dilated convolutions with different dilation rates.

14. The system according to claim 8, characterized in that, The target recognition algorithm employs a cross-task joint loss function, which includes classification loss, bounding box regression loss, and segmentation mask loss.

15. A computer-readable storage medium storing computer-executable instructions for performing the method of any one of claims 1 to 7.

Images