Multi-modal anti-drone detection method, device, equipment, storage medium and program product

Abstract

The application discloses a multi-modal anti-UAV detection method, device, equipment, storage medium and program product, and relates to the technical field of target detection. The method independently calculates the credibility score of each spatial position in the detection space based on multi-scale features, generates a spatial quality soft mask matrix corresponding to the mode, and enables the detection equipment to accurately perceive the local signal-to-noise ratio and reliability level of each pixel position on the feature map, thereby distinguishing the quality difference of different spatial positions in the same feature map. The multi-scale features are fused based on the spatial quality soft mask matrix after the constraint of each mode, so that the fusion process only performs cross-modal interaction between spatial positions with high credibility in each mode, and the features in the low-quality area are shielded and do not participate in the fusion calculation. The high-quality-oriented fusion strategy makes the finally generated multi-modal fusion features have high signal-to-noise ratio and discrimination ability, and improves the detection precision.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to multimodal anti-drone detection methods, devices, equipment, storage media, and program products. Background Technology

[0002] With the rapid popularization of consumer and industrial drones, the problem of unauthorized low-altitude flights has become increasingly prominent, posing a serious threat to airport operational safety, military facility protection, urban public safety, and personal privacy. Counter-drone technology has become a research hotspot and a necessity in the global security field. Currently, deep learning-based visual detection and tracking solutions, with their advantages of low deployment cost, intuitive target recognition, and strong scalability, have become the mainstream technology for civilian and short-range military counter-drone systems.

[0003] At the software algorithm model level, the current mainstream anti-drone technology can be divided into two main branches: single-modal visual detection and simple multimodal feature fusion. The performance of single-modal visual detection is highly dependent on lighting conditions and background complexity. In scenarios such as nighttime, strong backlight, fog, complex cloud cover, or urban building obstruction, the visible light features of small drones will be severely degraded or even completely disappear, leading to a sharp increase in the false negative rate. To compensate for the environmental adaptability defects of single-modal visible light detection, existing solutions usually introduce more modalities. However, in actual engineering practice, since each sensor of different modalities will generate localized, spatially unevenly distributed failure areas (such as overexposed areas of RGB and heat source interference areas of IR), forced fusion will introduce noise and erroneous information from inferior modalities into superior modalities, leading to feature contamination, and consequently causing detector positioning drift and decreased accuracy. Summary of the Invention

[0004] The main objective of this application is to provide a multimodal anti-drone detection method, apparatus, equipment, storage medium, and program product, aiming to solve the technical problem of low recognition accuracy in existing multimodal anti-drone detection methods.

[0005] To achieve the above objectives, this application proposes a multimodal anti-drone detection method, which includes: Obtain multi-scale features corresponding to different modalities in the detection space; Based on the multi-scale features, a confidence score is calculated for each spatial location in the space to be detected to obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. Obtain the lower bound suppression value of the quality, and apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality to obtain the constrained spatial quality soft mask matrix; Based on the spatial quality soft mask matrix after each modal constraint, the multi-scale features are fused to obtain multi-modal fused features; Based on the multimodal fusion features, detection is performed to obtain multimodal anti-drone detection results.

[0006] In one embodiment, the step of numerically constraining the matrix elements in the spatial quality soft mask matrix based on the lower quality bound suppression value to obtain the constrained spatial quality soft mask matrix includes: The matrix elements in the spatial quality soft mask matrix are traversed to determine the first matrix element that is lower than the lower quality suppression value and the second matrix element that is not lower than the lower quality suppression value. Set the first matrix element in the spatial quality soft mask matrix to zero, and suppress the second matrix element in the spatial quality soft mask matrix based on the lower quality limit suppression value to obtain the constrained spatial quality soft mask matrix.

[0007] In one embodiment, the step of acquiring multi-scale features corresponding to different modalities in the detection space includes: Modal images of different modalities in the space to be detected are acquired, wherein the temporal synchronization and spatial registration of each modality are achieved. Each modal image is input into the corresponding backbone network for feature extraction to obtain corresponding multi-scale features, wherein the multi-scale features include shallow high-resolution feature maps and deep feature maps, and the deep feature maps include mid-level feature maps and deep semantic feature maps; Accordingly, the step of calculating a confidence score for each spatial location in the space to be detected based on the multi-scale features to obtain the spatial quality soft mask matrix of the corresponding modality includes: Based on the deep feature map, a confidence score is calculated for each spatial location in the space to be detected, and the spatial quality soft mask matrix of the corresponding modality is obtained.

[0008] In one embodiment, the step of fusing the multi-scale features based on the spatial quality soft mask matrix after each modality constraint to obtain multi-modal fused features includes: The current query matrix is ​​constructed based on the current deep feature map of the current modality, and the target key matrix and target value matrix are constructed based on the target deep feature map of the target modality. The current modality is any one of the different modalities, and the target modality is any one of the different modalities other than the current modality. The spatial quality soft mask matrix after the target modal constraint is multiplied element-wise with the target value matrix to obtain the adjustment gating; The current branch output of the target mode, guided by the current mode, is determined based on the adjustment gating, the current query matrix, and the target key matrix. The multimodal fusion feature is obtained by weighted fusion of the branch outputs corresponding to each modality, the multi-scale features, and the constrained spatial quality soft mask matrix.

[0009] In one embodiment, the step of detecting based on the multimodal fusion features to obtain multimodal anti-drone detection results includes: The multimodal fusion features and the shallow high-resolution feature maps of each modality are input into the detection head for target detection to obtain the target bounding box and classification confidence score corresponding to each modality. The multimodal anti-drone detection result includes the target bounding box and classification confidence score corresponding to each modality.

[0010] In one embodiment, after the step of detecting based on the multimodal fusion features to obtain the multimodal anti-drone detection result, the method further includes: A weighted fusion score for each target bounding box is determined based on the classification confidence of each modality; Determine the intersection-union distance and weighted fusion score difference between all current target bounding boxes and predicted trajectories in the current frame, and determine the matching cost matrix between the current target bounding box and the historical predicted trajectory based on the intersection-union distance and the weighted fusion score difference. The modal image includes images of multiple consecutive frames in the corresponding modality, and the current frame is any one of the multiple consecutive frames. The current predicted trajectory of the current frame is obtained by constraining the matching cost matrix.

[0011] Furthermore, to achieve the above objectives, this application also proposes a multimodal anti-drone detection device, which includes: The feature acquisition module is used to acquire multi-scale features corresponding to different modalities in the detection space; The matrix generation module is used to calculate the confidence score for each spatial location in the space to be detected based on the multi-scale features, and obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. The matrix constraint module is used to obtain the lower bound suppression value of the quality, and to apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality, so as to obtain the constrained spatial quality soft mask matrix. The feature fusion module is used to fuse the multi-scale features based on the spatial quality soft mask matrix after each modality constraint to obtain multi-modal fused features; The target detection module is used to perform detection based on the multimodal fusion features to obtain multimodal anti-drone detection results.

[0012] In addition, to achieve the above objectives, this application also proposes a multimodal anti-drone detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal anti-drone detection method described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multimodal anti-drone detection method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multimodal anti-drone detection method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application obtains multi-scale features corresponding to different modalities in the detection space; calculates a confidence score for each spatial location in the detection space based on the multi-scale features to obtain a spatial quality soft mask matrix for the corresponding modality. The spatial quality soft mask matrix characterizes the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding modality at each spatial location; obtains a quality lower bound suppression value and applies numerical constraints to each matrix element in the spatial quality soft mask matrix based on the quality lower bound suppression value to obtain a constrained spatial quality soft mask matrix; fuses the multi-scale features based on the constrained spatial quality soft mask matrices for each modality to obtain multi-modal fused features; and performs detection based on the multi-modal fused features to obtain multi-modal anti-drone detection results. By independently calculating a confidence score for each spatial location in the detection space based on multi-scale features and generating a spatial quality soft mask matrix for the corresponding modality, the detection device can accurately perceive the local signal-to-noise ratio and reliability level of each pixel location on the feature map. The aforementioned spatial quality soft mask matrix achieves pixel-level granularity. Compared to existing fusion methods employing global equal-weight strategies or channel-level attention mechanisms, this scheme can distinguish quality differences at different spatial locations within the same feature map, achieving refined quality assessment with "different weights for the same image." By fusing multi-scale features based on the spatial quality soft mask matrix constrained by each modality, the fusion process only involves cross-modal interactions between spatial locations with high credibility within each modality. Features in low-quality regions are masked and do not participate in the fusion calculation. This fusion mechanism breaks through the paradigm limitations of "global equal-weight fusion" and "blindly forced fusion" in existing technologies. Each modality only provides effective information to another modality in its reliable regions, actively blocking information transmission in its degraded regions to avoid contaminating the other. This high-quality-oriented fusion strategy results in multi-modal fused features with high signal-to-noise ratio and discriminative power, effectively suppressing performance degradation in complex environments compared to existing fusion methods. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the multimodal anti-drone detection method of this application. Figure 2This is a schematic diagram of the improved dual-stream YOLO dual-branch backbone network provided in an example embodiment of the multimodal anti-drone detection method of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the multimodal anti-drone detection method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the multimodal anti-drone detection method of this application; Figure 5 This is a flowchart illustrating the multi-scale feature fusion and trajectory tracking process in one implementation of the multimodal anti-UAV detection method of this application. Figure 6 This is a flowchart illustrating the overall process of 7D motion-visual semantic encoding and large model-driven cognitive reasoning in one implementation of the multimodal anti-drone detection method of this application. Figure 7 This is a schematic diagram of the module structure of the multimodal anti-drone detection device according to an embodiment of this application; Figure 8 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the multimodal anti-drone detection method in this application embodiment.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is as follows: First, acquire multi-scale features corresponding to different modalities in the detection space. Second, calculate the confidence score for each spatial location in the detection space based on the multi-scale features to obtain the corresponding spatial quality soft mask matrix. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding modality at each spatial location. Third, acquire the lower bound suppression value of the quality and apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value to obtain the constrained spatial quality soft mask matrix. Fourth, fuse the multi-scale features based on the constrained spatial quality soft mask matrix of each modality to obtain multi-modal fused features. Fifth, perform detection based on the multi-modal fused features to obtain the multi-modal anti-drone detection result.

[0023] This application provides a solution that, when obtaining multi-scale features of different modalities in the space to be detected, first independently calculates a confidence score for each spatial location, forming a corresponding spatial quality soft mask matrix. This breaks the default assumption of "global equal weight" in traditional fusion paradigms, instead giving the system the ability to independently evaluate the quality of each pixel in the feature map. Specifically, the value range of each element in the spatial quality soft mask matrix is ​​constrained to the interval (0, 1). The closer the value is to 1, the higher the feature signal-to-noise ratio and the stronger the reliability of the corresponding modality at that spatial location. Conversely, the closer the value is to 0, the more severely the feature at that location has been affected by degradation factors such as overexposure, blurring, or thermal interference. Through this design, the system can clearly identify low-quality areas such as "which area in the RGB image has failed" or "which pixel in the infrared image is interfered with."

[0024] Meanwhile, this application introduces a clear lower bound suppression value for quality and applies a hard numerical constraint to each matrix element in the spatial quality soft mask matrix based on this value: for any spatial location below this threshold, its quality score is forcibly set to zero; for any spatial location above this threshold, its quality score is proportionally remapped to the (0, 1) interval. By hard truncation precisely setting the mask value of low-quality regions to zero, the value matrix of these locations is completely masked when participating in subsequent calculations, ensuring that even if there is a large overexposure in the visible light (RGB) image or local interference in the infrared (IR) image in a frame, the features of these failed regions will not be transferred to another mode.

[0025] Finally, guided by the constrained spatial quality soft mask matrix for each modality, this embodiment completes the cross-attention fusion of multi-scale features. Specifically, the mask matrix directly acts on the Value matrix in the attention mechanism, and after element-wise multiplication, the feature values ​​of low-quality regions are set to zero and then fed into Softmax for weighted summation. The essence of this fusion method is to perform cross-modal complementarity within the "trusted region," while abandoning all information of that modality in the "untrusted region" and completely handing over the fusion leadership to another modality. Through this mechanism, the system can dynamically decide "who is trustworthy to follow" at different spatial locations—RGB trusted regions are detected mainly using visible light features, IR trusted regions are detected mainly using infrared features, and regions where neither is trustworthy are naturally suppressed. Compared with the existing technology that forcibly sums or globally weights regardless of pixel quality, this "guided" approach avoids the problem of injecting erroneous information into the high-quality feature stream, so that the fused feature map no longer contains contamination components caused by local modal failures. Meanwhile, when performing target detection in multiple consecutive frames, different modal bounding boxes can be used to achieve more accurate target detection and tracking, avoiding the reduction in detection accuracy caused by the disappearance of the target in a certain modality due to occlusion, temperature changes, noise, etc.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, server, etc., or an electronic device or virtual device capable of realizing the above functions. The following description uses a multimodal anti-drone detection device (hereinafter referred to as the detection device) as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the embodiments of this application provide a multimodal anti-drone detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multimodal anti-drone detection method of this application.

[0028] In this embodiment, the multimodal anti-drone detection method includes steps S10 to S50: Step S10: Obtain multi-scale features corresponding to different modalities in the detection space; Step S20: Calculate the confidence score for each spatial location in the space to be detected based on the multi-scale features to obtain the spatial quality soft mask matrix of the corresponding modality. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features corresponding to the modality at each spatial location.

[0029] It should be noted that the aforementioned space to be detected can refer to the target space range monitored by the detection equipment. Within this space, acquisition devices such as visible light cameras, infrared thermal imagers, lidar, and acoustic detectors can be deployed to collect modal images of different modes. By acquiring these modal images, the multi-scale features corresponding to different modes within the space to be detected can be determined, such as the visible light multi-scale features of the visible light mode, the infrared multi-scale features of the infrared mode, and the acoustic multi-scale features of the sound mode. This space to be detected can be airspace requiring UAV monitoring, such as airport airspace, airspace above important urban facilities, or the perimeter of military protected areas; this application does not impose any limitations on this.

[0030] For ease of explanation, this application uses visible light and infrared modes as examples to illustrate the solution, but this does not constitute a limitation on the practical application of the solution. In practical applications, more modes can be selected to achieve anti-drone detection.

[0031] Understandably, the visible light mode can be used to collect the apparent texture, contour, and color information of each target in the detection space, while the infrared mode can be used to collect the thermal radiation distribution information of each target in the detection space. By fusing the multi-scale features corresponding to the information collected from different modes, more accurate anti-drone detection can be achieved.

[0032] It should be understood that the aforementioned multi-scale features can refer to a set of feature maps with different spatial resolutions and semantic abstraction levels obtained after multi-level feature extraction of the input image. These typically include shallow and deep features, and this application does not impose any limitations on this. The aforementioned spatial location can refer to the spatial coordinates of each pixel on the feature map, and the feature vector of each spatial location can be used to characterize the visual information of the corresponding local region in the modal image.

[0033] It should be noted that the aforementioned spatial quality soft mask matrix can be a two-dimensional matrix matching the spatial size of the feature map. Each element of the spatial quality soft mask matrix corresponds to a confidence score (such as signal-to-noise ratio, reliability, etc.) at a spatial location on the feature map, thus characterizing the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding modality at each spatial location. For example, for a deep semantic feature map with a resolution of 20×20, its corresponding spatial quality soft mask matrix is ​​also a 20×20 two-dimensional matrix. The aforementioned local signal-to-noise ratio and reliability can refer to the ratio of the effective signal intensity to the noise intensity of a certain local region on the feature map, or a comprehensive evaluation of the confidence level of the features in that local region based on this ratio, thereby reflecting whether the features at that location should be preserved or masked in subsequent cross-modal fusion.

[0034] In some embodiments of this application, the step of obtaining multi-scale features corresponding to different modalities in the detection space includes: acquiring modal images of different modalities in the detection space, wherein each modality is time-synchronized and spatially registered; inputting each modal image into a corresponding backbone network for feature extraction to obtain corresponding multi-scale features, wherein the multi-scale features include shallow high-resolution feature maps and deep feature maps, and the deep feature maps include mid-level feature maps and deep semantic feature maps; correspondingly, the step of calculating a confidence score for each spatial location in the detection space based on the multi-scale features to obtain a spatial quality soft mask matrix of the corresponding modality includes: calculating a confidence score for each spatial location in the detection space based on the deep feature maps to obtain a spatial quality soft mask matrix of the corresponding modality.

[0035] It is understood that the aforementioned modal images can refer to raw image data collected by different sensors. Each modal image can be used to reflect different physical attribute information of the target in the same detection space. The modal image can be an image sequence composed of multiple frames. For modal images of different modalities, time synchronization and spatial registration can be performed in advance to align the different modal images in both spatial and temporal dimensions.

[0036] Taking the visible light mode and the infrared mode as examples, in order to achieve multi-scale feature extraction, this application proposes a dual-stream front-end sensing network, which is an RGB and IR dual-branch front-end sensing network built on dual-stream YOLO.

[0037] In some embodiments of this application, the hardware front end can acquire modal images of different modes with strict time synchronization and spatial registration, such as visible light (RGB) image sequences and infrared (IR) thermal imaging image sequences. To ensure multi-scale feature alignment, for heterogeneous input image frames, the resolution of the input images can be uniformly scaled to a preset resolution using a resolution normalization network. For example, using a dual-stream resolution normalization network, the resolution of the input images can be uniformly scaled to 640 × 640 pixels. ; If the infrared sensor outputs a single-channel grayscale image, it can be normalized to a three-channel matrix structure identical to RGB by copying the channel. Subsequently, the normalized visible light image... With infrared images The images are input in parallel to an improved dual-stream YOLO dual-branch backbone network for feature extraction, thereby obtaining multi-scale spatial features. That is, each modality image is input into its corresponding backbone network for feature extraction to obtain the corresponding multi-scale features. The implementation process can be as follows: Figure 2 Example, Figure 2 This diagram illustrates an improved dual-stream YOLO dual-branch backbone network provided in an example of the multimodal anti-drone detection method of this application. It shows a process of extracting spatial features from corresponding modal images (e.g., RGB and IR backbones) using a YOLO26-based backbone network with different modalities. By directly outputting the final detection result and skipping the traditional non-maximum suppression (NMS) processing step, a high initial recall rate is achieved.

[0038] In the example of this application, the backbone network can extract three spatial local feature matrices at different scales at different depths, namely feature maps P3, P4, and P5: ; ; One example is as follows: the P3 feature map has a resolution of 80×80 and 64 channels, preserving high-resolution spatial details and texture features; the P4 feature map has a resolution of 40×40 and 128 channels, balancing spatial structure and moderate semantic features; the P5 feature map has a resolution of 20×20 and 256 channels, possessing extremely high semantic information and global contour features. The P3 feature map (P3 layer stride = 8) is not fused to preserve high-frequency details of a single modality, while the P4 and P5 feature maps (P4 layer stride = 16, P5 layer stride = 32) are input into the subsequent feature fusion module for deep cross-modal association.

[0039] It is understandable that the aforementioned shallow high-resolution feature maps can refer to the feature maps output by the early convolutional layers near the input end of the backbone network. These feature maps have a large spatial size and high resolution, preserving the fine spatial location information and edge contour details of the input image, such as the P3 feature map (resolution 80×80). The aforementioned deep feature maps can refer to the feature maps output by the deep convolutional layers near the output end of the backbone network. These feature maps have a small spatial size but a large number of channels after multiple downsampling, and have a large receptive field and rich semantic information. Deep feature maps further include mid-level feature maps and deep semantic feature maps, such as the P4 feature map (resolution 40×40) and the P5 feature map (resolution 20×20), where P4 is a mid-level feature map and P5 is a deep semantic feature map.

[0040] In some embodiments of this application, a deep multimodal feature fusion module with Quality-Aware Cross-Attention (MQCA) can be deployed. Specifically, embodiments of this application can deploy MQCA modules in the P4 and P5 deep feature layers of the backbone network output for selective fusion. The MQCA module proposed in this application aims to evaluate the spatial local quality of each pixel in the feature map, thereby dynamically adjusting the fusion ratio and suppressing noise propagation of degenerate modalities.

[0041] It should be noted that, in order to quantify the reliability of each spatial location in deep feature maps of different modalities, this embodiment of the application may introduce a spatial quality estimation network. For the input deep feature maps of different modalities, their spatial quality soft mask matrices can be extracted respectively through the corresponding lightweight two-layer convolutional networks.

[0042] For example, for visible light modes and infrared modes, their corresponding spatial quality soft mask matrices are extracted. and The method is as follows: ; ; in, , As a lightweight convolutional layer, it can reduce the dimensionality of the input channels of the input features while maintaining the spatial resolution. This represents the GELU activation function. Let represent the Sigmoid activation function, such that the spatial quality soft mask matrix corresponding to the visible light mode is... Spatial quality soft mask matrix corresponding to infrared mode The value of is strictly limited to the range of (0,1). The closer the value is to 1, the higher the credibility of the spatial location feature (without exposure, noise or motion blur interference).

[0043] In its implementation, the detection device first acquires modal images of different modalities within the detection space. For example, the detection device controls a visible light camera and an infrared thermal imager deployed at the front end to synchronously perform image acquisition, ensuring precise alignment between the visible light image output by the visible light camera and the infrared thermal image output by the infrared thermal imager at the acquisition time. Simultaneously, the detection device performs a geometric transformation operation on the infrared image using a pre-stored spatial transformation matrix, maintaining a pixel-level correspondence between the infrared and visible light images in the spatial field of view. This ensures that information at the same pixel coordinate position in both modal images corresponds to the same spatial point in the physical world. After completing the aforementioned time synchronization and spatial registration, the detection device obtains both the visible light modal image and the infrared modal image. Subsequently, the detection device can input the modal images of different modalities into the corresponding backbone network for feature extraction, obtaining shallow high-resolution feature maps, mid-level feature maps, and deep semantic feature maps. These feature maps collectively constitute the multi-scale features of the corresponding modality. Based on these obtained mid-level and deep semantic feature maps, a confidence score can be calculated for each spatial location in the detection space, yielding the spatial quality soft mask matrix for the corresponding modality. This design allows for independent feature modeling of different modalities' physical characteristics, avoiding the misalignment problem caused by forcibly inputting heterogeneous data into a single network. This significantly improves the feature representation quality of small and occluded targets. Quality assessment is performed only on deep semantic feature maps because deep features have a larger receptive field and richer semantic information, more accurately reflecting the overall credibility of the target region. Meanwhile, shallow high-resolution feature maps, without quality estimation network processing, directly retain their original spatial detail information for subsequent detection head use, thus ensuring the accuracy of quality assessment while avoiding the loss of detailed information. Through the generation of the aforementioned spatial quality soft mask matrix, the detection device provides a refined spatial gating basis for selectively masking low-quality features during subsequent cross-modal fusion, blocking the injection path of inferior noise into effective features at the source.

[0044] Step S30: Obtain the lower bound suppression value of the quality, and apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality to obtain the constrained spatial quality soft mask matrix. Step S40: Based on the spatial quality soft mask matrix after each modality constraint, the multi-scale features are fused to obtain multi-modal fused features; Step S50: Detection is performed based on the multimodal fusion features to obtain multimodal anti-drone detection results.

[0045] It should be noted that the aforementioned lower quality limit suppression value can be a pre-set numerical threshold, which can be used to determine which matrix elements in the spatial quality soft mask matrix correspond to features with too low quality and need to be completely masked. This lower quality limit suppression value can be taken in an open interval from 0 to 1, for example, the lower quality limit suppression value can be set to 0.10.

[0046] It is understandable that the aforementioned numerical constraints involve mathematically transforming and restricting each matrix element in the spatial quality soft mask matrix using a lower quality bound suppression value. This process transforms the matrix elements in the spatial quality soft mask matrix into constrained matrix elements based on the lower quality bound suppression value, thus obtaining the constrained spatial quality soft mask matrix. This lower quality bound suppression method effectively blocks the influence of low-quality noise.

[0047] It should be understood that the aforementioned multimodal fusion feature is a feature representation obtained by fusing multi-scale features of multiple modalities. This multimodal fusion feature contains effective information from multiple modalities and can be used for subsequent target detection tasks to obtain multimodal UAV detection results.

[0048] This application embodiment acquires multi-scale features corresponding to different modalities in the detection space; calculates a confidence score for each spatial location in the detection space based on the multi-scale features to obtain a spatial quality soft mask matrix for the corresponding modality, wherein the spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding modality at each spatial location; acquires a quality lower bound suppression value, and applies numerical constraints to each matrix element in the spatial quality soft mask matrix based on the quality lower bound suppression value to obtain a constrained spatial quality soft mask matrix; fuses each multi-scale feature based on the constrained spatial quality soft mask matrix for each modality to obtain a multi-modal fused feature; and performs detection based on the multi-modal fused feature to obtain a multi-modal anti-drone detection result. By independently calculating a confidence score for each spatial location in the detection space based on multi-scale features and generating a spatial quality soft mask matrix for the corresponding modality, the detection device can accurately perceive the local signal-to-noise ratio and reliability level of each pixel location on the feature map. The aforementioned spatial quality soft mask matrix achieves pixel-level granularity. Compared to existing fusion methods employing global equal-weight strategies or channel-level attention mechanisms, this scheme can distinguish quality differences at different spatial locations within the same feature map, achieving refined quality assessment with "different weights for the same image." By fusing multi-scale features based on the spatial quality soft mask matrix constrained by each modality, the fusion process only involves cross-modal interactions between spatial locations with high credibility within each modality. Features in low-quality regions are masked and do not participate in the fusion calculation. This fusion mechanism breaks through the paradigm limitations of "global equal-weight fusion" and "blindly forced fusion" in existing technologies. Each modality only provides effective information to another modality in its reliable regions, actively blocking information transmission in its degraded regions to avoid contaminating the other. This high-quality-oriented fusion strategy results in multi-modal fused features with high signal-to-noise ratio and discriminative power, effectively suppressing performance degradation in complex environments compared to existing fusion methods.

[0049] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating Embodiment 2 of the multimodal anti-drone detection method of this application.

[0050] like Figure 3 As shown in this embodiment, the step of numerically constraining the matrix elements in the spatial quality soft mask matrix based on the lower quality limit suppression value to obtain the constrained spatial quality soft mask matrix includes: Step S21: Traverse each matrix element in the spatial quality soft mask matrix to determine the first matrix element that is lower than the quality lower bound suppression value and the second matrix element that is not lower than the quality lower bound suppression value. Step S22: Set the first matrix element in the spatial quality soft mask matrix to zero, and suppress the second matrix element in the spatial quality soft mask matrix based on the lower quality suppression value to obtain the constrained spatial quality soft mask matrix.

[0051] It is understandable that the aforementioned traversal can refer to the process by which the detection device sequentially accesses each element of the spatial quality soft mask matrix in a certain order, performing read and judgment operations on each element. For example, it can scan all elements in the matrix row by row in a left-to-right, top-to-bottom order. The first matrix element can refer to those matrix elements in the spatial quality soft mask matrix whose values ​​are lower than the lower quality limit suppression value. The spatial location corresponding to these matrix elements has an excessively low quality score and belongs to a severely degraded region. The second matrix element can refer to those matrix elements in the spatial quality soft mask matrix whose values ​​are not lower than the lower quality limit suppression value. The spatial location feature quality score corresponding to these elements is within an acceptable range and has not yet reached the level that requires complete masking.

[0052] It should be understood that the lower limit suppression values ​​for different modes can be the same or different, and the embodiments of this application do not impose any restrictions on this.

[0053] In some embodiments of this application, in order to completely block noise injection of severely degraded features, the spatial quality mask is truncated and mapped by setting a lower quality bound suppression value: matrix elements corresponding to spatial locations below the lower quality bound suppression value are forcibly cleared to zero, so that the network completely distrusts the spatial information of the severely degraded region during backpropagation; matrix elements not below the lower quality bound suppression value can be suppressed; the specific processing can be shown in the following example: ; Here, `clamp(a,b,c)` is a numerical constraint function used to constrain the value `a` to between a specified minimum value `b` and a maximum value `c`. For example, in the formula above, Less than the lower limit of mass inhibition value hour If the value is less than 0, it can be set to zero. The elements of the suppressed matrix are .

[0054] In its implementation, when obtaining the lower quality bound suppression value, the detection device can perform numerical constraint operations on the spatial quality soft mask matrix corresponding to each mode. The detection device can iterate through each matrix element in the soft mask matrix corresponding to each mode, sequentially reading the value of each element. During this iteration, the detection device compares the currently read matrix element value with the lower quality bound suppression value for that mode to determine the type of the element. When the value of a matrix element is lower than the lower quality bound suppression value, the detection device marks it as the first matrix element; when the value is not lower than the lower quality bound suppression value, it marks it as the second matrix element. For elements marked as first matrix elements, the detection device performs a zeroing operation, directly setting the value of the element to 0. After this zeroing operation, the corresponding positions in the soft mask matrix of the previously low-quality degradation regions become 0. In subsequent fusion processes, the features at these positions will be completely masked and will not affect the fusion result. Elements marked as second matrix elements can be suppressed, so that the quality edge regions that just reached the threshold standard (such as the original value of 0.10) are mapped to values ​​close to 0, while the original high quality regions (such as the original value of 1.00) are mapped to values ​​close to 1, further widening the numerical gap between high quality regions and edge quality regions, and enhancing the distinguishability of the soft mask matrix for reliable regions.

[0055] This embodiment of the application iterates through each element of the spatial quality soft mask matrix to determine the first matrix element below the lower quality bound suppression value and the second matrix element above the lower quality bound suppression value. The first matrix element in the spatial quality soft mask matrix is ​​set to zero, and the second matrix element is suppressed based on the lower quality bound suppression value to obtain a constrained spatial quality soft mask matrix. By iterating through each element of the spatial quality soft mask matrix and comparing the value of each element with the lower quality bound suppression value, the first matrix element below the threshold and the second matrix element above the threshold are accurately identified. The granularity of the above traversal and classification operations reaches the pixel level. Compared with existing technologies that use image-level quality assessment or channel-level attention weighting, this scheme can distinguish quality differences at different spatial locations in the same feature map, marking "severely degraded areas" and "acceptable areas" respectively. For example, in the same frame feature map, cloud-occluded areas are marked as the first matrix element, and UAV target areas are marked as the second matrix element, laying the classification foundation for subsequent implementation of completely differentiated processing strategies for these two types of areas. By directly setting the first matrix element below the quality lower bound suppression value to zero, the corresponding value in the spatial quality soft mask matrix becomes zero, ensuring that the features of low-quality regions are completely zeroed in the spatial quality soft mask matrix. This eliminates the possibility of low-quality noise participating in cross-modal fusion calculations and avoids the problem of inferior modes contaminating the features of superior modes. By performing numerical suppression on the second matrix element based on the quality lower bound suppression value, the element values ​​originally distributed between the threshold and 1 are remapped to the 0-1 range. This widens the numerical gap between high-quality regions and edge-quality regions, making the spatial quality soft mask matrix more clearly and sharply distinguishable of reliable regions, providing a more explicit spatial gating signal for subsequent fusion modules.

[0056] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and / or second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating Embodiment 3 of the multimodal anti-drone detection method of this application.

[0057] like Figure 4 As shown in the embodiment of this application, the step of fusing the multi-scale features based on the spatial quality soft mask matrix after each modality constraint to obtain multi-modal fused features includes: Step S41: Construct the current query matrix based on the current deep feature map of the current modality, and construct the target key matrix and target value matrix based on the target deep feature map of the target modality, wherein the current modality is any one of the different modalities, and the target modality is any one of the different modalities other than the current modality; Step S42: Based on the target modal constraints, the spatial quality soft mask matrix and the target value matrix are multiplied element-wise to obtain the adjustment gating; Step S43: Determine the current branch output of the target mode guided by the current mode based on the adjustment gating, the current query matrix, and the target key matrix; Step S44: Perform weighted fusion based on the branch outputs corresponding to each modality, the multi-scale features, and the constrained spatial quality soft mask matrix to obtain multi-modal fusion features.

[0058] It is understandable that the current modality is the currently selected modality, which can be any one of the different modalities, while the target modality is any other modality besides the current modality. The current deep feature map is the mid-level feature map and deep semantic feature map corresponding to the current modality, and the target deep feature map is the mid-level feature map and deep semantic feature map corresponding to the target modality.

[0059] It should be noted that the current query matrix is ​​the feature matrix obtained by multiplying the current deep feature map by a learnable linear transformation weight matrix. Each row vector in the current query matrix represents the feature representation of a spatial location in the current modality used to query information related to the target modality. The aforementioned target key matrix can be the feature matrix obtained by multiplying the target deep feature map by another learnable linear transformation weight matrix. Each row vector in this matrix represents the feature representation of a spatial location in the target modality features used for query matching. The aforementioned target value matrix can be the feature matrix obtained by multiplying the target deep feature map by a third learnable linear transformation weight matrix. Each row vector in this matrix represents the actual content information of a spatial location in the target modality features to be passed to the output.

[0060] It should be explained that the embodiments of this application utilize a cross-attention mechanism: a query matrix is ​​constructed based on the current deep feature map corresponding to the current modality, and a value matrix and a key matrix are constructed based on the current deep feature map corresponding to the target modality, thereby achieving cross-modal information interaction and allowing one modality to extract effective information from another modality. Therefore, the embodiments of this application can include branches corresponding to any two modalities, such as a visible light-guided infrared branch, an infrared-guided visible light branch, a lidar-guided infrared branch, etc.

[0061] For example, when executing the "visible light guides infrared" (v→i) branch, the query matrix comes from visible light features, while the key and value matrices come from infrared features. In this way, each spatial location in the visible light features can find its most relevant spatial location in the infrared features, and the infrared feature content of that location can be extracted and added to its own feature representation, thus achieving cross-modal information extraction. The specific matrix representation can be as follows: ; in, , , These represent the linear change weights of the query matrix, key matrix, and value matrix, respectively. , , These are the current query matrix, the target key matrix, and the target value matrix, respectively. This is the deep feature map corresponding to the visible light mode. This is the target deep feature map corresponding to the infrared mode.

[0062] It should be noted that by element-wise multiplying the spatial quality soft mask matrix constrained by the current modality with the value matrix, attention gating of the spatial dimension can be adjusted, resulting in the adjustment gating. A specific formula example is as follows: ; in, Indicates adjustment gate control, This is the target value matrix.

[0063] Furthermore, a cross-attention mechanism can be used to extract a fused feature matrix corrected by a spatial quality soft mask, based on the adjusted gating, the current query matrix, and the target key matrix, as the output of the current branch. ; in, Represents visible light modes Guided infrared mode Branch output, This represents the preset scaling factor to prevent the gradient of the Softmax normalization function from entering the saturation region due to an excessively large dot product result.

[0064] In some embodiments of this application, to prevent the randomly initialized Transformer structure from disrupting the existing pre-trained feature distribution of the backbone network and causing task alignment learning to collapse during the early stages of training, embodiments of this application may introduce zero-initialization learning parameters at the output of each branch. (Initial value set to 0.0), the attention residuals after LayerNorm normalization are gated to obtain the deep feature maps of each modality after modulation: ; ; in, This represents the deep feature map corresponding to the visible light mode after modulation. This represents the deep feature map corresponding to the infrared mode after modulation.

[0065] It should be noted that, because during training step 0, and When the value is 0, the output of the fusion branch is completely suppressed, and the model fully inherits the detection performance of the single modality. With gradient updates during backpropagation during training, the model gradually learns "when and with what strength" to fuse cross-modal features. The final multimodal fusion output is processed by globally quality adaptive weights. Sure: ; ; in, Indicates the global fusion weight. represents the Sigmoid activation function, GAP represents the global average pooling layer, and MLP represents the channel compressed perceptron. This is the preset residual control coefficient, and its value can be 0.2; This represents the spatiotemporally aligned infrared feature map. This represents the multimodal fusion feature.

[0066] Understandably, the above methods enable multiple modalities to guide each other. For example, the visible light modality utilizes the thermal radiation information of the infrared modality to supplement the perception capability under degraded lighting conditions, while the infrared modality utilizes the texture information of the visible light modality to supplement the resolution capability of target details. Finally, through adaptive weighted fusion, the contribution ratio of each modality is dynamically determined based on the global quality of each modality, thereby obtaining the final multimodal fusion feature.

[0067] In some embodiments of this application, the overall flowchart of multi-scale feature fusion and trajectory tracking can be as follows: Figure 5 As shown, Figure 5 This is a flowchart illustrating the multi-scale feature fusion and trajectory tracking process in one implementation of the multimodal anti-drone detection method of this application.

[0068] like Figure 5As shown in this embodiment, the fused multimodal features can be input into the detection head for target detection, directly regressing the target bounding box and classification confidence of the object, thus overcoming the missed detection defects caused by traditional non-maximum suppression in dense and overlapping regions. Specifically, the step of detecting based on the multimodal fusion features to obtain multimodal anti-drone detection results includes: inputting the multimodal fusion features and the shallow high-resolution feature maps of each modality into the detection head for target detection, obtaining the target bounding box and classification confidence of the corresponding modality, wherein the multimodal anti-drone detection results include the target bounding box and classification confidence of each modality.

[0069] It should be noted that during the training phase, the bounding box regression uses the intersection-over-union (CIoU) loss function to improve the fitting accuracy: ; ; in, Indicates the loss by intersection and union. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Measuring the difference between the predicted box b and the true box The Euclidean distance between the center points; c is the diagonal distance between the minimum bounding rectangle of the predicted box and the ground truth box; v is used to measure the aspect ratio consistency between the predicted box and the ground truth box. Given that drones are usually elongated, the proportional penalty term in CIoU can significantly improve the localization and fitting accuracy at small scales.

[0070] It should be noted that when determining the target bounding box and classification confidence, the sequence of target bounding boxes composed of each frame's target bounding boxes can be input into the lightweight tracking framework for temporal association (ByteTrack). To address the issue of trajectory breakage and target ID switching caused by the temporary occlusion of small targets by clouds and trees in videos, this application proposes an adaptive weighted multimodal matching mechanism. For each target bounding box, a weighted fusion score can be obtained by adaptively calculating the weighted fusion of its confidence scores under different modalities. An example formula is as follows: ; in, This represents the classification confidence score of a target bounding box in the infrared modality. This represents the classification confidence of the bounding box corresponding to the visible light mode. This indicates an adaptive scene adjustment weight, which can be adaptively adjusted according to the perceived quality of the current scene environment. It is used to adaptively allocate the proportion of visible light appearance features and infrared heat source signals in the matching process. The specific adjustment method is not limited in the embodiments of this application.

[0071] Furthermore, the matching cost matrix between the target bounding box a and its corresponding predicted trajectory b can be... Defined as a joint function of the intersection-union distance and the difference in the overall score: ; in, The location distance is used to control the weight. This represents the intersection-union ratio (IoU) between the target bounding box and the predicted detection box corresponding to the predicted trajectory in the current frame. This represents the weighted fusion score corresponding to the target bounding box 'a'. This represents the weighted fusion score corresponding to the predicted bounding box in the predicted trajectory b.

[0072] In some embodiments of this application, the Hungarian algorithm can be used to obtain the optimal temporal matching between the current frame and the historical trajectory pipeline, and a unique and spatiotemporally continuous flight trajectory pipeline (i.e., predicted trajectory) can be generated for each target bounding box, thereby effectively solving the flickering and trajectory breakage phenomena in single-frame detection.

[0073] It should be noted that the Hungarian algorithm is a combinatorial optimization algorithm that solves the task allocation problem in polynomial time. Specifically, for the trajectory corresponding to each detection box in the previous frame, a Kalman filter is used to predict its possible position and shape in the current frame (i.e., the "predicted box"). The "matching cost" between all detection boxes and all predicted boxes in the current frame is calculated. The Hungarian algorithm then predicts a matching scheme with the minimum total cost, which is used as the current predicted trajectory for the updated current frame.

[0074] In some embodiments of this application, zero-sample false alarm elimination and intent assessment can also be performed based on 7D physical spatiotemporal cue word encoding and large language model. A detailed flowchart can be found below. Figure 6 , Figure 6 This is a flowchart illustrating the overall process of 7D motion-visual semantic encoding and large model-driven cognitive reasoning in one implementation of the multimodal anti-drone detection method of this application.

[0075] like Figure 6As shown, for the current predicted trajectory (i.e., the output spatiotemporal trajectory segment) corresponding to the updated current frame, continuous temporal bounding box parameters and corresponding local image region patches can be extracted. In this way, the target region in the original high-resolution image can be cropped out, retaining only the local image information related to the target, instead of directly transmitting the entire high-resolution image to the visual language model, thus avoiding extremely high inference latency and hardware overhead. In this embodiment, 7D Structured Physics Prompt encoding is used to transform the target's visual features, thermal features, and kinematic features into a structured text description. Specifically, the 7D structured prompt can include kinematic channel prompts and visual geometric channel prompts. In this embodiment, the kinematic channel includes: absolute velocity. (Unit: pixels / frame), coefficient of variation of acceleration Scale change rate Mean trajectory curvature wing flapping frequency domain variance The visual geometry channel includes: Harris corner density. and heat source concentration .

[0076] Among them, the first dimension Absolute velocity can be determined by calculating the average displacement of the target's center of gravity over N consecutive frames: ; in, Let be the centroid coordinates of the target bounding box in frame t.

[0077] The second dimension coefficient of variation of acceleration Used to characterize the active driving characteristics of motion: ; in, and These are the standard deviation and mean of the acceleration sequence, respectively. Represents the instantaneous acceleration in frame t. This represents the instantaneous velocity in frame t. Different types of flying targets exhibit significant distribution characteristics on this parameter; for example, rotary-wing drones and birds show obvious statistical differences in this parameter, which can be filtered out using a preset coefficient of variation threshold.

[0078] The third dimension Scale change rate By calculating the relative height of the target and the rate of change of its projected area: ; in, This represents the area of ​​the target bounding box in frame t. This represents the width of the target bounding box in frame t. This represents the height of the target bounding box in frame t.

[0079] The fourth dimension mean of trajectory curvature Used to characterize the smoothness of flight paths: ; Since the maneuvering trajectory of drones is mostly a straight line or a smooth curve with a large radius, while the trajectories of biological targets such as birds or kites pulled by ropes have significant differences in smoothness and mean curvature, they can be distinguished by a preset mean curvature threshold.

[0080] The fifth dimension Flapping wing frequency domain characteristics include maximum amplitude With main frequency Or the frequency domain variance of flapping wings. Specifically, in order to completely eliminate long-tailed false positives of birds in videos, this application calculates the temporal aspect ratio sequence of the target bounding box within the trajectory. Fast Fourier Transform (FFT): ; If the maximum value The amplitude is greater than the preset threshold, and its corresponding dominant frequency is... If the target is within a specific wing-flapping frequency range, it will trigger the "strong organic flapping wing feature" marker, indicating that the target has biological flapping wing characteristics.

[0081] The sixth dimension Visual corner density Extracting the total number of corner points within a local image of a target using the Harris operator. : ; in, This represents the pixel area (number of pixels) of the target bounding box.

[0082] Because rigid mechanical drones have straight and multi-pointed mechanical structures such as arms, propellers, and battery packs, their corner density is significantly higher than that of interfering targets such as birds and kites with relatively rounded outlines. Therefore, they can be distinguished by setting a corner density threshold.

[0083] The seventh dimension Infrared heat source concentration Determined by calculating the luminance spatial variance of the infrared ROI image: ; in, This represents the matrix of infrared image pixel values ​​corresponding to the region of the target bounding box. This represents variance. Compared to the surface temperature distribution of a biological target, the extremely strong hotspot of a drone's motor exhibits differences in infrared brightness spatial variance and heat source concentration. The data exhibits significant differences in concentration, which can be effectively classified by setting temperature variance and concentration thresholds.

[0084] It should be noted that by formatting the aforementioned 7D physical and geometric attributes into highly structured XML-style text segments, combining them into a physics commonsense decision prompt, and inputting it into a separately deployed large language model (such as DeepSeek-Chat or other models), the model incorporates knowledge of aerodynamics and physical motion laws (such as "birds must flap their wings, drones can hover, kites have no infrared temperature difference and are constrained by the swing of their ropes"). Through zero-sample chain of thought, the model can completely eliminate more than 85% of long-tail false alarms such as birds and kites that are extremely difficult to distinguish, while ensuring recall. Ultimately, this improves the system's accuracy to 95.82%, and simultaneously outputs the target's high-level tactical intent (such as hovering reconnaissance, flyaround mapping, or straight-line approach). Specifically, after the step of obtaining the current predicted trajectory of the current frame based on the matching cost matrix, the method further includes: extracting a seven-dimensional physical feature set from the current predicted trajectory, the seven-dimensional physical feature set including: absolute velocity, acceleration coefficient of variation, scale change rate, mean trajectory curvature, flapping wing frequency domain features, visual corner density, and infrared heat source concentration; formatting the seven-dimensional physical feature set into structured text prompts; inputting the structured text prompts into a large language model, the large language model using built-in aerodynamics and physical motion common sense to perform zero-shot logistic chain inference on the structured text prompts, and outputting the target category determination result and behavior intention recognition result of the trajectory pipeline.

[0085] It is understandable that the aforementioned aerodynamics and physical motion common sense can refer to the basic knowledge about the laws of motion, mechanical principles, and flight physics of objects learned by the large language model from massive amounts of text during the pre-training phase. Examples include common-sense knowledge such as "birds flap their wings periodically during flight," "drones have active propulsion systems that allow them to hover," and "kites rely on ropes for traction, and their motion is constrained by cycloids." The aforementioned zero-sample logical chain reasoning can refer to the large language model outputting conclusions directly based on the input structured text prompts without any fine-tuning or training for the current task, through step-by-step reasoning (e.g., first analyzing speed characteristics, then wing flapping characteristics, and finally comprehensively determining the target category). The aforementioned target category determination result can refer to the target classification label output by the large language model after reasoning. This result includes excluding interfering targets such as birds, kites, and balloons while retaining drone targets. The aforementioned behavioral intent recognition result can refer to the target tactical behavior label output by the large language model after reasoning. This result includes typical low-altitude threat behavior types such as high-altitude hovering reconnaissance, straight-line close-in penetration, and fixed-point circumnavigation mapping.

[0086] In some embodiments of this application, the specific method for formatting the seven-dimensional physical feature set is as follows: the detection device organizes the numerical parameters of the seven dimensions—absolute velocity, acceleration coefficient of variation, scale change rate, mean trajectory curvature, maximum amplitude and dominant frequency of flapping wing in the frequency domain, visual corner density, and infrared heat source concentration—into a text string with a clear semantic hierarchy according to the XML tag format. Each physical feature corresponds to an independent tag, and the tag contains the numerical value of the feature and its physical meaning description.

[0087] In some embodiments of this application, during the zero-sample logic chain inference process described above, the large language model progressively analyzes the information contained in each physical feature. The large language model first determines whether the infrared heat source concentration is higher than a threshold; if it is higher, it indicates that the target has mechanical heat source characteristics and is likely to be identified as a drone; if it is lower than the threshold, it indicates that the target lacks a concentrated heat source and may be a bird or kite. The large language model then determines whether the maximum amplitude of the flapping wing frequency domain is within the typical frequency band of bird flapping; if so, it excludes drones and identifies the target as a bird. The large language model then comprehensively judges whether the acceleration coefficient of variation and the mean trajectory curvature conform to the characteristics of an actively powered aircraft; if the acceleration coefficient of variation is low and the trajectory is smooth, it tends to be a drone; if the trajectory exhibits cycloidal characteristics, it tends to be a kite. The large language model combines visual corner density for final verification; if the corner density is high, it is consistent with rigid mechanical structure characteristics; if the corner density is low, it is consistent with biological or flexible structure characteristics.

[0088] In some embodiments of this application, after the above-mentioned multi-dimensional stepwise reasoning, the large language model outputs the target category determination result of the current predicted trajectory (excluding interfering targets such as birds, kites, and balloons and retaining UAV targets) and the behavior intention recognition result (such as high-altitude hovering reconnaissance, straight-line approach penetration or fixed-point circumnavigation mapping).

[0089] It should be noted that, through the technical solution of combining the above-mentioned seven-dimensional physical feature encoding with large-scale zero-sample common sense reasoning, the detection equipment can accurately distinguish between drones and interference targets such as birds and kites with highly similar underlying visual features from the cognitive level, reducing the false alarm rate of the system to an extremely low level; at the same time, it can also automatically judge the tactical intentions of the target based on the temporal movement pattern, realizing the functional upgrade from passive detection and alarm to active threat prediction.

[0090] This application embodiment constructs a current query matrix based on the current deep feature map of the current modality, and constructs a target key matrix and a target value matrix based on the target deep feature map of the target modality. The current modality is any modality among different modalities, and the target modality is any modality other than the current modality among different modalities. An element-wise multiplication of the spatial quality soft mask matrix constrained by the current modality and the target value matrix yields an adjustment gating. Based on the adjustment gating, the current query matrix, and the target key matrix, the current branch output guiding the target modality from the current modality is determined. Weighted fusion is performed based on the branch outputs corresponding to each modality, multi-scale features, and the constrained spatial quality soft mask matrix to obtain multi-modal fusion features. Since the query matrix is ​​constructed from the deep feature map of the current modality (any one of multiple modalities, such as visible light or infrared), and the key and value matrices are constructed from the deep feature map of the target modality, unidirectional feature querying and information extraction from the current modality to the target modality is achieved. By calculating the correlation strength between spatial locations of the current modality and spatial locations of the target modality through a cross-attention mechanism, the direction and intensity of cross-modal information interaction are precisely controlled by the attention weight matrix. This enables explicit information querying of "modality-guided modality." For example, the visible light modality can actively query thermal radiation distribution information from the infrared modality to compensate for insufficient perception under light degradation, and the infrared modality can actively query texture detail information from the visible light modality to enhance target resolution. The two modalities complement and reinforce each other. By performing element-wise multiplication of the spatial quality soft mask matrix constrained by the current modality with the target value matrix, an adjustment gating is obtained. This sets all feature channel values ​​corresponding to untrusted spatial locations in the target modality to zero in the value matrix. Thus, in subsequent attention calculations, even if the current modality assigns a high attention weight to that location, no actual feature content can be extracted. This avoids the problem of feature pollution of high-quality modalities by low-quality modalities and effectively avoids detection box positioning drift and accuracy degradation caused by forced fusion of low-quality noise.

[0091] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multimodal anti-drone detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0092] This application also provides a multimodal anti-drone detection device; please refer to... Figure 7 , Figure 7 This is a schematic diagram of the module structure of the multimodal anti-drone detection device according to an embodiment of this application. The multimodal anti-drone detection device includes: Feature acquisition module 10 is used to acquire multi-scale features corresponding to different modalities in the detection space; The matrix generation module 20 is used to calculate the confidence score for each spatial location in the space to be detected based on the multi-scale features, and obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. The matrix constraint module 30 is used to obtain the lower bound suppression value of the quality and to perform numerical constraints on each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality to obtain the constrained spatial quality soft mask matrix. The feature fusion module 40 is used to fuse the multi-scale features based on the spatial quality soft mask matrix after each modality constraint to obtain multi-modal fused features; The target detection module 50 is used to perform detection based on the multimodal fusion features to obtain multimodal anti-drone detection results.

[0093] The multimodal anti-drone detection device provided in this application, employing the multimodal anti-drone detection method described in the above embodiments, can solve the technical problem of low recognition accuracy in existing multimodal anti-drone detection methods. Compared with the prior art, the beneficial effects of the multimodal anti-drone detection device provided in this application are the same as those of the multimodal anti-drone detection method provided in the above embodiments, and other technical features in the multimodal anti-drone detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0094] This application provides a multimodal anti-drone detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal anti-drone detection method in the above embodiment 1.

[0095] The following is for reference. Figure 8The diagram illustrates a structural schematic suitable for implementing the multimodal anti-drone detection device of the embodiments of this application. The multimodal anti-drone detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablet computers, PMPs (Portable Media Players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The multimodal anti-drone detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 8 As shown, the multimodal anti-drone detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multimodal anti-drone detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multimodal anti-drone detection equipment to communicate wirelessly or wiredly with other devices to exchange data. Although a multimodal anti-drone detection equipment with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0097] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0098] The multimodal anti-drone detection device provided in this application, employing the multimodal anti-drone detection method described in the above embodiments, can solve the technical problem of low recognition accuracy in existing multimodal anti-drone detection methods. Compared with the prior art, the beneficial effects of the multimodal anti-drone detection device provided in this application are the same as those of the multimodal anti-drone detection method provided in the above embodiments, and other technical features of this multimodal anti-drone detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0099] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multimodal anti-drone detection method described in the above embodiments.

[0102] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0103] The aforementioned computer-readable storage medium may be included in the multimodal anti-drone detection equipment; or it may exist independently and not be assembled into the multimodal anti-drone detection equipment.

[0104] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the multimodal anti-drone detection device, cause the multimodal anti-drone detection device to: Obtain multi-scale features corresponding to different modalities in the detection space; Based on the multi-scale features, a confidence score is calculated for each spatial location in the space to be detected to obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. Obtain the lower bound suppression value of the quality, and apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality to obtain the constrained spatial quality soft mask matrix; Based on the spatial quality soft mask matrix after each modal constraint, the multi-scale features are fused to obtain multi-modal fused features; Based on the multimodal fusion features, detection is performed to obtain multimodal anti-drone detection results.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0108] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multimodal anti-drone detection method, which can solve the technical problem of low recognition accuracy in existing multimodal anti-drone detection methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multimodal anti-drone detection method provided in the above embodiments, and will not be repeated here.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal anti-drone detection method described above.

[0110] The computer program product provided in this application can solve the technical problem of low recognition accuracy in existing multimodal anti-drone detection methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multimodal anti-drone detection methods provided in the above embodiments, and will not be repeated here.

[0111] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A multimodal anti-drone detection method, characterized in that, The method includes: Obtain multi-scale features corresponding to different modalities in the detection space; Based on the multi-scale features, a confidence score is calculated for each spatial location in the space to be detected to obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. Obtain the lower bound suppression value of the quality, and apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality to obtain the constrained spatial quality soft mask matrix; Based on the spatial quality soft mask matrix after each modal constraint, the multi-scale features are fused to obtain multi-modal fused features; Based on the multimodal fusion features, detection is performed to obtain multimodal anti-drone detection results.

2. The multimodal anti-drone detection method as described in claim 1, characterized in that, The step of numerically constraining the matrix elements in the spatial quality soft mask matrix based on the lower quality bound suppression value to obtain the constrained spatial quality soft mask matrix includes: The matrix elements in the spatial quality soft mask matrix are traversed to determine the first matrix element that is lower than the lower quality suppression value and the second matrix element that is not lower than the lower quality suppression value. Set the first matrix element in the spatial quality soft mask matrix to zero, and suppress the second matrix element in the spatial quality soft mask matrix based on the lower quality limit suppression value to obtain the constrained spatial quality soft mask matrix.

3. The multimodal anti-drone detection method as described in claim 1, characterized in that, The step of acquiring multi-scale features corresponding to different modalities in the detection space includes: Modal images of different modalities in the space to be detected are acquired, wherein the temporal synchronization and spatial registration of each modality are achieved. Each modal image is input into the corresponding backbone network for feature extraction to obtain corresponding multi-scale features, wherein the multi-scale features include shallow high-resolution feature maps and deep feature maps, and the deep feature maps include mid-level feature maps and deep semantic feature maps; Accordingly, the step of calculating a confidence score for each spatial location in the space to be detected based on the multi-scale features to obtain the spatial quality soft mask matrix of the corresponding modality includes: Based on the deep feature map, a confidence score is calculated for each spatial location in the space to be detected, and the spatial quality soft mask matrix of the corresponding modality is obtained.

4. The multimodal anti-drone detection method as described in claim 3, characterized in that, The step of fusing the multi-scale features based on the spatial quality soft mask matrix after each modal constraint to obtain multi-modal fused features includes: The current query matrix is ​​constructed based on the current deep feature map of the current modality, and the target key matrix and target value matrix are constructed based on the target deep feature map of the target modality. The current modality is any one of the different modalities, and the target modality is any one of the different modalities other than the current modality. The spatial quality soft mask matrix after the target modal constraint is multiplied element-wise with the target value matrix to obtain the adjustment gating; The current branch output of the target mode, guided by the current mode, is determined based on the adjustment gating, the current query matrix, and the target key matrix. The multimodal fusion feature is obtained by weighted fusion of the branch outputs corresponding to each modality, the multi-scale features, and the constrained spatial quality soft mask matrix.

5. The multimodal anti-drone detection method as described in claim 4, characterized in that, The step of detecting based on the multimodal fusion features to obtain multimodal anti-drone detection results includes: The multimodal fusion features and the shallow high-resolution feature maps of each modality are input into the detection head for target detection to obtain the target bounding box and classification confidence score corresponding to each modality. The multimodal anti-drone detection result includes the target bounding box and classification confidence score corresponding to each modality.

6. The multimodal anti-drone detection method as described in claim 5, characterized in that, After the step of detecting based on the multimodal fusion features to obtain the multimodal anti-drone detection result, the method further includes: A weighted fusion score for each target bounding box is determined based on the classification confidence of each modality; Determine the intersection-union distance and weighted fusion score difference between all current target bounding boxes and predicted trajectories in the current frame, and determine the matching cost matrix between the current target bounding box and the historical predicted trajectory based on the intersection-union distance and the weighted fusion score difference. The modal image includes images of multiple consecutive frames in the corresponding modality, and the current frame is any one of the multiple consecutive frames. The current predicted trajectory of the current frame is obtained by constraining the matching cost matrix.

7. A multimodal anti-drone detection device, characterized in that, The multimodal anti-drone detection device includes: The feature acquisition module is used to acquire multi-scale features corresponding to different modalities in the detection space; The matrix generation module is used to calculate the confidence score for each spatial location in the space to be detected based on the multi-scale features, and obtain the spatial quality soft mask matrix of the corresponding mode. The spatial quality soft mask matrix is ​​used to characterize the local signal-to-noise ratio and reliability of the multi-scale features of the corresponding mode at each spatial location. The matrix constraint module is used to obtain the lower bound suppression value of the quality, and to apply numerical constraints to each matrix element in the spatial quality soft mask matrix based on the lower bound suppression value of the quality, so as to obtain the constrained spatial quality soft mask matrix. The feature fusion module is used to fuse the multi-scale features based on the spatial quality soft mask matrix after each modality constraint to obtain multi-modal fused features; The target detection module is used to perform detection based on the multimodal fusion features to obtain multimodal anti-drone detection results.

8. A multimodal anti-drone detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal anti-drone detection method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multimodal anti-drone detection method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multimodal anti-drone detection method as described in any one of claims 1 to 6.

Images