Method and system for identifying and classifying low-slow small flight target based on multi-modal large model

Abstract

The invention relates to a multi-modal large model-based low-slow small flight target identification and classification method and system, and aims at solving the problem of monitoring and identification of low-slow small flight targets in a complex scene. According to the method, various sensor data including visible light images, infrared images, radar signals, sound features and radio signals are fused, the multi-mode large model is used for processing and fusing data of different modes, combined features are extracted, efficient detection and classification of low-speed small targets are achieved, and the method has an automatic early warning function. The method has the advantages of real-time response, automatic processing and high recognition precision, and is suitable for low, slow and small flight target defense scenes such as airports, military bases and urban safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of flying target recognition, and in particular relates to a method and system for recognizing and classifying low, slow and small flying targets based on a multi-modal large model, which is suitable for the fields of unmanned aerial vehicle defense, aviation safety and urban protection. Background Art

[0002] With the increasing use of low, slow and small flying targets (such as small drones), the problem of illegal intrusion caused by them has become increasingly prominent. Due to the characteristics of these flying targets being small in size, slow in flight speed and low in altitude, the existing detection technology based on a single modality (such as radar or optical images) cannot effectively cope with the challenge of target recognition in complex environments. Therefore, there is an urgent need for a system that integrates multimodal data, can comprehensively analyze information from different sensors, and improve the ability to recognize and classify low, slow and small targets. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a method and system for identifying and classifying low, slow and small flying targets based on a multimodal large model. The system fuses a variety of sensor data (including optical images, infrared images, radar signals, sound features and radio signals) through a multimodal large model, and uses a deep learning model to extract and fuse data features of different modes to achieve accurate detection and classification of low, slow and small flying targets. The real-time warning function of the system significantly improves the monitoring accuracy and response speed, and can effectively deal with the safety hazards caused by low, slow and small flying targets.

[0004] The technical solution adopted by the present invention is: a low, slow and small flying target recognition and classification system based on a multi-modal large model, which constructs the following hierarchical architecture to realize the early warning and defense of low, slow and small flying targets, including:

[0005] The device layer is a multimodal data acquisition module composed of multiple groups of sensor devices, which is used to collect multimodal heterogeneous data;

[0006] The middle layer builds a multimodal network architecture model and trains and optimizes it to obtain an ideal model, which is used to extract and fuse features from the collected multimodal heterogeneous data, and further detect and classify low-flying, slow-moving, and small-flying targets based on the fused features.

[0007] The data layer uses a system database to store multimodal data, extracted features, detection and classification results;

[0008] The application layer adopts a user-oriented visual interface to receive the input sensor acquisition parameters, the model parameters and training parameters of the middle layer, and the original, process and result data of the receiving device layer, middle layer and data layer for visual display.

[0009] The multimodal heterogeneous data includes visible light images, infrared radiation images, radar signals, radio signals and sound signals; the multiple groups of sensor devices include visible light cameras, infrared cameras, radar sensors, radio signal receivers and microphones.

[0010] The intermediate layer comprises:

[0011] The data preprocessing module preprocesses the collected multimodal data, including data alignment, denoising and data enhancement;

[0012] The multimodal feature extraction module includes building a multimodal feature extraction model and training and optimizing it to obtain an ideal model, which is used to extract the key features of each modality data and perform feature fusion;

[0013] The object detection module adopts a self-attention mechanism model, takes the fused features as input, and further extracts joint features to identify its shape, position and trajectory;

[0014] The target classification module classifies the joint features and distinguishes different types of low-flying, slow-flying, and small targets;

[0015] The early warning module generates early warning information based on the low, slow and small flying targets identified through detection and classification results.

[0016] The data alignment operation is to align data in two dimensions of time and space, including synchronizing timestamps of data of different modalities and integrating data of the same region as multi-modal heterogeneous data of the same low, slow and small flying target.

[0017] The multimodal network architecture model of the intermediate layer includes:

[0018] The data preprocessing module uses convolutional neural networks to extract the features of visible light and infrared images; uses convolutional neural networks or recurrent neural networks to extract the time-frequency graph features of sounds, or uses the Transformer structure to process time series data; uses 1D or 2D convolutional networks to extract the features of radar signals and obtain distance-speed feature maps; processes radio signals through convolutional networks, spectrum analysis or FFT to obtain frequency domain information; thus, preprocessed multi-modal heterogeneous data are obtained respectively;

[0019] The multimodal feature extraction module uses a fully connected layer or Transformer structure to fuse data from different sensors and extract fusion features;

[0020] The target detection module uses a self-attention mechanism network to further learn and fuse features to enhance the detection accuracy of low, slow and small flying targets;

[0021] The target classification module uses a softmax classifier to classify low, slow and small flying targets.

[0022] The training of the multimodal network architecture model includes:

[0023] Construct a long time series data set of low-flying, slow-flying and small targets {joining of multi-modal heterogeneous data, target type labels}; divide it into training set, validation set and test set; the training set is used for training, the validation set is used to verify the model accuracy, and the test set is used for testing;

[0024] Establish a loss function, using cross entropy loss or regression loss;

[0025] Set training parameters, including optimizer, learning rate, number of iterations, and cutoff conditions.

[0026] The application layer includes:

[0027] Situational awareness unit, which monitors and analyzes drone activity in the airspace in real time to identify potential threats;

[0028] The trajectory reproduction unit is used to display the long-time flight trajectory of the identified low-flying, slow-flying, small target;

[0029] The warning information display unit is used to display the identification and warning information of low, slow and small flying targets.

[0030] The early warning information display unit notifies the management personnel of abnormal flight targets in real time through sound and light alarms and SMS notifications.

[0031] A method for identifying and classifying low-flying, slow-flying, and small-flying targets based on a multimodal large model comprises the following steps:

[0032] Step S1: input sensor acquisition parameters, model parameters and training parameters of the intermediate layer, and low-speed, slow, and small flying targets to be identified through the visual interface of the application layer; control multiple groups of sensor devices in the device layer to collect multi-modal heterogeneous data;

[0033] Step S2: The middle layer constructs a multimodal network architecture to extract and fuse the collected multimodal heterogeneous data, and further detects and classifies low-flying, slow-flying, and small-flying targets based on the fused features;

[0034] Step S3: using a system database to store multimodal data, extracted features, detection and classification results to a data layer;

[0035] Step S4: Display the multi-modal heterogeneous data, feature data, and classified warning result data of the low, slow, and small flying target through a visual interface on the application layer, and reproduce its flight trajectory.

[0036] The step S2 specifically includes:

[0037] Step S201, preprocessing the collected multimodal data, including data alignment, denoising and data enhancement;

[0038] Step S202: construct a multimodal feature extraction model and train and optimize it to obtain an ideal model for extracting key features of each modality data and performing feature fusion;

[0039] Step S203: adopt a self-attention mechanism model, take the fusion features as input, further extract joint features, and thus identify their shape, position, and trajectory;

[0040] Step S204: classify the joint features to distinguish different types of low-flying, slow-flying, and small-flying targets;

[0041] Step S205: Generate warning information based on the low, slow and small flying targets identified by the detection and classification results.

[0042] Beneficial effects and advantages of the present invention:

[0043] 1. Multimodal data fusion: By fusing multimodal data such as optical, infrared, radar, sound and radio, the system significantly improves the detection and recognition accuracy of low, slow and small flying targets.

[0044] 2. Real-time warning function: The system has real-time warning capabilities and can issue alarms in time with low latency to ensure regional safety.

[0045] 3. Efficient feature extraction: Using multimodal large model technology, the system can extract key features of multi-source data from complex environments to ensure high accuracy of target detection and classification.

[0046] 4. Historical data management: The system database module can store multimodal data, features, detection results and classification results to provide support for subsequent research and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural schematic diagram of the system of the present invention;

[0048] Figure 2 It is the work flow chart of the method of the present invention;

[0049] Figure 3 This is a diagram of the multimodal network architecture of the present invention. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is further described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0051] Figure 1 Shown is a structural schematic diagram of the recognition and classification system for low-flying, slow-moving, and small-flying targets based on a multimodal large model of the present invention. The system includes: a device layer, which is a multimodal data acquisition module composed of multiple groups of sensor devices, used to collect multimodal heterogeneous data; an intermediate layer, which is used to extract and fuse features of the collected multimodal heterogeneous data, and further detect and classify low-flying, slow-moving, and small-flying targets based on the fused features; a data layer, which uses a system database to store multimodal data, extracted features, and detection and classification results; an application layer, which uses a user-oriented visual interface to receive input sensor acquisition parameters, as well as various model parameters and training parameters of the intermediate layer, and receive the original, process, and result data of the device layer, intermediate layer, and data layer for visual display. The implementation method is as follows:

[0052] Multimodal data acquisition: Collect multimodal data of the target through different types of sensors (such as visible light cameras, infrared cameras, radar sensors, radio receivers, and microphones).

[0053] Data preprocessing: align the collected multimodal data in time and space, remove noise, and perform data enhancement.

[0054] Multimodal feature extraction: Use a large multimodal model to extract key features of different modalities from the data and fuse them through deep learning methods.

[0055] Target detection: Based on a large multimodal model, a self-attention mechanism is used to perform target detection on the fused features to identify the shape, position, and trajectory of low, slow, and small flying targets.

[0056] Target classification: The detected targets are classified through the classification model to distinguish different types of low, slow and small flying targets (such as small drones, micro drones, floating balloons in the sky, flying birds, etc.).

[0057] Data storage and early warning: The detection and classification results are stored in the system database, and when an abnormal target appears, an audible and visual alarm or SMS notification is issued in time through the early warning module. In the present invention, an abnormal target refers to a flying target that the user needs to identify.

[0058] Situational awareness: Used to monitor and analyze drone activity in the airspace in real time to identify potential threats and respond accordingly.

[0059] Trajectory reproduction: used to display the long-time flight trajectory of the identified low, slow and small flying targets; warning information display unit, used to display the identification and warning information of low, slow and small flying targets.

[0060] Figure 2 The flowchart of the identification and classification method of low-speed, slow, and small flying targets based on the multimodal large model of the present invention is shown. First, a variety of sensor data are collected through the multimodal data acquisition module; then, the data preprocessing module preprocesses the data; then, the multimodal feature extraction module extracts features; the target detection module identifies low-speed, slow, and small flying targets; the target classification module classifies the targets; the system database module stores relevant data; finally, the warning module issues a warning signal for abnormal situations. The specific steps are as follows:

[0061] 1. Data collection and preprocessing

[0062] Visible light image data: A large number of visible light images are collected from surveillance cameras and drone flight scenes. These images need to be annotated and the target drones need to be separated from the background. Annotation is to label the data according to different types of low, slow and small flying targets (such as small drones, micro drones, floating balloons in the sky, flying birds, etc.).

[0063] In the flying target recognition algorithm, the flying target types that need to be recognized may include but are not limited to the following:

[0064] 1) Drones: Consumer Drones, Industrial Drones, Military Drones;

[0065] 2). Birds: Small Birds, Large Birds;

[0066] 3) Others: Traditional Kites, Power Kites, Aerostats, Hot Air Balloons, Rigid Airships, Paragliders.

[0067] Infrared image data: Use infrared cameras or thermal imaging equipment to collect infrared images of drones under different temperature and light conditions. These images need to be matched with visible light images. Matching means taking pictures of the same target.

[0068] Sound feature data: Use microphone arrays or directional microphones to collect the sound wave characteristics of drones. Different drones make different sounds when flying, so it is necessary to collect sounds of different models, speeds, and environments.

[0069] Radar signal data: The radar system captures the distance, speed, direction and other information of the drone and matches it with the image data. Matching is the signal collection of the same target.

[0070] Radio signal data: The radio signal characteristics of the drone are obtained through the RF detection system, including frequency, modulation mode, signal strength, etc., and matched with the image data. Matching is the signal collection of the same target.

[0071] Preprocessing steps:

[0072] Image data (visible light and infrared) needs to be unified in size, color channel, and normalized.

[0073] Perform spectral analysis on sound data and convert it into a time-frequency diagram (such as Mel spectrum) or directly process the original waveform data.

[0074] Radar and radio signals are filtered and feature extracted to generate time series or feature vectors.

[0075] 2. Multimodal Data Alignment and Synchronization

[0076] Since the data sources come from different sensors, the time and space dimensions may be inconsistent and need to be synchronized:

[0077] Time synchronization: ensuring that data from different modalities are aligned in time. For example, as a drone passes over a surveillance area, the captured image, infrared, sound, radar, and radio data need to be time-stamped accordingly.

[0078] Spatial synchronization: Ensure that the monitoring areas of each sensor are consistent, and align different modal data into the same spatial coordinate system through coordinate transformation.

[0079] At this point, a spliced ​​dataset of long-term multimodal heterogeneous data of a certain type of low-flying, slow-flying, small target is obtained {spinning of multimodal heterogeneous data, target type label}.

[0080] 3. Multimodal feature extraction

[0081] Each modality data needs to be extracted through a specific network:

[0082] Image feature extraction: Use convolutional neural networks (CNNs) to extract features from visible and infrared images. You can use pre-trained models (such as ResNet, VGG) and then fine-tune them for drone recognition tasks.

[0083] Sound feature extraction: Use convolutional neural networks or recurrent neural networks (RNNs) to extract the time-frequency graph features of sounds, or use the Transformer structure to process time series data.

[0084] Radar feature extraction: Use 1D or 2D convolutional networks to extract features of radar signals, such as range-velocity feature maps.

[0085] Radio signal feature extraction: The frequency domain information of radio signals can be processed through convolutional networks, spectrum analysis or FFT.

[0086] 4. Multimodal feature fusion

[0087] The features of different modalities are combined using fusion strategies, which include the following three fusion strategies: early fusion, mid-term fusion, and late fusion.

[0088] Early fusion: In this strategy, feature data from different modalities are pre-merged before entering the model. This merging can be achieved by feature concatenation, feeding into a unified network (the multimodal network architecture entrance constructed by the inventor) for processing, or other combined methods. This step aims to improve the performance of the model by making full use of multimodal information. However, early fusion may lead to the introduction of information redundancy and noise.

[0089] Mid-term fusion: This strategy implements feature fusion after each modality feature has been processed independently. In this stage, the features of each modality are first extracted and processed separately (referring to the preprocessing network of the five modal data in the above multimodal feature extraction), and then these features are combined (finally adding a fully connected layer or Transformer structure). The advantage of mid-term fusion is that it can retain the unique features of each modality and reduce the impact of noise on the final result.

[0090] Late fusion: In this strategy, each modality is first processed independently and generates its own predictions. Finally, these predictions are further combined by voting, weighted averaging, attention network or other synthesis methods to extract joint features. The advantage of late fusion is that it allows the processing of each modality to be optimized independently, although the deep correlation information between the modalities may be lost.

[0091] 5. Model structure design

[0092] Multimodal network architecture: Build a network architecture that can handle multiple modalities. You can use parallel branch networks to process data from each modality and then fuse them at a certain layer. For example, image data is processed by CNN, sound data is processed by RNN, and finally features are fused in a fully connected layer or Transformer encoder.

[0093] Transformer model: Using a cross-modal Transformer architecture, the features of each modality are modeled through a multi-head self-attention mechanism to capture the relationship between modalities.

[0094] Target classification, a softmax classifier is used to classify low, slow and small flying targets.

[0095] So far, we have obtained Figure 3 The multimodal network architecture diagram shown.

[0096] 6. Model Training

[0097] Loss function design: Design the loss function according to the task (e.g., classification, object detection, tracking). Joint loss functions such as cross entropy loss, regression loss, etc. can be used in multimodal learning.

[0098] Multi-task learning: If drone detection, identification, and tracking are required simultaneously, multi-task learning can be used to allow the model to optimize multiple objectives simultaneously during training.

[0099] Data enhancement: Perform conventional data enhancement on images (flipping, rotation, cropping, etc.), perform data enhancement on sounds (time-frequency domain jittering, time cropping, etc.), and perform enhancement operations such as noise addition or time stretching on radar and radio signals.

[0100] Training process: Use batch stochastic gradient descent (SGD) or Adam optimizers to gradually update model parameters. You need to choose a suitable learning rate and use the learning rate scheduler to dynamically adjust the learning rate.

[0101] 7. Model validation and testing

[0102] Use independent validation sets to evaluate the performance of the model on each modality, especially to test the performance improvement after multimodal fusion.

[0103] Test the robustness of the model in real scenarios, such as different weather conditions, interference environments, etc., to verify the generalization ability of the model.

[0104] 8. Model optimization and deployment

[0105] Pruning and quantization: To reduce the computational cost and memory usage of the model, the trained model can be pruned and quantized so that it can be deployed on edge devices or embedded systems.

[0106] Real-time inference: Deploy the model on devices with GPU acceleration to achieve real-time drone identification and early warning.

[0107] System database module:

[0108] Stores collected multimodal data, extracted features, detection results, and classification results, and provides historical data query and management functions.

[0109] Early warning module:

[0110] Analyze the detection and classification results. When an abnormal flying target is detected, a warning signal is generated and relevant personnel are notified in time through the alarm system (such as sound and light alarm, SMS notification, etc.).

[0111] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions, improvements, etc. made by any technician familiar with the technical field within the technical concept of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-speed, slow-moving, small-flying target recognition and classification system based on a multi-modal large model, characterized in that: The following hierarchical architecture is constructed to achieve early warning and defense of low-speed, slow, and small flying targets, including: The device layer is a multimodal data acquisition module composed of multiple groups of sensor devices, which is used to collect multimodal heterogeneous data; The middle layer builds a multimodal network architecture model and trains and optimizes it to obtain an ideal model, which is used to extract and fuse features from the collected multimodal heterogeneous data, and further detect and classify low-flying, slow-moving, and small-flying targets based on the fused features. The data layer uses a system database to store multimodal data, extracted features, detection and classification results; The application layer adopts a user-oriented visual interface to receive the input sensor acquisition parameters, the model parameters and training parameters of the middle layer, and the original, process and result data of the receiving device layer, middle layer and data layer for visual display.

2. According to claim 1, a low-speed, slow, and small flying target recognition and classification system based on a multi-modal large model is characterized in that: The multimodal heterogeneous data includes visible light images, infrared radiation images, radar signals, radio signals and sound signals; the multiple groups of sensor devices include visible light cameras, infrared cameras, radar sensors, radio signal receivers and microphones.

3. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 1 is characterized in that: The intermediate layer comprises: The data preprocessing module preprocesses the collected multimodal data, including data alignment, denoising and data enhancement; The multimodal feature extraction module includes building a multimodal feature extraction model and training and optimizing it to obtain an ideal model, which is used to extract the key features of each modality data and perform feature fusion; The object detection module adopts a self-attention mechanism model, takes the fused features as input, and further extracts joint features to identify its shape, position and trajectory; The target classification module classifies the joint features and distinguishes different types of low, slow and small flying targets. The early warning module generates early warning information based on the low, slow and small flying targets identified by the detection and classification results.

4. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 3 is characterized in that: The data alignment operation is to align data in two dimensions of time and space, including synchronizing timestamps of data of different modalities and integrating data of the same region as multi-modal heterogeneous data of the same low, slow and small flying target.

5. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 3 is characterized in that: The multimodal network architecture model of the intermediate layer includes: The data preprocessing module uses convolutional neural networks to extract the features of visible light and infrared images; uses convolutional neural networks or recurrent neural networks to extract the time-frequency graph features of sounds, or uses the Transformer structure to process time series data; uses 1D or 2D convolutional networks to extract the features of radar signals and obtain distance-speed feature maps; processes radio signals through convolutional networks, spectrum analysis or FFT to obtain frequency domain information; thus, preprocessed multi-modal heterogeneous data are obtained respectively; The multimodal feature extraction module uses a fully connected layer or Transformer structure to fuse data from different sensors and extract fusion features; The target detection module uses a self-attention mechanism network to further learn and fuse features to enhance the detection accuracy of low, slow and small flying targets; The target classification module uses a softmax classifier to classify low, slow and small flying targets.

6. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 5 is characterized in that: The training of the multimodal network architecture model includes: Construct a long time series data set of low-flying, slow-flying and small targets {joining of multi-modal heterogeneous data, target type labels}; divide it into training set, validation set and test set; the training set is used for training, the validation set is used to verify the model accuracy, and the test set is used for testing; Establish a loss function, using cross entropy loss or regression loss; Set training parameters, including optimizer, learning rate, number of iterations, and cutoff conditions.

7. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 1 is characterized in that: The application layer includes: Situational awareness unit, which monitors and analyzes drone activity in the airspace in real time to identify potential threats; The trajectory reproduction unit is used to display the long-time flight trajectory of the identified low-flying, slow-flying, small target; The warning information display unit is used to display the identification and warning information of low, slow and small flying targets.

8. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 7 is characterized in that: The early warning information display unit notifies the management personnel of abnormal flight targets in real time through sound and light alarms and SMS notifications.

9. A method for identifying and classifying low-flying, slow-flying, and small-flying targets based on a multimodal large model, characterized in that: The following steps are involved: Step S1: input sensor acquisition parameters, model parameters and training parameters of the intermediate layer, and low-speed, slow, and small flying targets to be identified through the visual interface of the application layer; control multiple groups of sensor devices in the device layer to collect multi-modal heterogeneous data; Step S2: The middle layer constructs a multimodal network architecture to extract and fuse the collected multimodal heterogeneous data, and further detects and classifies low-flying, slow-flying, and small-flying targets based on the fused features; Step S3: using a system database to store multimodal data, extracted features, detection and classification results to a data layer; Step S4: Display the multi-modal heterogeneous data, feature data, and classified warning result data of the low, slow, and small flying target through a visual interface on the application layer, and reproduce its flight trajectory.

10. The low, slow and small flying target recognition and classification system based on multi-modal large model according to claim 9 is characterized in that: The step S2 specifically includes: Step S201, preprocessing the collected multimodal data, including data alignment, denoising and data enhancement; Step S202: construct a multimodal feature extraction model and train and optimize it to obtain an ideal model for extracting key features of each modality data and performing feature fusion; Step S203: adopt a self-attention mechanism model, take the fusion features as input, further extract joint features, and thus identify their shape, position, and trajectory; Step S204: classify the joint features to distinguish different types of low-flying, slow-flying, and small-flying targets; Step S205: Generate warning information based on the low, slow and small flying targets identified by the detection and classification results.

Images